[N.B. Views are my own, not in any way officially representative of Harvard. This blog began when I worked for Harvard until mid-October 2019, and to date I still have access to the account, even though I am no longer affiliated.]
Simmons University in the U.S. hosts the Open Access Directory, a community-maintained wiki about the global movement for open access to scholarship. The directory was founded in 2008 by open access experts Peter Suber and the late Robin Peek.
In this post, I describe preparation and ingest of the directory’s list of “Blogs about Open Access” (oad.simmons.edu/oadwiki/Blogs_about_OA) to Wikidata, the free and open knowledge base. The goal: to increase discoverability.
1. Determine notability
The Open Access Directory is a standard reference source about the open access movement. Most of its listings meet the Wikidata notability requirement (www.wikidata.org/wiki/Wikidata:Notability).
2. Structure data
For this particular example, the directory’s wiki page contained a list of more than 300 blog titles and URLS in a somewhat consistent pattern. However, the semi-structured data required some reformatting prior to upload to Wikidata.
First, I pasted the content into a plain text document, then used the find and replace function to clean up the formatting.
Eventually the text was ready to be pasted into a spreadsheet, where a bit more cleanup was required.
3. Evaluate data
Some of the directory’s URLs had become outdated. To check for broken links, I followed Adham El Banhawy’s illustrated instructions for creating a short JavaScript script in Google Sheets.
function getStatusCode(url){
var options = {
‘muteHttpExceptions’: true,
‘followRedirects’: false
};
var url_trimmed = url.trim();
var response = UrlFetchApp.fetch(url_trimmed, options);
return response.getResponseCode();
}
The returned HTTP status codes (200, 301, 302, 403, 404, 410, 500, 503, etc.) were interpreted with httpstatuses.com.
4. Reconcile
I then downloaded the cleaned spreadsheet as a CSV file, then uploaded it to OpenRefine. In OpenRefine, I reconciled the title and URL against Wikidata. For this particular content, I was not surprised that only a few blogs matched Wikidata items. For unmatched blogs, I selected: Reconcile/Actions/Create-a-new-item-for-each-cell.
5. Build schema
The OpenRefine extension feature includes an “edit Wikidata schema” option, which allows one to build an import template for a group of items that have properties and references in common.
A schema can be put together by a combination of dragging and dropping the reconciled column icons, and manually typing statement components. In this case, all items were instances of blogs, all had official websites, all addressed the subject open access.
Each statement also included the same source: the Wikidata representation of the Open Access Directory page “Blogs About Open Access” (www.wikidata.org/wiki/Q56229362).
I previewed the schema prior to uploading. OpenRefine automatically found a few duplicates, which I then fixed.
6. Check quality
After the OpenRefine batch uploader created new items in Wikidata, I carefully reviewed each one. Next steps: the addition of details about language, editor, license, motto, and date of inception, for example, will be welcome.
Several blogs had live websites but otherwise seemed inactive. This information will be recorded in both Wikidata and the directory.
Finally, Wikidata Q numbers for each blog will be listed in the directory.
Resources
OpenRefine reconciliation guide by Antonin Delpeuch
https://github.com/OpenRefine/OpenRefine/wiki/Reconciliation
OpenRefine Google group
https://groups.google.com/forum/#!forum/openrefine
Wikidata help
https://www.wikidata.org/wiki/Help:Contents
Getting started as an Open Access Directory contributor
http://oad.simmons.edu/oadwiki/Getting_started_as_an_OAD_contributor
