Oxford Text Archive

The blog will focus on:

  • announcements of new resources in the Oxford Text Archive, and new features in the web interface;
  • explanations of how the OTA works and why, to help users make the most of it (e.g. how to log in, how SAML2 authentication works);
  • announcement of and reports from events with OTA involvement;
  • news about collaborations and connections, e.g. with CLARIN and DARIAH, and with developments in the UK research infrastructure;
  • legacy posts about the OTA and digital humanities, rescued from unreliable platforms.

The blog represents personal views, and is not the official blog of any organisation.

The Oxford Text Archive in 2016

Originally posted at blogs.it.ox.ac.uk on January 5, 2017 by Martin Wynne. IT Services at the University of Oxford has decided to delete a large number of historical blogs, and this is one of a number of posts related to the Oxford Text Archive which are being re-published here, after being laboriously retrieved from the archive provided by the Wayback Machine.

Analysis of the logs for downloads of resources from the Oxford Text Archive in the calendar year 2016 reveal a continuing increase in usage. A few years ago there was a big leap in downloads thanks to the ingest of a large number of texts from the Text Creation Partnership which became available via the OTA, when it became legally possible to share them openly.

Other factors aiding increased usage of the OTA include:

  • BNC free for download: during 2016 the British National Corpus was made available for direct download without having to fill in a form and wait for authorization, and as a result downloads continue to increase;
  • Freeing the texts: an ongoing programme of reassessing legacy data, and, where possible, removing access restrictions;
  • Higher visibility: resource discovery via the CLARIN Virtual Language Observatory, which aggregates OTA records and offers a new way for users to find the texts;
  • Shibbolization: a small number of resources are available currently for UK users only, but also slowly being opened up Europe-wide thanks to the CLARIN and EduGAIN;
  • More digital research: demand grows as more users in the humanities start to engage in digital scholarship.

The grand total for the discrete downloads of resources from the Oxford Text Archive was 1263810, or 1.26 million. Each of these represents the successful download of the content of a resource, and the numbers were calculated after filtering out all hits from spiders, crawlers, robots and other automated processes, and ignoring failed downloads.  The total is an increase of around 38% on last year’s total. Of these 395812 could be identified as originating from users in the University of Oxford, approximately 40%, and more than double the number from last year. Of the total downloads, more than 99.6% were direct downloads of resources made available at open URLs, the rest made up of the various resources where access restrictions require authorization.

Here are this year’s top ten:

Number of downloads Title Author ID Class
9313 The poems of John Keats Keats, John, 1795-1821 3259 text
8351 VOICE: Vienna-Oxford International Corpus of English Barbara Seidlhofer 2542 corpus
6543 British National Corpus, XML edition BNC Consortium 2554 corpus
4936 British National Corpus, Baby edition BNC Consortium 2553 corpus
4616 The four seasons, and other poems. By James Thomson Thomson, James, 1700-1748. 3549 ECCO
4407 An account of the proceedings against the rebels, and other prisoners, tried before the Lord Chief Justice Jefferies: and other judges in the west of England, in 1685. for taking arms under the Duke of Monmouth. … To which is prefix’d, the Duke of Monmouth’s, the Earl of Argyle’s, and the Pretender’s declarations, that the reader may the better judge of the cause of the several rebellions.   4431 ECCO
3696 Beggar’s opera. Libretto. Gay, John, 1685-1732 3257 text
3663 New York newspaper advertisements and news items: 1777-1779   3151 text
3613 The history of the most noble Order of the Garter: Wherein is set forth an account of the town, castle, chappel, and college of Windsor; … To which is prefix’d, a discourse of knighthood in general, … Collected by Elias Ashmole, … The whole illustrated with proper sculptures. Ashmole, Elias, 1617-1692. 5268 ECCO
3564 The peerage of Scotland: containing an historical and genealogical account of the nobility of that Kingdom. … By George Crawfurd, Esq;. Crawford, George, fl. 1710. 5301 ECCO

There is also a  table with the top 20 downloads of 2016. Overall, more than 36000 different resources were downloaded.

The table below shows the most popular items with access restrictions, which required an online application and manual authorization before they could be downloaded. There were 4401 of these downloads – over the year an average of more than ten per day which needed to be manually authorized by a member of staff. Last year there were 3681. Some tf the resources below were made freely available during the year, and so were accessed via direct download as well.

Number of downloads ID Title Notes
2664 British National Corpus, XML edition 2554 Also 3543 direct downloads and 356 via Shibboleth
320 British National Corpus, Baby edition 2553 Also 4439 direct downloads  and 176 via Shibboleth
236 The Lancaster Corpus of Mandarin Chinese 2474  
111 Helsinki corpus of English texts 1477  
97 British Academic Written English Corpus 2539 Also with 663 direct downloads
97 Complete corpus of Old English: the Toronto dictionary of Old English corpus / compiled by the University of Toronto Centre for Medieval Studies 0163  
74 Parsed Corpus of Early English Correspondence (PCEEC) 2510  
67 British Academic Spoken English corpus 2525  
55 Cat on a hot tin roof / Tennessee Williams 1233  
43 A Corpus of English Dialogues 1560-1760 (CED) 2507  
43 British National Corpus Sampler 2551  
43 The York-Toronto-Helsinki Parsed Corpus of Old English prose (YCOE) 2462  
31 Dictionary of Old English Corpus in Electronic Form (DOEC) 2488  

There were 556 downloads from the experimental site hosted by the Oxford e-Research Centre, where users can download one of a small number of resources (of which the BNC is the most popular) by authenticating with their institutional single sign-on. This is an increase from 321 last year, despite some periods of down-time for the service. Only thirty-six of these downloads were from the University of Oxford.

 

Oxford Text Archive moving to the Bodleian Libraries

Originally posted at blogs.it.ox.ac.uk on October 24, 2016 by Martin Wynne. IT Services at the University of Oxford has decided to delete a large number of historical blogs, and this is one of a number of posts related to the Oxford Text Archive which are being re-published here, after being laboriously retrieved from the archive provided by the Wayback Machine.

The Oxford Text Archive (OTA) is getting a new lease of life. It is moving to the Bodleian Library, in a transition which should guarantee its long-term sustainability, and open up many new opportunities.

The OTA has had a home in Oxford University Computing Services, now named IT Services, since it was founded by Susan Hockey and Lou Burnard in 1976. You can read a little more on the history in the OTA at 40 post. The OTA archives, preserves and makes available digital texts and related resources, and is involved in a number of collaborations with other repositories, researchers around the world, and the CLARIN European Research Infrastructure. In recent times it has become increasing clear that these activities are nowadays a better fit for the mission and strategic plan of the university library, and so, the decision has been taken to move the OTA into a new partnership with the Bodleian Library, starting from the 1st November 2016.

The OTA will continue to offer the same services that it offers now, mainly texts for download for free for academic, educational and research use, from the website at http://ota.ox.ac.uk/, and will remain committed to the long-term preservation of digital literary and linguistic resources, and making them available for re-use. The OTA will work closely with Electronic Enlightenment – letters and lives online to help people find connections between primary texts and online information about people and social networks. With the backing of the extensive research data management facilities and services of the library, the OTA will be closer to the centre of exciting ongoing developments in digital preservation, data access, resource creation, and digital publishing in the University of Oxford.

In the short term, as far as users are concerned, there should be no visible differences in the service, apart maybe from some subtle changes to the branding. But in the longer term, watch out for lots of improvements and more links with digital collections in the library and beyond!

The Oxford Text Archive at 40

Originally posted at blogs.it.ox.ac.uk on August 4, 2016 by Martin Wynne. IT Services at the University of Oxford has decided to delete a large number of historical blogs, and this is one of a number of posts related to the Oxford Text Archive which are being re-published here, after being laboriously retrieved from the archive provided by the Wayback Machine.

The Oxford Text Archive (OTA) was founded by  Lou Burnard in 1976, and has been in continuous operation at the University of Oxford ever since.

The OTA is 40 years old

2016 is therefore our fortieth anniversary. Ten years ago we organized a one-day event to celebrate the thirtieth anniversary, and to look back and forwards. A summary of the day can be found one the OTA Thirtieth Birthday page.

The OTA is a repository for digital texts. The collection includes large numbers of digital editions, language corpora, and some more complex digital collections, such as databases, collections of data from websites, and images and audio data. Most of the items are the outputs of academic research projects, and one of the main roles of the OTA is to offer a route for digital outputs to be preserved, shared and reused beyond the end of fixed-term projects.

The OTA offers long-term preservation for its collections with secure storage in an HFS archive account. Accession of new items to the collection continues, although the OTA does not currently actively seek new accessions. Funded research projects from any institution are welcome to get in touch to discuss deposit of new works.

The OTA continues to participate in a number of collaborations. It is a centre in the CLARIN European Research Infrastructure Consortium, and is home to the coordination of CLARIN-UK. One of the results of this is that resources in the OTA can be discovered via the CLARIN Virtual Language Observatory. Users of the OTA can explore many of the texts online by clicking on the link to explore the text in Voyant Tools. Since all of the unrestricted resources are made available with stable and position URIs, other services can be used to process individual texts or batches of them.

tandards-conformant texts encoded in XML can be accessed in a variety of formats thanks to the OxGarage document conversion service – see for example this edition of Twelfth Night.

Below is a brief timeline of some of the key milestones in the past forty years:

  • 1976 Start of the Oxford Text Archive, based in Oxford University Computing Services (OUCS)
  • 1978 Oxford Concordance Programme launched by Susan Hockey
  • 1979 Kurzweil data entry machine (KDEM) installed in OUCS
  • 1987 Start of the Text Encoding Initiative (TEI)
  • 1989 Start of the project to build the British National Corpus
  • 1989 Computers in Teaching Initiative (CTI) Centre for Textual Studies
  • 1994 Launch of British National Corpus
  • 1995 First publication of TEI Guidelines
  • 1995 Humanities Computing Unit formed in OUCS
  • 1996 Start of Arts and Humanities Data Service
  • 2008 Start of Common Languages Resources and Technology Infrastructure (CLARIN)
  • 2008 End of Arts and Humanities Data Service
  • 2015 EEBO TCP texts available via OTA

Read more

Burnard, Lou (1988), ‘Report of Workshop on Text Encoding Guidelines’, Literary and Linguistic Computing 3: 131–3.

Burnard, Lou, and Harold Short (1996), An Arts and Humanities Data Service, JISC [http://www.ahds.ac.uk/about/documents/ahds-feasibility-study.pdf]

Burnard, Lou (undated), ‘Humanities Computing in Oxford: a Retrospective’ [http://users.ox.ac.uk/~lou/wip/hcu-obit.txt]

Hockey, Susan (2004), ‘The History of Humanities Computing’, in A Companion to Digital Humanities, ed. Susan Schreibman, Ray Siemens, John Unsworth. Oxford: Blackwell, [http://www.digitalhumanities.org/companion/]

Proud, Judith K. (1989). The Oxford Text Archive. London: British Library Research and Development Report.

Pajares Tosca, Susana  (2000), Report on the Humanities Computing Unit,[https://pendientedemigracion.ucm.es/info/especulo/hipertul/HCUreport/HCUeng.htm]

Oxford Text Archive – Downloads in 2015

Originally posted at blogs.it.ox.ac.uk on March 23, 2016 by Martin Wynne. IT Services at the University of Oxford has decided to delete a large number of historical blogs, and this is one of a number of posts related to the Oxford Text Archive which are being re-published here, after being laboriously retrieved from the archive provided by the Wayback Machine.

Analysis of the logs for downloads of resources from the Oxford Text Archive in the calendar year 2015 reveal a dramatic increase in usage. This increase can clearly be largely attributed to a number of factors, of which the most significant if the large number of additional texts from the Text Creation Partnership which became available via the OTA at midnight on the first day of the year, when it became legally possible to share them openly.

Most of the credit for this is due to the late Sebastian Rahtz, who did most of the work, ably assisted by James Cummings and Magdalena Turska. Sebastian, who passed away last week, has been instrumental in building and maintaining all of the technical infrastructure of the Oxford Text Archive in the past eight years or so. He will be sorely missed for this, and for the numerous other activities in which so many people became so reliant on him for his hard work, energy and brilliance.

Other factors aiding increased usage of the OTA include:

  • BNC free for download: at the start of 2014 the British National Corpus was made available for download for free, replacing the old system of paying for postal delivery of optical disks, and as word continues to spread about this development, so downloads continue to increase;
  • Freeing the texts: an ongoing programme of reassessing legacy data, and, where possible, removing access restrictions;
  • Higher visibility: resource discovery via the CLARIN Virtual Language Observatory, which aggregates OTA records and offers a new way for users to find the texts;
  • Shibbolization: a small and growing number of resources are available currently for UK users only, but soon to be opened Europe-wide thanks to the CLARIN and EduGAIN;
  • More digital research: demand grows as more users in the humanities start to engage in digital scholarship.

The grand total for the discrete downloads of resources from the Oxford Text Archive was 917077. Of these 180452 could be identified as originating from users in the University of Oxford, approximately 20%. Of the total downloads, more than 99.5% were direct downloads of resources made available at open URLs, the rest made up of the various resources where access restrictions require authorization.

The table below shows the top twenty of the downloads of all types:

Number of downloads ID (with link) Title
10659 2542 VOICE: Vienna-Oxford International Corpus of English
9181 5268 The history of the most noble Order of the Garter: Wherein is set forth an account of the town, castle, chappel, and college of Windsor; … To which is prefix’d, a discourse of knighthood in general, … Collected by Elias Ashmole, … The whole illustrated with proper sculptures.
6957 3016 The spy who came in from the cold
6382 4431 An account of the proceedings against the rebels, and other prisoners, tried before the Lord Chief Justice Jefferies: and other judges in the west of England, in 1685. for taking arms under the Duke of Monmouth. … To which is prefix’d, the Duke of Monmouth’s, the Earl of Argyle’s, and the Pretender’s declarations, that the reader may the better judge of the cause of the several rebellions.
5266 5314 The peerage of Scotland: containing an historical and genealogical account of the nobility of that kingdom, … collected from the public records, and ancient chartularies of this nation, … Illustrated with copper-plates. By Robert Douglas, Esq;.
4806 5301 The peerage of Scotland: containing an historical and genealogical account of the nobility of that Kingdom. … By George Crawfurd, Esq;.
4480 3549 The four seasons, and other poems. By James Thomson
4255 5299 The history and antiquities of the town and county of the town of Newcastle upon Tyne: including an account of the coal trade of that place and embellished with engraved views of the publick buildings, &c. … By John Brand, … [pt.1]
4146 3151 New York newspaper advertisements and news items: 1777-1779
3377 5244 The history of Newcastle upon Tyne: or, the ancient and present state of that town. By the late Henry Bourne, …
3175 3094 The Life of Charlotte Brontë by Elizabeth Gaskell
3154 5309 The history of English poetry: from the close of the eleventh to the commencement of the eighteenth century. To which are prefixed, two dissertations. … By Thomas Warton, … [pt.2]
2984 4835 The history and antiquities of the county palatine, of Durham: by William Hutchinson … [pt.2]
2948 4652 Miscellaneous works: of Edward Gibbon, Esquire. With memoirs of his life and writings, composed by himself: illustrated from his letters, with occasional notes and narrative, by John Lord Sheffield. In two volumes. … [pt.1]
2410 5308 The history of English poetry: from the close of the eleventh to the commencement of the eighteenth century. To which are prefixed, two dissertations. … By Thomas Warton, … [pt.1]
2321 4949 The history of the parishes of Whiteford, and Holywell
2252 4786 The history of Scotland from the accession of the House of Stuart to that of Mary. With appendixes of original papers. By John Pinkerton. In two volumes.: [pt.1]
2209 5730 Treasure Island by Robert Louis Stevenson
2118 4384 Charles and Charlotte: In two volumes. [pt.2]
2079 2554 British National Corpus, XML edition

And the table below shows the most popular items with access restrictions, which required an online application and manual authorization before they could be downloaded. There were 3681 of these downloads – over the year an average of ten per day which needed to be manually authorized by a member of staff.

Number of downloads ID (with link) Title
1857 2554 British National Corpus, XML edition
411 2553 British National Corpus, Baby edition
324 2539 British Academic Written English Corpus
220 2474 The Lancaster Corpus of Mandarin Chinese
93 1477 Helsinki corpus of English texts
81 2551 British National Corpus Sampler
79 2525 British Academic Spoken English corpus
62 0163 Complete corpus of Old English: the Toronto dictionary of Old English corpus / compiled by the University of Toronto Centre for Medieval Studies
53 2462 The York-Toronto-Helsinki Parsed Corpus of Old English prose (YCOE)
52 2510 Parsed Corpus of Early English Correspondence (PCEEC)
50 2507 A Corpus of English Dialogues 1560-1760 (CED)
31 2488 Dictionary of Old English Corpus in Electronic Form (DOEC)

There were 321 downloads from the experimental site hosted by the Oxford e-Research Centre, where users can obtain authorization for an instant download of a small number of resources (of which the BNC is the most popular) by authenticating with their institutional single sign-on. Only eighteen of these downloads were from the University of Oxford.

 

The Oxford Text Archive and the British National Corpus: an annual report (2014)

Originally posted at blogs.it.ox.ac.uk on September 22, 2014 by Martin Wynne. IT Services at the University of Oxford has decided to delete a large number of historical blogs, and this is one of a number of posts related to the Oxford Text Archive which are being re-published here, after being laboriously retrieved from the archive provided by the Wayback Machine.

The Oxford Text Archive continues to deliver open access to language resources to the academic community, via the website at http://ota.ox.ac.uk/. This year there were 5278 downloads of datasets from the OTA. An exciting development in this period was the arrival of the British National Corpus (BNC) in the OTA collection. This major reference work for the English language is now available from the OTA website, and was downloaded 397 times by researchers from around the world after it went online in January 2014. Two subsets of the corpus, BNC Baby and the BNC Sampler, are also available. Thousands of texts created as part of the Eighteenth Century Collections Online Text Creation Partnership (ECCO-TCP) are available via the OTA in high-quality XML format, and many thousands more will be available in 2015 from the Early English Books Online Text Creation Partnership (EEBO-TCP).

Two new services, introduced as the result of a collaboration with the Oxford e-Research Centre, offer new ways for users to access and use the literary and linguistic texts in the OTA. Users can download certain texts (including the BNC) without waiting for manual authorization of their requests by using their institutional single sign-on, thanks to Shibboleth federated access and identity management. At the moment, only users who are members of an institution which is part of the UK Access Management Federation can use this facility, but we are working to open it to cross-border access to more users throughout Europe via the CLARIN and EduGain federations. More than 300 instant downloads have been made already using this facility.

The second new service is BNCweb, a sophisticated online interface to the BNC, which allows researchers, teachers and language learners across the University to submit queries to identify and analyse distributions and patterns of usage in this large dataset of English speech and writing. In the coming year, we will start to implement an enhanced service offering access to more datasets via a common interface.

The OTA obtained certification as a CLARIN Centre in 2014, which confirms and strengthens its role as a key hub in the European research infrastructure. As a result of the collaboration with CLARIN, OTA resources can now be found via the Virtual Language Observatory, an online research portal, which offers access to electronic language resources held in repositories worldwide.

The development of these services, and the expertise in these areas, has enabled staff from IT Services to offer specialized teaching and support in digital methods to members of the University, including teaching on Masters course in English Language, induction sessions for new postgraduate students, and a course open to all in the IT Learning Programme on corpus linguistics.

Changes to the distribution of the British National Corpus

Originally posted at blogs.it.ox.ac.uk on January 13, 2014 by Martin Wynne. IT Services at the University of Oxford has decided to delete a large number of historical blogs, and this is one of a number of posts related to the Oxford Text Archive which are being re-published here, after being laboriously retrieved from the archive provided by the Wayback Machine.

In January 2014 there will be some changes in the way that the British National Corpus (BNC) is distributed.

It is now possible to download the British National Corpus at no cost from the Oxford Text Archive at the following URL:

http://www.ota.ox.ac.uk/desc/2554 [now http://hdl.handle.net/20.500.12024/2554 – updated October 2020]

BNC Baby, a 4-million word sample of the BNC is also available:

http://www.ota.ox.ac.uk/desc/2553 [now http://hdl.handle.net/20.500.12024/2553 – updated October 2020]

Click on the ‘apply for approval’ link to request a copy. The BNC continues to be subject to the same user licence conditions, which can be viewed at http://www.natcorp.ox.ac.uk/docs/licence.html. If you have already paid for permission to use the BNC, you should consider that this continues to be valid in perpetuity.

There is an even simpler download option if you have a login ID from a UK or eduGAIN Shibboleth identity provider (usually, this applies to all members of UK universities, and many European institutions). You can follow the links at the locations above to download the corpus directly without applying for approval. We hope that this facility will soon be extended to users from other countries who participate in the CLARIN Federation.

It will remain possible to order the BNC on disks from the University of Oxford until the end of March 2014, with the current administrative charges still applying, from the following URL:

http://www.oxforduniversitystores.co.uk/browse/category.asp?compid=1&modid=1&catid=1049

As part of this process, I have to announce that the University of Oxford can no longer offer any support for the XAIRA software, which has for many years been made available with the corpus. We have tried to offer support on a ‘best efforts’ basis in recent years, but we do not have the resources or expertise to help with the installation or use of XAIRA on the latest hardware and software. Users of XAIRA are encouraged to visit http://xaira.sourceforge.net/ and check out the forums and mailing lists which you will find there. The future of XAIRA depends on a committed user community, so please get involved if you have questions or can contribute expertise.

There are excellent services offering instant online access to the BNC, such as those listed at http://www.natcorp.ox.ac.uk/. I am convinced that there is still further potential for the integration and use of the corpus in online services and web applications. There are plans to integrate access to the BNC with the emerging CLARIN infrastructure, enabling a range of applications and web services to be used in conjunction with this and many other corpora. See https://www.clarin.eu/ for more details.

If you know of other ways of using the BNC, or have any more ideas about its future, I would welcome a discussion on this email list, or email me.

The Oxford Text Archive in 2013

Originally posted at blogs.it.ox.ac.uk on January 3, 2013 by Martin Wynne. IT Services at the University of Oxford has decided to delete a large number of historical blogs, and this is one of a number of posts related to the Oxford Text Archive which are being re-published here, after being laboriously retrieved from the archive provided by the Wayback Machine.

The New Year promises to be an exciting one for the Oxford Text Archive. As well as new accessions to the archive, new services and new collaborations, we plan to integrate the archive further into the new research data management services at the University of Oxford. This will involve working more closely with the Bodleian Libraries, who are embarking on a number of ambitious projects to serve the requirements of researchers for working with digital data.

The last year has seen the biggest ever expansion in the archive, with the accession of more than 2,000 texts from the Eighteenth Century Collections Online text creation partnership. These are made available under Creative Commons licences, another new venture for the OTA, and we plan to release future accessions with the relevant CC licence. These texts, along with all other XML resources, are now made available in a variety of formats, including popular ebook formats, converted automatically by the Oxgarage web service. We are planning future releases of Early English Books Online (EEBO) texts as they come into the public domain.

The Oxford Text Archive has taken over the management and distribution of the British National Corpus. We are not able to give support for the Xaira software, which continues as an open source project, but we continue to distribute copies of the corpus. In 2013 we will open a consultation on how to open access to the corpus with the corpus linguistics community and other stakeholders. We aim to make more widely available a BNCWeb service hosted by the National e-Infrastructure Service with secure authentication for users in educational establishments. The excellent online services listed at http://www.natcorp.ox.ac.uk/ continue to be available online.

The OTA also hopes that in 2013 we will be able to make more links with CLARIN infrastructure services and projects. OTA resources are already visible via the CLARIN Virtual Language Observatory, and we hope to participate in the federated content search demonstrator which is being built now. However, proper participation for service centres like the OTA, and for other institutions and individual researchers, does require that the UK funders and policymakers finally acknowledge the importance of the emerging European research infrastructure. Regretfully, attempts to engage research councils, JISC and the UK Access Management Federation in these processes continue to founder. Let’s hope for more progress in 2013, and that policy-makers start to act on their promises about building and promoting digital research infrastructure in the UK.

Discovering Babel – final outcomes

Discovering Babel – final outcomes
Posted on October 19, 2011 by Martin Wynne

Originally posted at blogs.it.ox.ac.uk on October 19, 2011 by Martin Wynne. IT Services at the University of Oxford has decided to delete a large number of historical blogs, and this is one of a number of posts related to the Oxford Text Archive which are being re-published here, after being laboriously retrieved from the archive provided by the Wayback Machine.

This is a summary of some of the key outcomes of the Discovering Babel project, with links to where you can find out more.

Next steps

For those of you looking to find electronic literary and linguistic resources please visit the Oxford Text Archive (OTA) and the CLARIN Virtual Language Observatory. The OTA will shortly relaunch with a new look and feel,and many new resources. The VLO is constantly improving and under development.

Those of you creating and sharing language resources, please join the CLARIN-UK mailing list. This list is a forum for creators and users of linguistic resources and tools to discuss how we can go forward to develop better facilities and shared services, and to gather user requirements.

Evidence of Re-use

The metadata that has been made available as part of the Discovering Babel project is being harvested by the CLARIN Virtual Language Observatory, and can be viewed on their portal. At the moment, we still have some performance issues with delivering the files via OAI-PMH, so there may only be a few records listed there, but we have identified the problem and will be fixing it in the next few days!

The work in Discovering Babel has contributed to an enhanced Oxford Text Archive, with more reliable and more easily discovered catalogue records, and with open access texts at persistent locations. This is designed to allow others to build services on top of our data, in a distributed environment. It has already helped to make possible the JSC-funded Great Writers project, which will, among other things, link to source texts in various formats, including epub, in the OTA.

The OTA is now also working together with the creators of Voyant at the University of Alberta, who have under development exactly the sort of tools that we imagined would bring our texts alive. Visit https://voyant-tools.org/ [link updated in 2019 -was http://voyeurtools.org/] and paste in the following URI to get a flavour of what will be possible:

https://ota.bodleian.ox.ac.uk/repository/xmlui/bitstream/handle/20.500.12024/3253/3253.xml [link updated in 2019 – was http://www.ota.ox.ac.uk/text/3253.xml]

You can see more about this text at http://www.ota.ox.ac.uk/desc/3253. At the beginning of 2011, texts from the OTA were only available on request for download. Already now, thanks in large part to Discovering Babel, we are seeing on our desktop the emergence of seamless access to distributed texts with remote tools in a service-oriented architecture.

Further collaborations with the National Grid Service in the UK to host language resources in the Cloud for UK researchers, with the development of a cross-repository search service for CLARIN, and shared services in Project Bamboo will all be underpinned in part by work done in Discovering Babel.

Skills needed for the project

The basic technical skills needed were for processing XML, e.g. XSLT 1.0 and 2.0, plus installation of modules in an Apache server, including Shibboleth access and identity management software. Various perl scripts were also deployed. Exactly how to do these things in this circumstances in which we were working were not things that anyone in the team had done before. For example, we had to read about and learn the specifications for the Open Archives Initiative Protocol for Metadata Harvesting, and the about the element set for describing language resources from the Open Language Archives Community, as well as the Shibboleth software. We were able to call on expertise in the Oxford University Computing Services for the fundamental technical areas and administrative procedures, and on experts in the CLARIN network across Europe for guidance on implementation in the specific scenarios for sharing language resources. Perhaps more than technical skills, knowledge of the work that was going on in our institution, nationally, and around Europe in the relevant areas were key to the success of the project.

Most significant lessons learned

  • Don’t build a digital silo: engage with infrastructure initiatives, such as CLARIN, and find out about recommendations for good practice in connecting resources, such as the Resource Discovery Task Force, and avoid building an online resource which is difficult to find and unconnected to other data and tools;
  • At the technical level, be flexible. This work touched on fast-changing fields, and we needed to be prepared to learn about new things, and to change the technological solutions which we deployed. This also meant planning for future change in order to make services sustainable;
  • Keep it simple: our successes were not the result of great leaps forward, or building complex and flashy front-ends and tools. Instead, we applied good practice in a systematic way in order to provide reliable services to underpin and fit into a shared services infrastructure. So simply providing crosswalks to Dublin Core from our metadata, and establishing an OAI-PMH service opened many doors. Putting the resource files at accessble URIs on the web allows new types of service to be developed, with much easier access and more powerful functionality.

Making your language resources discoverable and reusable

Originally posted at blogs.it.ox.ac.uk on August 31, 2011 by Ylva Berglund and Martin Wynne. IT Services at the University of Oxford has decided to delete a large number of historical blogs, and this is one of a number of posts related to the Oxford Text Archive which are being re-published here, after being laboriously retrieved from the archive provided by the Wayback Machine.

The JISC-funded Discovering Babel project has enabled the Oxford Text Archive to improve the ways in which we make our language resources available for users to find and use. Here we will explain some of the ways in which other resource creators might be able to follow in our footsteps.

Language resources are electronic collections of language data that can be used for language study and research, and are created in a number of contexts. Sometimes the main purpose of a project is to create a dataset, and in many other cases language resources are created as a part of or simply as the result of a larger project to investigate a particular aspect of language. Irrespective of why and how a resource is created, there is usually scope for making the resource available to others. This report will examine some simple ways in which creators of language resources can make it easier for others to find and reuse them.

Why make resources available?

There are many reasons why you may want to make your language resources available to others. It may be a requirement for your funding. It may be that you simply want to give something back to the community, and contribute to assisting the our accumulation of knowledge. Sharing your resources can also be a way of drawing attention to your work and getting recognition for what you are doing, and showing that it is having an impact on wider research goals. If you are able to show that something you have created is valuable to a larger group of users, this is likely to work in your favour in future grant applications, and when looking for to find collaborators, and support from the community.

Making language resources available is also a way of minimizing duplication of effort. If you have created a resource that others can use, they do not have to spend time and resources on creating their own resource.

Replicability of research results is another important issue. If others are to test and reproduce your results, or attempt to extend or refine them, then they will need to have access to the data, tools and methods which you used. Making resources available in this way is essential to testing, refining and building on research results, and is considered necessary for the verification of research findings and interpretations in many scientific domains.

Assuming that for one of the above reasons, or for another, you want others to know about and maybe also to reuse your language resources, what are the issues that you need to consider before sharing your resources? Thinking about the questions below should make your task easier and the sharing of the resource more effective.

Issues to consider when deciding whether and how to share your resources include:

  • How do you share?
    • Will you offer metadata, to help users find, evaluate and understand your resource?
    • Will you offer a service for users to access the resource (e.g. online access, or download option only)?
    • Will you deposit the resource in an archive or repository (instead of, or in addition to your own service)?
    • Or do you want to only share on request to users who get in touch with you?
  • Legal issues
    • Do you have the right to share the resources?
    • How do you protect your rights?
    • What kind of licence will you ask users to agree to?
  • Administrative and organizational issues
    • Do you have access to the resources needed to share your resources (server, staff, admin, user support, etc.)?
    • Who will be responsible for the service?
    • Are these reliable, sustainable and likely to be available in the long term?
  • Finding your users
    • How do users find your resource?
    • How can you make it easier for users to find/use your resource?
    • Can you support users?
  • Sustainability
    • How do you ensure you have the necessary resources/support/infrastructure to share your resource?
    • How do you ensure continuation of service?

Let’s now examine in more detail some of the issues relating to how to help users to find your language resources.

Making your resources discoverable

If you want to share your resources you have to make sure people know about them and can find them. The most effective way to do this is to make your metadata available to a portal which brings together information about where to find language resources in different locations. These exist in particular sub-domains (e.g. endangered languages, child language acquisition, learner language, sign language, for particular languages or language families, for historical periods, etc.), and there are a couple of more comprehensive initiatives: the Open Language Archives Community, and the CLARIN Virtual Language Observatory. Some questions to explore in order to market your resource effectively include:

  • Who are the potential users? Where do they currently look for resources?
  • What are the relevant mailing lists, conferences, and publications for your target audience?
  • Where in other domains, or sets of users, or geographical regions (beyond your immediate community or target audience) might you find interest in the resource?
  • If the resource is available online, or has a webpage associated with it, make sure you make it easy for search engines to find and index your page, for example by including the correct keywords in the website metadata (see Google’s guidelines for webmasters).

Once you have decided how and to whom you will make your resource descriptions available, it is necessary to provide the necessary information in the right formats. If you decide to deposit your resource in a repository, you will get some assistance in doing this. If you deposit with the Oxford Text Archive, you will need to fill in a deposit form, and then the repository staff will create an electronic metadata record. This will be transformed automatically to the correct formats for the online catalogue record, for OLAC and for CLARIN. If you want to create your own records, you can follow the guidelines provided by the different repositories. Some expertise in creating and manipulating XML documents will be required.

Social media

You may use social media forums such as blogs, twitter, facebook, dig, de.licious, and zotero, if you think that this might be a way to reach your potential users. It might prove to be a way to reach unexpected groups of users by reaching outside of the academy. Your funders might consider this to be a useful way to increase wider impact. It’s probably still not clear how appopriate and useful such methods are, and it’s a fast-changing field. But it doesn’t take much effort to tweet, announce things on facebook, make links on various services. Furthermore, writing blogs can be a good way to report your work to a wide variety of stakeholders and potential users.

The point of making your language resources discoverable is to facilitate the reuse of them by others. Let us now briefly examine some of the issues relating to how you can make this happen as effectively as possible, starting with avoiding any potential legal pitfalls.

Before you can share – a little more on legal issues

Before you make your resource available you have to make sure you have the right to share it. You may also want to look at what you can do you protect your rights (for example release the resource under a particular licence). You also need to consider if there are any restrictions on what users of the resource are allowed to do with it. Can they share it, add to it or develop it further? This could be specified in a user licence which you specify. Rights issues can be complex and often vary between different countries. If you have questions about what rights you have or what you need to do to have the right to share a resource, you may want to consult a legal representative for your area, for example the University lawyers or legal department.
If you are making the resource ‘freely available’, you may want to specify this with an open access licence. One way to encourage reuse is by making it simple for users is to see under what conditions a resource is available.

Creative Commons (CC) licences can be used as a “a simple, standardized way to grant copyright permissions to [your] creative work”. The CC licences can be used to specify that there are no restrictions whatsoever on re-use, or, for example, that people may only use the resource for non-commercial purposes or that they have to acknowledge the original creator when using it. It is also possible to specify that people may create derivatives (for example use part of the resource and/or add to it) and that such derivatives have to be made available under the same licence conditions. For more information about Creative Commons, please see http://creativecommons.org/.

Whatever rights or restrictions you assign to your resource you need to consider if the situation is likely change in the future. For example, will it be the case that restrictions can be lifted after a certain date? Or do you have permission to sue certain source texts only for a limited time? If so, you have to ensure that you can deal with this.

As well as considering the legal and ethical issues relating to making your language resources available, you should also certainly consider the licensing of the metadata associated with your resources. In order for users to be able to find, evaluate and reuse the resources, good descriptions of their nature and context are necessary. It is usual in the domains using language resources for this descriptions to be made freely available, but usually there is not a specific and clear statement of the terms under which they are made available. In order to avoid any restrictions on the free sharing of metadata, and to ensure that maximum use is to be made of it, it is better to assign a specific open access licence to all metadata records, such as ODC-PDDL or a Creative Commons licence.

In the case of the Oxford Text Archive, we found that because some of our resources are TEI XML documents, with the metadata embedded in the header of a single file which also contains the resource in the body, then it was necessary to apply a single licence to both metadata and data, and we have found that the Creative Commons best fulfills our needs for licensing the textual data (in most cases), we opted for that. In cases where we make just the metadata available, for example as a catalogue, and to metadata harvesters, we will apply the least restrictive possible Creative Commons licence, usually know as the ‘no copyright’ or ‘CC0′ licence (http://creativecommons.org/publicdomain/zero/1.0/).

How do you enable reuse of your language resources?

Depending on the nature of the resources at your disposal, you can opt to share your resources in various ways. Whatever way you choose, the key point is to ensure that the solution that you choose is not dependent on specific people, machines, projects, etc. which are likely to be transient, but rather that it is embedded in stable organizational set-up which is adequate for providing persistent service with high availability. The key questions to ask in deciding what sort of service to offer and how to provide it are the following:

  • Is what I am setting up sustainable?
  • Is the solution technically robust and not subject to discontinuation should current funding/staffing/equipment be cut
  • Who is responsible for the service?
  • Is this a person (named or defined by function) or an organisation (unit, department, institution)?
  • Who is responsible for the various bits of infrastructure on which the service depends?
  • Technology (server, scripts, physical server space, etc)
  • Human resources (server maintenance, user support)
  • What will the situation be in 1, or 2, or 5, or 10, years time?
  • What happens if you (or the person responsible for the service or part of it) leave or take on a different role?
  • What happens at the end of the current round of funding?
  • Will additional funding be needed/be available to continue the service?
  • Would it be better to look to move the service to another institutional home?

How can I make it easier for users to use the resources?

Let’s examine some of these options in a little more detail.

Distribution via email or on disk

A simple option, especially where small resources are concerned, is to simply send the resource to whoever requests it either as an email attachment (suitable for very small resources only) or on a CD or DVD.
This is only suitable for low-demand, small resources. You still have to consider legal issues and what provision there is for making the resource available also if you are not available personally to respond to requests. For distribution on disk there is also a cost – for the media and postage. What is more, the end user is left to their own devices when it comes to getting their resources connected to the relevant analysis tools. It can be tricky to work out which ones to use – will you be prepared to offer advice and technical support to users? Some will ask for it.

Online delivery

If you make your resource available online, you can opt to either make it available for download only (with some of the same problems identified above), or you may offer an online service where people can access and use via their web browser (for example a corpus with a search interface) . Now a new set of questions arise:

  • Who maintains the website?
  • Can the site handle the volumes of traffic, and the amount of processing required?
  • How will you know how many users have visited the site and downloaded your resource, or performed other operations? Do you need to report this to funders or other stakeholders?
  • Who will maintain the server and ensure that the service is available?
  • Will you offer a service level description, setting down exactly what you offer and under what terms?
  • Can you monitor the availability of the online services (i.e. tell if everything is up and working properly)?
  • Do you need to restrict access to certain classes of user? If so, how will you do this?
  • Do you need to recognize users so that they can come back to datasets and workflows that they have started to assemble on previous visits?
  • How will you deal with user support or queries (technical or about the resource/service)?Will it be available even if you leave the institution, or change your ISP?
  • Is the URL stable, or is it likely to change when the university re-designs its website (or the website host goes into administration)?
  • What happens when the technology behind the service needs updating/renewing (for example to work on different operating systems or in different browsers)?
  • Are you prepared to offer any guarantees of availability and persistence of service to users who might require stable datasets and tools for their research, or who may want to be able to come back and reproduce results at a later date?
  • How will users cite your datasets and services in their reports and publications?

Depositing your resource in an archive or repository

A lot of the issues arising from running your own web service can be avoided if you deposit your resources in a repository, which will deal with distribution, as well as perhaps offering long-term preservation, help with generating and sharing metadata, and connection with other tools and resources. So, you may also opt to deposit the resource in a repository. In deciding whether to do this, and whether a repository is appropriate, you may wish to consider:

  • Is there a cost associated? If so, is it a once-off, annual, etc? How will you pay ongoing fees after the end of the project?
  • What do you have to do to deposit (for example format of resource and metadata)?
  • How stable and reliable is the repository? How long is their funding likely to be continued?
  • Who knows about the repository? Is it known to potential users of your resource? Does it share metadata with relevant aggregators, and announce new deposits in appropriate forums?
  • Who has the right to use it? Is access restricted to members of particular institutions, associations, countries, etc.? Are there technical barriers which might exclude some sets of users?

There are several archives and repositories available. The Oxford Text Archive offers a service to deposit resources for a small administrative fee. This has the advantage of being a specialist archive for literary and linguistic resources, offering metadata to aggregators in this domain, and part of the emerging research infrastructure being developed by CLARIN. Other services exist for more specialist resource types, such as SCOTS at the University of Glasgow for Scottish and historical resources, CHILDES for language acquisition studies, ICAME for English language resources, and the Endangered Languages Archive at the School of Oriental and African Studies, University of London. Each is well embedded in their research communities, and so deposit with such an archive is an excellent way to reach particular sets of users.

There is also a lot of ongoing work in developing institutional repositories in Universities in the UK. While some of these are focussed exclusively on e-prints, some offer repository services for research data as well. Creators of resources should check on the facilities and services available in their institution (often based in the library or information services department), and deposit with your institutional repository it may be a viable option. This may be useful for raising your profile locally and as a secure storage solution. It is however highly unlikely to satisfy all of your needs. An institutional repository which aims to cater for research output of all types and for all disciplines cannot have specialist curation expertise in all areas, and will not, for example, know about all of the relevant metadata standards, best practice in digital preservation of language resources, or connection to relevant discipline-specific resource discovery services. Repositories will typically offer non-exclusive deposit agreements, which means that when you deposit your resources, you do not give up any of your rights. There is normally no barrier to you depositing your resource in numerous archives. This is effective for preservation purposes, although you may need to consider the impact that it might have in terms of version control (will the resource be updated, and how to you check that the latest version is available in all places?), and monitoring usage.

Furthermore, it is increasingly likely that federations of archives, with the possibilities of cross searching resources, and connecting disparate collections and tools. Beyond this, sophisticated virtual research environments will emerge allowing more operations, as well as collaborations between groups of scholars, and connections to publications and other outputs. It is likely to be the specialist repositories which are connected to this new infrastructure, and it is likely to become increasingly difficult for the individual scholar to connect up their resources without the assistance of the repository and infrastructure specialists.

Whichever of the options you choose, you can help to ensure that users can work with your resource as effectively as possible by considering offering the following facilities:

  • A full description of the resource, carefully crafted user guidelines, FAQ, instructions (preferably with screenshots);
  • Support for answering user queries;
  • A forum for users where they can discuss issues that come up. Make sure that you, or someone with good knowledge of using the resource, is available to respond to queries, in particular if the forum is new or under-used;
  • Make it easy for users to give appropriate accreditation to resource creators and access services, thereby also further promoting your resource and announcing its availability;
  • Make it clear what the title of the resource is, who the creator is and where it is found (at a persistent URL);
  • Make any licence restrictions clear (especially if your licence stipulates that the resource creator/owner should be attributed by any user);
  • Include on your website a sample citation/bibliography entry that users can use for reference;
  • If you are offering an online service, test the interface during development, and try to find some resources for ongoing development in response to user feedback.

In summary, you need to take as wide a view as possible about who the potential users are, how they will find the resources, how they might want to use them, and then to think about how the arrangements will continue in the future. Good luck!