Father Najeeb Michaeel examines a manuscript

Father Najeeb Michaeel is an Iraqi Christian priest who speaks Arabic, English, French, Aramaic and Syriac, not to mention being able to read Latin and Greek. In the garden of Zaytun library, Erbil I hear this gentle man tell me how his community of friars used to live in Mosul, a traditional centre for Christianity in Iraq, having the highest proportion of Assyrian Christians of all the Iraqi cities. Father Najeeb’s community has  had to leave Mosul due to persecution.  Later on during The House of Books workshop he gives us a presentation of the magnificent early Christian manuscripts they are digitising.  Over coffee he gives us a moving rendition of the ‘Our Father’ sung in Aramaic.  I wasn’t expecting to feel so moved by a  religion I have become increasingly frustrated by, and in Iraq.

Early Christian manuscript, Centre Numerique des Manuscrits Orientaux, Mosul, Iraq.

Iraq has often compared to a mosaic in terms of the diversity of its religious diversity.  Iraq is a Shia majority country and contains the sacred Shia cities of Najaf and Karbala. Most sources estimate that around 65% of Iraqis follow Shia Islam, and around 35% follow Sunni Islam. What is not so well known is that Christians have inhabited what is modern day Iraq for about 2,000 years. The person who is supposed to be respnsible for the transmission of Christianity in Iraq is St Thomas the Apostle. Assyrians (also called Syriacs and Chaldeans) most of whom are adherents of the Chaldean Catholic Church, Syriac Orthodox Church and the Assyrian Church of the East account for most of Iraq’s Christian population, along with Armenians.  Tariq Aziz was born to an Assyrian family and is a member of the Chaldean Catholic church. There are also small populations of Mandaeans, Shabaks, Yarsan and Yezidis. The Iraqi Jewish community, numbering around 150,000 in 1941, almost entirely left the country. There are also Gnostics in the form of Mandeans and sub sects thereof, Yazidis who believe in a god but have a blue peacock angel in their pantheon, and of course the Zoroastrians which the ancient Babylonians followed.

Early Christian manuscript, Centre Numerique des Manuscrits Orientaux, Mosul, Iraq.

Despite this diversity they share some things, one being religious persecution within Iraq.  Estimates for the numbers of Christians suggest a decline from 8–10% in the mid-20th century to 5% at the turn of the century, to 3% in 2008.  About 600,000 Iraqi Christians have fled to Syria, Jordan or other countries or relocated to Iraqi Kurdistan.which is also the traditional homeland of the Assyrian people. Those who remain are very aware of their minority status and the threat to their lives. Another shared thing they have in common is a respect for their heritage, both in terms of artefacts and records.  In the aftermath of the war in Iraq where countless manuscripts where destroyed, these communities are well aware of the importance of their documentary heritage. Each group have their own manuscript collection. In addition, an attempt by Saddam Hussein to centralise private collections of archives has also made them wary of any notion of centralisation/government control. This has resulted in many collections being hidden away and lost.

 It was in this context that I met Father Najeeb in Erbil. He was there to speak about his work at the Centre Numérique des Manuscrits Orientaux (CNMO) Mosul  and their work involving the digitisation of their  Iraqi Christian manuscripts. This is a small scale project conducted by the Dominican community in Kurdistan. Their move to Iraqi Kurdistan was due to their persecution in Mosul and the government in Kurdistan ensures as much they can the community’s safety.  Father Najeeb and his community are being helped by Father Columba Stewart, a Benedictine monk in St John’s monastery in Minnesota. Based at the the Hill Museum and Manuscript Library, the library began in the cold war, hoping to retain a record of Europe’s heritage in case the Soviets came.  Father Stewart’s  goal since 2003 has been to  digitise as many Eastern Christian manuscripts in the Middle East as possible, because  these manuscripts are endangered from a variety of causes. The main danger is the ethnic genocide which has afflicted Iraq but also neglect.

CD of a collection for donor

Father Najeeb’s aim is clear, to preserve and generate awareness and interest in these ancient Iraqi Christian manuscripts and protect their heritage from disaster and cultural genocide. The Centre actively collects and digitises collections from private donors, who in turn get a copy of their manuscripts on CD (see above). I met other members of the Christian community in Iraq who had found a safe haven in Ankawa a district of Erbil which we visited. They all speak several languages including Arabic and Syriac, an ancient language closely related to Aramaic.  While Erbil is known as a safe haven withing Iraq, driving around in Father Najeeb’s car  one realises that hanging a rosary bead from a rear view mirror is not a casual gesture a brave declaration of faith as is wearing the traditional clothes of a Catholic priest.

The manuscripts which Father Najeeb is digitising are extarordinarily beautiful, and all the more so considering their provenance, age and the number of destructive forces they have endured. They have of some similarity to other e arly Christian manuscripts of the age, as the religion spread as far east as Iraq and also to the wilds of western Ireland where scribes worked on  similar texts.  In some way digitisation and the digital age has brought people together again  to protect these texts and hopefully raise awareness of this and other vulnerable communities.

Young Iraqi Christian at Centre Numérique des Manuscrits Orientaux

We are very pleased to announce that the next Digital Preservation Training Programme will take place on the 14th -16th November 2011, in SOAS, London. The Digital Preservation Coalition is providing three scholarships to attend and applications are invited from DPC members.

Full details are available on the DPTP website.

Flying into Erbil at night

“What you destroy, we will rebuild, only better” – Slogan of Kurdish Peshmerga.

The garden I am standing in is so beautiful that I find it difficult to imagine that it was a former detention centre  operated by Saddam Hussain’s Ba’ath party, a place  of imprisonment and torture.  It is now a garden full of  flowers  and trees and in its centre rises the impressive Zaytun Library of Erbil.  This is no accident, the Kurdish Peshmerga vowed that all these sites would be rebuilt this way once Saddam’s regime ended and the people would reclaim such poisoned land for purposes such as libraries and gardens. Erbil or or Hawler as it is called by locals like much of Iraq has seen a lot of history pass its way, Alexander the Great sorted out the Persian King Darius near here and the citadel of Erbil is the oldest inhabited city in the world and a soon to be UNESCO heritage site.

Erbil citadel

But let’s take a step back. What is a London based Corkonian doing in the middle of former detention centre/ garden in Iraqi Kurdistan? This  region in the north is the ancestral homelands of the Kurds – the oft persecuted minority in Iraq.  The Kurds constitute the largest minority without a homeland. I was at the library as part of the third House of Books workshop funded by the EU and UNESCO and run by a Humanitarian NGO called Un Ponte Per…. You can read more about their involvement here. It is the last in a series of workshops which has been looking at digitisation of texts and their preservation and its main partner is the Iraq National Library and Archives (INLA). Many institutes from Iraq joined us including the National Museum of Iraq, Centre Numérique des Manuscrits Orientaux and other projects. From the Middle East the  National Library of Jordan and the  American University of Beirut also took part. My story with the INLA goes back to 2004 when I managed after some effort to persuade Dr Saad Eskander to write his  diary about his day to day life reconstructing the destroyed library in Baghdad.

Iraq National Library and Archives

The INLA was destroyed during and post war in 2003.  Of its 417,000 books, 2,618 periodicals dating from the late Ottoman era to modern times, and a collection of 4,412 rare books and manuscripts, an estimated 60 percent of its total archival materials, 25 percent of its books, newspapers, rare books, and most of its historical photographs and maps were destroyed in various ways. This was not just a loss for Iraq, it was a catastrophe for the world on many levels.

In 2011, Dr Eskander has built up a library few could ever have imagined possible. The INLA now leads the way in much best practise in librarianship for both traditional and digital material in Iraq.  There are also many digitisation projects being hatched in Iraq and around the Middle East, big and small and the drive to join up previously physically separated collections of journals, manuscripts, photographs is strong. The potential of the digital has long been recognised as a powerful means of disseminating information in the Middle East. The workshop  is trying to make projects understand that digitisation has a catch, and that is preservation or how to ensure that access is maintained over time. This idea was well introduced in Jordan during sessions such as ”Digitisation is not preservation’ and other catchy titles, and the event this time saw progression and developments since Jordan. In fact it seemed that projects had reassessed their approach to digitisation. Some would admit that where previously they were just scanning, (blogs passim) they revisited their projects in light of what they had learned in Jordan. Some  added technical metadata which they had not done at point of digitisation, others looked at having master copies as well as access copies of their digitised content and kept in different locations.  Policies were reconsidered. Storage solutions were considered. All steps in the right direction. The value of having 2 workshops in a year with a lot of the same people in both proved useful  as there seemed to be a lot of consolidation and desire to demonstrate improvements between one meeting and the next.

Workshop overview

The workshop was kicked off by UPP, and some short contributions by the EU’s representative in Kurdistan, Hala Al Sharifa , followed by the UNESCO programme officer for Kurdistan, Sami Al Khoji who is clearly dedicated to his role and the revitalisation and distribution by digital means of information about Iraq’s cultural assets. Kanan Mufti who is  director general for antiquities in the western Kurdish region in Erbil and who resided in the ancient citadel reflected on the importance of documentary heritage for Kurdistan and declared the protection of documents as a priority for them.  Presentations the INLA showed us they have been digitising and accessioning digital content since 2008 and we heard about their  plan to develop an Iraq digital library making its materials available to all Iraqis online.  The material will cover all aspects of Iraqi life and society and all forms of document. The INLA sees as vital its contribution to intellectual and scientific research in Iraq and also endeavours to support programmes which will end illiteracy in Iraq. The INLA has embarked on training programmes in many aspects of digital library management. They have even managed to send 2 people to attend the Digital Preservation Training Programme, thanks to the British Council and BISI.

The National Library of Jordan spoke about the need for standardisation in the region in relation to cataloguing and indexing. The  American University of Beirut ‘s presentation lead on nicely as Basma Chebani reflected  on Arabic ontologies and the need for authority files in the Arabic speaking world. My 2nd session on metadata fitted in well here and we did a nice hands on exercise, working with the group of 30 through Arabic and Kurdish, it seemed to hit the mark and the right level. The translator also did a great job helping me, he is now a metadata expert!

Father Najeeb of the Dominican Order in Iraq who spoke movingly about their ancient texts and manuscripts and their ongoing digitisation. A small project with great ambitions from a community constantly under threat. I plan to write more about this in another blog.

Early Christian manuscript, Centre Numerique des Manuscrits Orientaux, Mosul, Iraq.

Many copyright issues arose, in particular that of forgeries in the traditional manuscript environment. It seems that a lot of illegal copying takes place and it is difficult to contain. Issues such as translation were interesting. During my session on digitisation and preservation the Iraqi born but Aberystwyth-reared translator ran out of his booth proclaiming, ‘What is a plug in?!’ To which everyone loudly  had an opinion in return.

Translator in action

The translator explains metadata

I led the concluding session, working with the group about recommendations for next steps arising from the workshop. This proved interesting, considering I was again working with a translator with a group who spoke Kurdish, Arabic and Syriac, not to mention an Arabic English keyboard! Getting individuals to think by themselves and for themselves about what they would accomplish on their return to work is good in a big group. They then worked as teams of 4 to consider what the next steps would be in Iraq for libraries and archives.  The key issues were networking, the establishment of a national network or syndicate of librarians and archivists and information specialists involved in libraries and archives is deemed of great importance. The group also want to consolidate the information from the workshops and ensure that people are not reinventing the wheel in terms of developing best practise. The establishment of education and training programmes in all aspects of librarianship and archives is also vital.

Metadata matters but other things do too.

What was also key and hidden beneath the discussion of texts and metadata and the like was that this was a   moment or space away from the day to day. These people work in circumstances which we cannot begin to comprehend. Just to come to the workshop involved endless checkpoints and danger. Life is unsafe and violent. Civil society as we know it here in the UK is almost non existent in most of Iraq. Electricity cuts are regular, resulting in me being stuck in a lift fo 5 minutes. This is not the same for Erbil on the whole but most colleagues came from Baghdad, Mosul and other regions which are the news for tragic reasons.

The value of this little group of archivists and librarians from different ethnic and religious groups is more than just about metadata and file formats (as important as they are) but about bringing disparate groups of people together with a view to the flowering of a new nation where religious and ethnic difference no longer matter, where censorship doesn;t exist, where ideas flow freely once again. This is the vision of the INLA director Dr Saad Eskander.  It is not an easy vision in a divided society where sectarianism is rife. However it is not so unusual to consider the power of libraries as a social phenomenom and yet we seem to treat them purely as an informational phenomenon. The House of Books demonstrates that it works on at least 2 levels

Iraq as everyone knows has  a violent history of occupation and war, however during periods of serenity, the emergence of civilisations who have made numerous extraordinary contributions to the history of civilisation, these include  writing, and the concept of zero or  sifr to name but a few.  Original texts survive from the era of Babylonian mathematics. On day 1 of archives school baby archivists learn that the Babylonians wrote on tablets of unbaked clay, using cuneiform writing. The symbols were pressed into soft clay tablets with the slanted edge of a stylus and so had a wedge-shaped appearance (and hence the name cuneiform). Experts studying these learned that the Babylonians had developed the concept of sifr or zero.

Cunieform from Babylonian times: Top: 64 (1 sixty + 4 ones) bottom: 3604 (1 sixty2 + 0 sixty + 4 ones)

Sifr is also used in Arabic to denote a clean slate, a blank page. In Iraq hard work has begun of the rebuilding from scratch  of a rich cultural heritage of Iraq for the future.  I am glad that the preservation of digital heritage of Iraq is a part of this.  More later!

Map of Erbil

I have been invited to the 3rd House of Books workshop organised by Un Ponte Per…. It will take place in Erbil, Kurdistan and is under the auspices of the National Library and Archives of Iraq. The workshop is looking at best practise in relation to digitisation and preservation, follwing on on topics discussed at the last workshop. Our keen and alert followers will notice that this project has already been on DA blog. I will be posting when I get back about this workshop.

If you haven’t seen it, can I recommend Kristen Snawder’s recent post on the Library of Congress Digital Preservation blog, Digitization is different than digital preservation. Kristen reiterates familiar points about the long-term commitment necessary for serious digital preservation, contrasted with the quick hit of a scanning project. “In the hurry to meet user expectations, institutions may scan large quantities of materials without having a solid plan for preserving the digital images into the future.”

However another recent find on the Web compels me to make an additional point, namely that we might do equally well to differentiate between scanning and digitisation. Anyone can set to work with a scanner and create a bunch of digital images – but that barely scratches the surface of what I think we should be expecting of a digitisation project in 2011.

First and foremost, we need metadata: the more the merrier, but something at least. Even if we expect to come back later and polish it up (once the images can be browsed and examined on screen). In the absence of any established metadata profiles for a project, at least try to cover as many Dublin Core elements as possible – title, creator, date, subject/keywords… Images, in particular, may prove tricky or time-consuming to find again, especially once there are thousands of them on a disk. We should probably keep the metadata in a database, and perhaps additionally store metadata with the objects. This can be as XML or plain text files stored alongside the digital images, or embedded in the files we create (many common file formats – TIFF, JPEG, MPEG, PDF – support metadata embedding, and there are many free tools available to help).

There is yet more, though, that we should be doing, particularly when we are scanning text-based objects (articles, books, magazines, reports, etc). Most importantly, we really should try and extract the text from the image if possible. ^[1]

My recent web find was the teaching blog of Dr Toine Bogers at the Royal School of Library and Information Science (RSLIS) in Copenhagen, Denmark. One fascinating post describes a Lab Session exercise, From OCR To NER, a set of comparatively simple command-line processes to get the most out of a scanned-text project.

Toine’s post walks us through the process. Once the article is scanned, we should apply some OCR. The exercise goes further and also describes the use of tools to clean up and spell-check the resulting OCR’d text. This will, at the very least, result in a separate text file, hopefully containing a fairly accurate version of the article text. Finally, the cleaned-up text can be submited to a Named Entity Recognition service. Toine’s exercise uses NER tools at University of Illinois. (We’ve been using similar functionality provided by OpenCalais and GATE for our AIM25 Open Metadata project.)

Why do all this? The most important, instant, result of this is that we can now easily index our article for full-text searching – in a local repository system, such as EPrints or DSpace provide – and of course by Google. None of this is possible if we leave the scanned image as just that – an image.

Another  side-effect of any successful OCR outcome, is that the text is now free to be re-flowed. This means that we might consider sharing it with users in a variety of forms enhancing usability and accessibility.

It’s important not to confuse preservation formats with formats for access and dissemination. You probably will have your scanned image masters in TIFF, RAW, JPEG2000, PostScript, SVG. None of these are likely to be of much use to your users over the Web. Not only are the formats not widely supported by Web browsers, but most users probably don’t need or want your master image. If it’s a high-resolution scan of a 100 page book, they might be looking at 100Mb download, or worse – slow to load, and probably slow to render and navigate.

Time taken thinking what formats will give users the best experience is time well spent. What platforms might they want to use now and in the foreseeable future? It’s less than 18 months since Kindle3 made e-book readers affordable, and the Ipad made them sexy. E-books look and function very impressively on both platforms (albeit in different ways): for an overview of some of the benefits of the EPUB format, see Martin Fenner’s post Beyond The PDF… is EPUB. PDF outputs may yet have their uses, if users can at least search for text within them. The point is that only with properly digitised text, do these kinds of accessibility options become possible.

Even image collections can also be disseminated as E-books – nice offline items some users might care to flick through on their tablet computers, possibly even smartphones. I’ve demonstrated how we can create OJS XML from EPrints XML on-the-fly with XSLT: since EPUB and Mobi/Kindle are XML-based formats, we should be able to do something similar to create e-books using repository APIs. Also, by using appropriately sized images in dissemination formats (Ipad screen is 1024x768px; Iphone4 is 960x640px) we can not only ship our users a sensibly-sized download, we can protect any capital we may have in the master images, without having to resort to ugly tricks like watermarking. (Giving users full-size, high-res images with embedded watermarks seems to me the worst of all worlds.)

Therefore I’d suggest that, in order to get the best out of a digitisation project, consider what would you like to see at the end of the project – and, more importantly, what would give your users the best experience, or even win you new users? Ask around, do some tests, with users if possible, and get an idea how they want to use the materials and how they will get the best out of them. Maybe there are comparable projects and systems that you admire, with features you’d like to be available for your collection. What about in five or ten years’ time: will your current project outputs help or hinder longer term accessibility goals?

This kind of vision of is essential. Without some conception of the end result, how the materials will be used and managed most effectively, all the scanning in the world isn’t going to amount to a successful digitisation project.

This survey is part of BlogForever, an EU-funded collaborative project that ULCC collaborates through the Digital Archives department.

The results of the survey, available at the end of the summer, will help to develop digital preservation, management and dissemination facilities for weblogs. Hence, we are keen to gather information about the content, context and usage patterns of current weblogs, so we could identify blogs users’ views on their long-term preservation, management, analysis, access and use. If you would like to take part on the survey please use the following link:

For a naïve non-tech archivist like me, this felt like a new way of looking at and thinking about digital data. The principle seems to be that it’s possible to capture an image of an entire disk, and then perform stepped analysis on all the data contained within that image. Most models of digital preservation, management or curation that I’m familiar with tend to focus on the file as the unit which we must identify, migrate, catalogue or preserve; here the target is different, and we’re dealing with a whole wodge of related digital spew that includes files, technical metadata, automated logs, and lots of system elements from the registry.

Digital forensics is often applied to “personal archives”, such as the disk image of a single author / researcher; or is used by commercial organisations seeking to prove something in a court of law; or in extreme cases in tracking down evidence of a crime. As presented by Leighton John, a number of characteristics of the forensic approach struck me as being quite resonant with the aims of digital preservation and/or electronic records management:

• No change: you have to capture information without changing it; demonstrate you have not changed it; and analyse it without changing it.
• Audit trail: forensics requires working out what happened in the past from traces of information, then having to defend it in a public place.
• It looks at everything: context is provided by examination of all data, records, files and system components in a case.
• It tells a story: forensics is the creation of a narrative, with lots of details and evidence.

This struck me as overlapping with aspects of records management, and descriptive of many features of what we’d expect an EDRMS to deliver. The “No Change” dimension is a crucial part of records management when it comes to the authenticity of the record, and also aligns with the requirements of BIP 0008 on legal admissibility of digital documents. EDRMS systems offer audit trailing as standard, but it’s mainly the audit trail of what happens to file objects once in the EDRMS itself. I wondered if digital forensics could indeed build this detailed technical “narrative” of record creation activity through analysis of an individual user’s disk image, and I asked a bemused question along these lines.

Naturally, forensics alone wouldn’t satisfy all the MOREQ requirements for records management, and we’d still need our mechanisms for managing retention and disposal. Some of the more obvious barriers to success would include:

• Privacy and rights: what member of staff working anywhere would welcome having their entire profile analysed? At the very least they’d feel pretty sensitive about it! But maybe it need not be all or nothing; there’s plenty of “rich data” in any given system and a lot of it could be associated with individual files without capturing the entire data image.
• The costs of doing it: I’m not sure but I get the impression it’s neither cheap nor trivial to carry out a forensic analysis of a disc image; it requires specialist technical skills, a dedicated environment, there are numerous steps to the process, and it takes a long time to complete.
• Potential loss of context, a problem which may be partially resolved by a specialist form of search and analysis, which can be used as means of indexing one’s forensics findings.

This is all just speculation of course, but I do like the mental exercise of thinking about digital data in ways that don’t involve mapping to paper analogues, a place where I think we have been trapped for too long.

Question: What do alcohol and a VLE discussion forum have in common?

Clue: you’ll have to read to the bottom of this article to find out.

History Spot at Institute of Historial Research ( IHR) looks very good.  IHR is one of the many jewels in the crown of ten member Institutes of the School of Advanced Study, part of the University of London. It offers many training opportunities for historians and the like. History SPOT is a platform and an opportunity seized by IHR to launch and disseminate IHR activities.  It will significantly increase and enhance access to the materials by anyone who needs these skills/information/knowledge.  What we saw so far, looks intelligently designed with the user at the heart of the model.

On Tuesday we gathered to hear more about HistorySPOT and also from the Open University about their issues with VLEs, they knowing more than a little about such matters.  History SPOT was launched in March 2011 thanks to a grant from an anonymous donor to develop the IHR’s online presence (especially to widen participation of its research seminar programmes).  It was soon decided that they would use this donation to develop two core areas of the IHR’s remit.  The first was to widen knowledge of and participation in their internationally renowned research seminars through the use of podcasting, live streamed events, and vodcasting (video).  The second was to provide for the first time, free access to content from our research training modules and courses.

Dr Matt Phillpott gave us an overview of presently available online training courses relating to the Humanities and the concluded that there were not that many. The Open University is the closest we have and of course  iTunes is an option but this is not perfect as it neglects much contextual information about the courses on offer.  The OU also ran us through their various offerings, including the online tasters they provide (all in Moodle) which used to be called Learning Space but now is called Open Learn.

So all in all putting courses into a VLE is a good thing right? Well, of course! At IHR it increases access to the wealth of material being built up at IHR and hopefully adds to IHR’s income stream which will enable it to deliver more high quality and useful stuff to put into History Spot.

An interesting discussion arose about the age cohorts of people using our VLEs, many young people will be very happy with a purely online experience but the quite high drop out rate at OU reflects that not all can motivate themselves in this way. Agreement was reached that it is very difficult to teach about the meaning of something as well as it is to train via a VLE.

Of course putting our courses on a VLE is not as simple as it seems. Transferring the pedagogical experience to the environment of a VLE is not straightforward.  To recreate an experience of the  group dynamic which occurs in a training course or that magic of the learning moment, we can resort to forums, discussion groups and the like. However these will only go so far and the more is not the merrier. In fact the more forums the less likely people are to contribute and more problems can arise.

A positive side to online discussions is that people who normally wouldn’t contribute in a face to face course may be more inclined to do so in a VLE.  However, and here is the pitfall, the experience of many hardened and wise VLE providers will indicate that all these courses and discussion groups need  constant moderation. People can write things and express themselves in ways online which they wouldn’t dream of doing face to face. Like alcohol it seems to have an emboldening effect and people will say things which whether intentional or not can be misconstrued and cause great upset. This has to be decided on when planning and designing the course.  The moderator will need to be alert to this at all times while running their course, and so the lesson is again ‘Moderation in everything’.

CERN Computer Centre

CERN Computer Racks

Last week I was in a very welcoming Geneva, exactly in the European Organization for Nuclear Research (CERN) to meet other partners working on BlogForever and to have several Invenio workshops. I felt very lucky to be in the hub of such an organization and to see how many young international students are getting the opportunity to be in the forefront of high physics research.

The Globe at CERN

CERN is home of the world’s biggest and most powerful particle accelerator, the Large Hadron Collider (LHC). This machine is installed in a 27 km circumference tunnel. The LHC records around 15 petabytes per year. All the data is stored in their vast computer centre, where open access and sharing has been the driving principle since their foundation in 1954 and an inspirational environment for the Web to be born there.

Invenio’s workshops showed us that their electronic document management system is robust and versatile, targets the management of more than 1.2 million documents and it can be used in 19 different languages. Its content is clean and complete. In just their High Energy Physics domain, they have around 700 collections and approximately 20K queries a day. As well Invenio is used for special programs like the UNESCO funded digital repos in Africa and EU funded projects like D4Science and OpenAIRE.

Me at CERN

In the case of BlogForever and Invenio, plenty of work to be done by the Invenio Team at the User and Document Service Group. At the moment, they have more than 30 readily available Python modules that can be adapted to the case of preserving huge amount of blogs. From the point of view of my work with repositories as part of the Digital Archives and Repositories Team at ULCC, I was inspired by Invenio’s advance search engines; indexing and ranking methods.

In a more personal level, if you are ever crossing the border between France and Switzerland near Geneva, get Tram 18 and hop off at CERN to see their Microcosm and Globe exhibitions.

Chris Awre, William Nixon, Rory McNicholl at the Texas Longhorns stadium

The conference kicked off with a keynote from Jim Jagielski of the Apache Software Foundation, describing the history and organisation behind Apache and its projects. It was observed by some in the Twitter backchannel that the talk could as easily have been from 2001 as 2011, but for all that it was a worthwhile reminder that, in all our efforts, we stand on the shoulders of the giants who created and maintain the infrastructure of the Web and the Internet. And also that many our endeavours benefit from a little more dedication and commitment than you can usually squeeze between 9-to-5.

The closing keynote was by repositories stalwart Clifford Lynch, who managed to touch on so many perennial repository themes, I won’t attempt to summarise them. There is a handy anthology of tweets about his talk on Storify.

In between were plenty of presentations and opportunities to meet friends old and new from the United States of Repoland – some we have worked with, some we would like to work with, and many with challenging ideas and insights into the many facets of working with repositories.

The OR conference hops back and forth across the Atlantic (I’ve previously attended OR08 in Southampton, OR09 in Atlanta and OR10 in Madrid). Unfortunately when the conference is held Stateside, the representation of the EPrints community tends to be noticeably smaller. Not that there aren’t EPrints users in the USA (we were particularly pleased to meet the team from Cal Tech Library, very happy users and advocates of EPrints), but the distribution of software platforms is significantly different from Europe in general, and the UK in particular (if you are interested in such things, you can check out the statistics at OpenDOAR). And of course travel logistics (and costs) are non-trivial. Luckily Rory and I had been saving our prize money from last year’s Developer Challenge!

While it eluded me in previous years, I think at last I am starting to grasp at least some of the salient points of the thing they call Duraspace (launched, if I recall, in Atlanta)! I’m certainly hoping to find time to take my free Duracloud trial. However other aspects still remain opaque to me. At one panel discussion about the prospects for implementing DSpace over Fedora (or Fedora under DSpace, depending which way up you look at it), I was surprised to hear a description of ongoing DSpace-Fedora alignment efforts as “more about the journey than the destination”. An enviable luxury: for the time being we need tangible outcomes for our repositories and customers, and that’s one reason why we’ll be sticking with EPrints for the foreseeable future.

Personal highlights for me are described elsewhere: the Developer Challenge, which we enjoyed immensely, and Changing Platforms the talk that I presented with Imma Subirats, of the UN Food & Agricultural Organisation, where we discussed migrating between repository platforms. Rory also had a chance to meet developers from Yale, who had worked on the other end of the SOAS-Yale Islamic Manuscripts collaboration, and show off some of his work for the SOAS repository. We were also hugely appreciative of the generosity of the EPrints team, who kept us generally amused and amazed, and kindly included us in their group dinner on the last evening.

Adrian Stevenson's got them all-night late bar open repository blues...

The facilities at UT’s AT&T Conference Centre were outstanding, as was the surrounding campus generally, including the Longhorns football stadium (with its insanely massive west stand) where the conference dinner was held. Austin has far more attractions than we could see in such a short time, and it is an impressive and vibrant city, from the spectacular grandeur of the Texas state capitol, to the noisy entertainment on 6th Street, where virtually every bar has some kind of rock or blues band playing. We were most impressed by UKOLN’s Adrian Stevenson who jammed on a borrowed guitar with the blues band in the Blue Moon bar at 2am. As if that wasn’t enough, our visit also coincided with the massive Republic Of Texas biker rally – an insanely noisy procession of up to 50,000 bikers through the main streets of the city. Our ears won’t forget OR11 in a hurry.

In addition to her work at FAO, Imma is Chief Executive for the E-LIS repository, a major international and multi-lingual repository of articles about Library and Information Science. E-LIS has operated since 2003 on EPrints, but last year migrated to DSpace, because CILEA in Italy, who generously donate support and hosting, now focuses exclusively on working with DSpace. The E-LIS migration has been largely successful, however a number of EPrints features on which the E-LIS editors and users depended, have been difficult to replicate in DSpace, or had to be put on ice. This is no reflection on the specialists at CILEA, but perhaps indicative of more profound differences between EPrints and DSpace, that aren’t always reflected in the usual comparisons of repository platforms, such as the otherwise informative JISC RSP Repository Software survey.

ULCC of course has just completed a repository migration from DSpace to EPrints for the School of Advanced Study. Our motivation was in many respects the same as that of CILEA – our expertise lies firmly in the EPrints camp. But I think the outcomes for our end-user community are more demonstrably positive: in fact I don’t think there’s a single feature of the new SAS-Space-on-EPrints that isn’t a major improvement over its previous incarnation.

Migration of metadata and data (at least from DSpace to EPrints) presented few issues (that weren’t of my own making!) – export, transform, import. Here the similarities between the models of the two platforms was extremely valuable. But we did encounter other significant differences, some of which are set out in more detail below.

Richard presenting Changing Platforms at OR11

Issues in EPrints

Perhaps the most significant issue we encountered with re-implementing SAS-Space on EPrints was the absence of built-in support for Handle persistent identifiers. Handle support comes out-of-the-box with DSpace, but not with EPrints, so the choice we faced was between re-implementing Handle support, or dropping it. We chose the latter, since the benefits of Handles to a relatively small IR like SAS-Space were not obvious, and so it was hard to justify the extra cost and effort. By ensuring that items kept the same ID when migrated from DSpace to EPrints, and implementing a simple rewrite rule, we have ensured that Handle URIs created while DSpace was operational continue to point to the same item – but for items added since EPrints went live, no new Handle URIs are coined.

(Shortly after we returned from OR11, an extended discussion broke out on Twitter, amongst several well-respected gentlemen in our field, about the benefits of using Handles. A considerable amount of scepticism was expressed about their usefulness.)

Issues in DSpace

Imma described some workflow issues encountered with the new implementation of her repository. The E-LIS team is accustomed to a very flexible EPrints-based workflow that allows items to have their workflow status changed quite freely. DSpace, by contrast, has a unidirectional workflow model, so that items cannot (for example) be reverted from Live to Pending, if some kind of error is spotted, but effectively need to be deleted and resubmitted. This is obviously a significant divergence between the superficially similar repository platforms.

Another example Imma gave of a perplexing feature of the default DSpace UI is the button on each abstract page that says “View Full Item Record”. It leads to a rather intimidating web page displaying the item metadata as Qualified Dublin Core. It’s not a very attractive display, nor is it actually a “data” rendering of the metadata (as you would get by explicitly choosing to Export As XML, or from some new-fangled Linked Data features). It’s not clear why this view would be of interest to general users of the repository: why is it there?

At OR11 I talked to several people working with DSpace, and all agreed that there’s room for improvement in the default Web UI. In some cases they have completely reimplemented the web templates. It’s also worth noting that the page layout in the default JSP UI is entirely implemented using HTML tables, and doesn’t pass W3C validation. For a Web application that’s nearly 10 years old, this is disappointing. (The alternative Manakin XML UI implements an attractive vision of UI abstraction using XSLT, but reports suggest that configuring/maintaining it is not for the faint-hearted.)

Quite a few Web design infelicities are perpetrated in the default Community, Collection and Abstract page templates. (During the conference, many of us enjoyed and applauded Simeon Warner’s timely rant,”Don’t bold the field name”.) Of course we can change them – it’s Open Source, isn’t it? – but is it unreasonable to expect default Web templates that are at least potentially usable as is? Of course the natural and reasonable response of the DSpace community is to ask that we report the issue as a bug or feature-request to the development team. Or fix it ourselves and share the fix. But where an absent feature is really important to a user (by which I probably mean a repository manager), then the choice faced is between “getting by” until it’s implemented in the core distribution, or doing it themselves (which probably means hiring a specialist developer to implement it for them).

Out-of-the-box

At the DSpace User Group meeting in 2007, I described how we considered that, back in 2005, DSpace offered a better “out-of-the-box” experience than EPrints. I never thought it was anything to write home about – in fact I remember being disappointed by the very UI issues I’ve described above – but to my untrained eye it did seem better than EPrints, at the time. But, as I’ve mentioned elsewhere, EPrints has improved remarkably since.

Of course a lot of people we admire have proved that you can create impressive repository systems using DSpace. It performs and provides a lot of essential repository functionality. Its Lucene search engine is certainly better than anything EPrints currently offers. But I’m still surprised how much more work seems to be necessary to make a DSpace installation as readily useful and usable as EPrints, and this seems to represent considerable additional cost in setting up DSpace.

I’ve heard it sometimes argued – in both EPrints and DSpace camps – that Repository setup shouldn’t be too easy, lest repository managers get in a mess and endanger the integrity of their system. In my opinion, as developers and solution providers, our job is to provide as many features and tools as possible to enable Repository Managers to manage their collections effectively and easily – not act as as gatekeepers to their systems and data.

By way of contrast, we have recently supported the Institute of Education (IOE) in setting up an EPrints repository of UK government publications, and we were pleased to see the repository manager called on us very little, other than to answer some questions and apply a few small configuration changes. The experience with SAS-Space has also confirmed to me that EPrints now has strong out-of-the-box appeal, and a rich set of features available through the Web UI, that enable a reasonably confident repository manager to get to work without needing to initiate a major technical project.

In the current climate, of straitened library budgets, this could make a considerable difference to the viability of a repository startup project. For a growing number of libraries and information services – not least at smaller research institutions, or in developing countries – that could be the difference between having a repository, or not.

Excitement at the OR11 Developer Challenge Show-and-Tell (Photo by @sparrowbarley)

An event that asked developers to demonstrate the Future of Repositories can only be considered a great success when it receives entries that include:

• Multiple real-time examples of using “Repositories As A Service (RaaS)”, not only exchanging data but also sharing sophisticated functionality between EPrints and DSpace – and even including an Android application
• A tool for bundling and depositing a whole raft of research related outputs from the Web via RDF
• A tactile repository search interface with dynamic search suggestions, specifically designed for tablets and smartphones
• A complete gesture and voice-driven system for depositing and searching in repositories

All these – and other great entries too – were achieved in a couple of days’ work during the course of the conference, for the annual OR Developer Challenge, and presented at a packed Show-and-Tell session on Thursday afternoon (true, there was free beer).

Stuart Lewis’s team were worthy winners with their RaaS project, particularly as they showed a genuine commitment to a cross-platform approach – something which, sensibly, backgrounds the individual software platforms, that often receive too much attention, and focuses on the Repository as an application and entity in its own right.

We were also really pleased to see a prize go to Patrick McSweeney and Matt Taylor. And enjoyed seeing Dave Tarrant stealing the show (again) with his live demonstration of using a Microsoft Xbox Kinect to submit items to a repository.

Our own entry may not have won, but several people liked it, and you may see more of it in future. For the second year running, the Developer Challenge was a great opportunity for Rory and me to concentrate on an idea that we’ve been kicking around, without having found a home for it in existing work (yet). This was true for our Semantic Metadata popup tools that won the challenge with last year.

This year we set about achieving our  longstanding desire to take the very tactile and dynamic search interface that Rory created for the MERLIN project, and turn it into a touchscreen app, for smartphones and tablets.  The results were pretty effective.

The MERLIN Mobile interface ("TEXAS") as demonstrated at OR11

The MERLIN interface on LASSO is quite complex, but at the heart of it is the tag cloud of related terms that the Termine text-mining suggests. This always looked like it might be good on a touchscreen, so we stripped it down, rearranged and tweaked it to make it viable on an Ipad screen. If you’ve got an Ipad, you can try it out by pointing your Safari browser at http://is.gd/texasweb. (It will work on desktop browsers too, but it looks best on a portrait oriented screen.)

If you’ve got an Android device, you can even download an app-based version of it from http://is.gd/texasapp. (It’s a bit cramped in a smartphone display, but still essentially working.)

If you’ve got neither we created this page to give you a rough idea what it looks like.

It’s worth mentioning that this is a real, live, working application: enter a search term (“logical positivism”, “climate change”, “Jeremy Bentham”,  …) and it searches over the full-text corpus of all articles in University of London Open Access repositories (SHERPA-LEAP consortium members), and makes suggestions about additional or alternative search terms, based on the results of the text-mining analysis of the articles

The hackathon approach of working closely together to create something quickly worked well again: Rory hacks, and I test and review each time he hits ‘Save’! All very agile and iterative.

In honor of our hosts in Austin, we decided to call the new interface Touchscreen Enhanced Cross-Search with Augmented Serendipity – or TEXAS for short.

Kudos to Mahendra Mahey, Peter Sefton and the DevCSI project for putting (and keeping) it all together, and to the awesome panel of judges (even though they didn’t pick us)!

Fáilte go Sráid na Banrighinn

The whole purpose of education is to turn mirrors into windows.  ~Sydney J. Harris

Football and digital preservation seem an unlikely combination but on May 15th Ed and I arrived to a slightly damp Glasgow celebrating en masse the end of the football season, to deliver our DPTP north of the border.  There, I am afraid the comparison between football and digital curation ends but it is worth noting that on Monday our students did not consist, in the main of bleary eyed Scots.  As I am myself half Scottish I was looking forward to hearing about the digital state of play in my mother’s country but curiously we had a very international group, many had travelled far from places such as the European Central Bank,  European Commission, PRONI,  and near  such as from the National Records Scotland (who clearly cannot get enough of a good thing as we were with them 4 weeks previous).  The group was very creative who seemed to work well together. I know  I know, we say this a lot but this group seemed to shine for many reasons.  They were keen to work together and shout out about their ideas right from the start. And though being a disparate international group they managed to work together very neatly and develop some great case studies for us.

Internationally (and sometimes nationally!) language has always been a barrier  for communication, but  it is also an enabler  and this is again where we see something such as the OAIS being a useful way of empowering people to be able to communicate with each other, and other professionals.  The DPTP rarely (and rightly so) has students just from the traditional information management world (libraries/archives/etc).  Many people now being redeployed from different parts of their organisation and as such are often unfamilair with the idea/concept of an archive or indeed of the traditional notion of the lifecycle of a document/record. This is where OAIS is very valuable due to the way it which it expresses the way a digital object should be ideally kept and uses a language describe this.

In many of our case studies with OAIS, we often find that many organisations are actually mapping very well to the OAIS which indicates that it reflects  good and real practise when it comes to managing our digital repository. Our job in the DPTP is to take someone who has never heard of the OAIS (suprisingly more often than not) and by the end of the three days have them fluent in the OAIS concepts both through listening and through application to their own environments.  Our feedback from the course has proven that this is a job well done….DPTP abú!

ULCC Digital Archives & Repositories Team, May 2011

Please introduce yourself
We are ULCC’s Digital Archives & Repositories Team, and you can generally find us in the basement of Senate House along with the rest of ULCC. We are five:

• Ed Pinsent and Patricia Sleeman, our digital archivists, provide digital preservation training and consultancy.
• Rory McNicholl is our lead developer and repository systems manager.
• Silvia Arango-Docio is project officer for Web archiving projects and repository support.
• Richard Davis is team leader, and actively contributes to all of the team’s projects, as well as pursuing new and interesting opportunities.

What does your team do?
Our department was created in 1997 to develop the National Digital Archive of Datasets (NDAD), and operate it for The National Archives. NDAD was originally a joint project of ULCC and Senate House Libraries, providing a specialised cataloguing and preservation service for government databases.

Since then we’ve worked on many digital archive and library projects. The NDAD service ceased in 2010, but we continue working on innovative projects for born-digital and digitised records, developing repositories and related information systems for education and research. We provide specialist training and consultancy for the HE and cultural heritage sectors through the highly-acclaimed Digital Preservation Training Programme (DPTP).

Our current or recent partners and customers include most of the University’s colleges and institutes, the British Library, the Parliamentary Archives, the JISC, the UK Web Archiving Consortium, and the European Commission. We work closely with the Digital Preservation Coalition (DPC), and local groups such as the AIM25 archives network and theSHERPA-LEAP consortium of University of London repository managers.

Tell us one thing about your work that colleagues might not know.
Digital preservation issues might seem esoteric, but they are fundamental to every electronic and computer-based activity – in other words, pretty much everything in business, education and research. Understanding the best ways to manage, describe and preserve all our electronic information – from a single email, to complex Web sites, or collections of digital photos or videos – is an essential 21st century skill.

What aspect of your work gives you the most satisfaction?
We are very lucky to work closely with information professionals from around the UK and internationally. Patricia has recently delivered preservation workshops at the National Library of Jordan; Ed and Silvia are working on Web archiving projects for the JISC and the EU; Richard and Rory won the annual JISC Developer Challenge at last year’s Open Repositories conference in Madrid.

What is the most challenging part of your job?
Keeping up-to-date with constantly changing ICT landscape in education, libraries and archives. Electronic information systems evolve continually, as do the tools and methods for managing them. Luckily there is a wealth of current information on the Web, if you know where to look! We actively share our thoughts and experiences through our long-running Digital Archives Blog and on Twitter.

Name one thing that would make your working environment better.
Sometimes it seems we work with everyone except colleagues in Senate House! It would be great to work more with colleagues at the Central University, and bring more of our accumulated experience to bear on some of the University’s information and records management challenges.

If you could meet anyone (dead or alive) who would it be and why?
We’re not going to agree on this any time soon. Suggestions so far include Sofia Loren, Hypatia of Alexandria, Richard Brautigan and Pete Townshend.

Name three hobbies team members pursue – the more unusual the better.
Most of us have small children, so hobbies take a back seat. Ed is an accomplished artist, writer, broadcaster, musician and samizdat publisher. Patricia is about to learn how to stain glass before starting Arabic lessons for the third time.

If, as a group, you stormed the musical charts, what genre of music would that be in, what would be your stage name and who would be your lead vocalist?
We could be a psychedelic beat group called the Dates Of Creation. Our lead singer would have to be Patricia.

Recently the British Library team approached all the users of the shared WCT tool. It seems that the curators involved in UKWA have been using these metadata fields slightly differently and the BL team have initiated a project to move towards more consistency. The project will involve deciding on definitive interpretations of how to use these fields, followed by a process of cleaning up legacy data stored in the system. Some of it is potentially useful, some of it not so useful; some is legacy from the earlier PANDAS phase of the project, mostly not needed, or entered into the wrong field.

As noted, a lot of this metadata is mainly to do with selection and evaluation decisions, curation information such as changes in status of the site, and as such it’s never been exposed anywhere except within WCT. However, one descriptive field will eventually end up exposed on the UKWA live site, and provide us cataloguer types an opportunity to describe the resources in more detail. It will appear on the Title Entry Page (TEP) for each instance.

I welcome any move towards exposing more descriptive metadata on the UKWA public site. I have always taken the view that the phrase which currently appears alongside a Title “The live site may provide more information” is not really very helpful in the context of a web archive, for three reasons. (1) we don’t want our users clicking away from UKWA; (2) the link to the live site may be dead by now, and; (3) as archivists and curators, I feel strongly that we are the ones who should be providing that “more information” in the shape of a catalogue description of some kind.

The JISC project sites, as a collection, have high evidentiary value as stages in development of very specific tools, services and activities that benefit the UK Higher Education community. The sites by themselves don’t always explain their history or intentions; I would argue that a lot of rich contextual detail about the reasons these sites existed (the JISC programme under which they were developed, the dates, the staff involved, the themes, the outputs) would help interpret the collection to the users and make it more intelligible.

A glimpse of Petra, Jordan

‘Machine replication of human functions, like reading, is an ancient dream’ *

One of the many topics discussed in the House of Books project in Amman was the issue of OCR and Arabic texts. Optical character recognition or OCR has become one of the most successful applications of technology in the field of pattern recognition and artificial intelligence. It is now a necessary step in the transition from analogue text to the elctronic world, particularly due to the quantity of information now available in the electronic age as it enables rapdi searching and scanning. In the last five decades, machine reading of text has grown from a dream to reality.

Software for OCR is now almost 100% sucessful for Roman scripts. Middle Eastern library content however, particularly for Arabic and other non-Roman language materials, poses special challenges to the creation of digital repositories of arabic texts.  Arabic, being a diacritic language has many characters (letters) which have exactly the same form, and are distinguished only by the position of various dots over, above, or inside the main character block. This  poses special difficulty for OCR, as dots can be ignored by software as speckling or error, or even removed. Most institutions digitising Arabic manuscripts use Sakhr OCR software, but it does not seem to pick up the intricacies of Arabic script. What to do?

Some arabic fonts http://university.arabsbook.com/

It seems that if prepared well the Sakhr recognition software package has the capability to recognize generic Arabic fonts (called Naskh or Kūfī) with a fair degree of accuracy. However the software has to be taught to recognize any peculiarities or unusual characteristics in the font of the scanned volume in question. This is extremely time consuming and requires technical expertise. Also it is taken for granted in such a process that the font will be more or less consistent throughout any given volume; in many cases the hand can change in any manuscript so I imagine it would need to be reinstructed according to each section where the hand or the font changes. In addition the quality of any OCR depends on the quality of the original scanned file.  Also not everyone wants to use generic fonts, think of how much we like to personalise our own? Another headache for Sakhr.

Our group in Amman as a whole expressed frustration with  Sakhr and really hoped that it could in some way be generally instructed to recognise characters which it consistently fails to pick up. We felt sure that it will be solved soon and I personally cannot imagine that the military have not got a  solution up their sleeve  about this considering the politics of the world these days.

Interestingly in terms of resources discovery, Google Scholar does not allow searching in Arabic, while it allows for searching of both Japanese and Chinese scholarly texts. Surely as complex for an OCR piece of software to recgnise as Arabic?   This means that any texts written in Arabic cannot be accessed, which means that scholarship in Arabic is not being picked up by one of  the biggest and widest search engines for scholarly literature. Why such an oversight by Google scholar? I have contacted them and have yet to find out!

This of course brought home the real need for more collaboration between libraries and archives involved in digitsiation projects in the Middle East itself. There are many  projects based in North America such as Ameel and in the UK such as SOAS (which our own Repository folk in DART have been working on!) which unify and make available digital resources from the Middle East.  There was also an interesting JISC study with the University of Exeter about user requirements for digitised resources in Islamic studies. These are of course a western approach to arabic material, albeit in their own collections. It often also is concerned with transalations of arabic texts to greek or Latin as was the norm.

The issue of OCR and its sucess rate for non Roman fonts also raises questions about the power of the digital and askes the question that if OCR canot serve one of the great languages – Arabic,  how many minority languages which are also very diacritic are not being served well by the OCR sofwtare available.  The result of this must a tip in the balance of available reserach material in favour of texts in Roman script and sees an imbalance in what is being made available online.

Baghdad at night, 2011

There is a need for the countries which created this material to work together on such projects. Many very interesting and topical projects  to do with the emergence of which were being proposed in Amman relating to digitisation and working together to track missing journals as well as trying to avoid duplicating efforts.

So how to do this? Several libraries attending our workshop in Amman highlighted the necessity to coordinate the effort for  Arabic texts digitization in order to avoid duplication, share best practices and develop common standards, index and software. To enable this  a decision was made to work on  developing new cultural cooperation interventions for digitisation in the Middle East; to fund-raise for this and to set up groups in a social network (facebook, linkedin) including all the participants from the House of Books project. Importantly further workshops will be run to encourage this cooperation and hopefully see strides being made in cooperation and digitisation of arabic texts in the Middle East.

* http://www.nr.no/~eikvil/OCR.pdf

**Thanks to Qaiss Hatef  Saeed of the Iraq National Library and Archives for his help.

Newspapers in National Library of Jordan

Baghdad, 2003 -  when Domenico Chirico, Director of Un Ponte Per… first asked various organisations for support and resources for the reconstruction of the Iraq National Library and Archives (INLA) destroyed during the Iraq invasion and occupation, he was met with cries of bemusement and disbelief:

‘Why worry about books and archives when we have lives to save?!’.

Some if not many  struggled to understand that a library in Baghdad could be a priority during such horrific times.  But they had not yet met Dr Saad Eskander, Director of the INLA…but that is another story which has been told.

Cairo,  2011 and the ‘Spring revolution’ is happening in the Middle East.  In Egypt almost as soon as unrest broke out, two public libraries in Cairo were  burnt to the ground however cheeringly in Alexandria, news came of staff and citizens forming a human chain around the Biblioteca Alexandrina (much supported by Susan Mubarack) and another around the National Mususem of Egypt to protect it, though it wasn’t so lucky. These stories as told via Twitter cited what had happened in Iraq. Iraq, sadly is not alone in having suffered such destruction to its cultural property. These attacks,  fire, pillage, looting to order and just plain old theft  do more than just destroy a building or some documents. They are attacks on civil society, denying Iraqi people  access to engage in democracy, to access information as well as to their cultural memory.

Amman, 2011 – In light of all this, it was with great interest that I agreed to participate in the ‘The House of Books/ Dar El Kataub’ workshop run by the aforementioned Domenico Chirico’s NGO UPP and UNESCO.   Its purpose was to work on challenges facing the digitisation of Arabic texts.  I was there to look at the preservation aspect of digitisation, a shock to some as it is often thought that digitisation is itself a preservation strategy! So there was work to be done…

William Kopycki and I leading a session

We had a wide range of people from Iraq, Lebanon, Qatar/Australia, Jordan, Egypt/USA, Italy and the United Kingdom.  From an digital archivist’s point of view (and in my view the right one),  the projects represented presented us with the gamut of activities which are now present in a digital library or archive, from the development of impressives copyright legisalation in Jordan to a start to finish overview of a project digitsing  journals relating to the 19th centroy arab cultural renaissance Al-Nahda. We also heard about training and devloping infrastructures for digital object management in Iraq, as well as an overview of the current project in the INLA to reconstitute many of its collections through digitisation. We had a good deal of course about preservation from myself and Giovanni Bergamin from the University of Florence and Maurizio Messina.

My work consisted partly of leading the group in a consolidation of ideas discussed during day one.  A very important goal of this workshop is collaboration and as a result we wanted the group to think individually and in groups of four about why they saw the need to collaborate and then to tell us how they would approach collaboration.  We ended up with great points which reinforced the absolute need for collaboration in terms of standardisation, best practise, resource discovery, lobbying and important networking opportunities across the arab speaking world.

I also was there to speak and work with the group a lot about digital preservation and digitisation. This is an area which can be neglected often as many considered digitisation to be simply about simply delivering access to materials.  In addition these digital objects are considered digital surrogates and little consideration is given to their preservation as the analogue copies are available. It is important to consider preservation of these digital surrogates over time at the point of their creation.  Do we really want to invest the time and money again in their re-digitisation? I very much doubt it! However unless consideration is given to their long term sustainability this is what will happen, data loss or re-digitisation. This is time and money few of course can afford to spend.

Touching of some of the issues we covered in terms of preservation we looked at planning digitisation projects in light of preservation and their sustainability and explored the main points to consider for preservation of digitised content, drawing a lot from our very popular report on digitisation and preservation produced by DPC/ULCC and PORTICO. I reinforced the real need for honest sharing and for sharing failures as well as successes, we have all had them so we can only progress through acknowledgement of both.

Looking at the 1923 magazine on women’s issues called  Layla, we explored in a session what characteristics we would like to preserve over time.  Many things which we assume will be kept in the paper based world, have to planned for in detail  in the digital world, not much if anything can be left to chance.

Workshop over!

All speakers were extremely interesting individually and collectively as they gave an outsider like me the overview of the digital library/archives world in this part of the Middle East.  Qaiss Hatef  Saeed of the INLA spoke about the motivation of the INLA to establish a digital library  due to the loss of their holdings in the library during and post conflict.  Much of the library is being rebuilt literally from ashes by means of digital content,  and popular holdings need to be digitised to give as Quaiss said ‘ books a rest’ from handling.

Qaiss also proposed that in 100 years hard copy books will no longer exist and everything will be digital. Fighting talk! But  digital libraries however will only survive as long as we invest in their sustainability.  Digital resources have great power in terms of access but they are also very vulneable in terms of long term sustainabilty.  As such action needs to be taken an now to stop bit rot and technology obsolesence being the next threat to access to information which is now increasingly digital.

*Note: The House of Books workshop was picked up by 16  newspapers in Jordan and Iraq. These include  articles in the Baghdad press, The Jordan Times and even the Iraq National Congress.

Over the last year or so we’ve installed and configured (in some cases reconfigured) the IRStats package for several of the LEAP repositories, including those hosted by ULCC. It seemed a good moment to share a few thoughts about the process of getting “all statted up” with EPrints.

By default, and without any further action, IRStats provides a kind of smorgasbord control panel, demonstrating the many optional graphs, charts and list available. You can see an example on our own ULCC Publications repository.

More recently we’ve seen growing demand among repository managers to share data on downloads with both their depositors and users at large. It’s really important for repository managers to select carefully which statistics views they actually want or need to display – we can only suggest things we think might work. Once you’ve decided on the views you want, we can look at the most effective ways to display them: and this is why I’ve been having fun souping up some of the displays already offered by IRstats.

The first display we’ve been working on is the Statistics digest. These are common enough and we’ve used the example of UCL Discovery repository as the basis of work for both SAS-Space and SOAS institutional repository.

The second approach has been to re-style the IRstats “dashboard” view to lay the graphs on top of each other and then use some Javascript to handle the tabbed navigation. This seemed a more elegant approach than inserting lots of charts in the abstract page itself (as, for example, at ECS EPrints). I’ve used this display technique to display statistics for individual eprints for the School of Pharmacy, as well as SAS and SOAS.

The tabbed display of graphs and tables was also combined with a ‘modal box’ display that keeps the height of page the same (for example on this Abstract page at SOAS. At the bottom of the Abstract page I’ve added a statistics section showing the number full-text downloads, and a link that displays detailed stats in an overlaid box.

This method doesn’t just work for individual items, but can be used on other datasets in too. For example, on SAS-Space we have added it to the bottom of their Collection browse pages, so that at the bottom of each Collection view there is an opportunity to view download statistics for that collection as a whole.

Additionally in SAS-Space, since it is a repository for a number of discrete institutes, there was a requirement for institutional editors to have access to their own institute’s statistics. To achieve this, I allowed access to a constrained version of the IRStats control panel for editor-users who had the appropriate editorial permissions for the institute in question. (Unless you are a SAS-Space editor, you won’t be able to access this.)

Which statistics views to insert as tabs is the decision of the repository manager. Views we’ve used include:

• Monthly downloads
• Daily downloads
• Unique visitors
• Referrers
• Search Engines
• Top 10 items downloaded (only for a Collection, Repository or Division)
• Top 10 search terms

From a technical point-of-view, we will have to review these configurations when we upgrade to EPrints version 3.3, possibly later in the year (if it’s released!!), in conjunction with our VM infrastructure migration, and start doing things with EPStats rather than IRStats. But we now have an effective framework for adding statistics quickly to any EPrints installation.

During the ArchivePress project at ULCC, we briefly considered the data and metadata generally made available with blogs and blog posts. As ArchivePress focused on the representations of blogs in newsfeeds, we examined the metadata that is generated in common, and exposed in the newsfeeds of three of the most common blog platforms, WordPress, Blogger and TypePad. Blogger and Typepad prefer the Atom newsfeed format; WordPress (particularly WordPress.com) prefers RSS (though it can be made to publish Atom feeds too). This analysis was done, about a year ago, things may have changed, but here is a summary of what we found.

For each Blog, the following core information is available in the feeds:

For each Post, we established that the following core information is available in the newsfeeds:

One interesting point we noted was that neither Blogger nor Typepad published a link to a Comments Feed for each post. This made our work on ArchivePress more difficult since it was predicated on being able to easily identify the Comments feed for each post, and harvest new Comments as they were published. Obviously for blogs generated other than by WordPress, this was not going to be so easy. (Our ace developer Emanuele found some workarounds, but that’s another story.)

I think this offers us an interesting overview of the core of standard, structured blog data and metadata, in three of the leading blog platforms. This is the data structure and metadata profile that is maintained in blog databases, in one of its native forms, and I’d expect it to be present in all blog platforms, since it arguably represents the essence of blogs. I hope this will be useful background when considering the core models for data and metadata handling that will be developed for BlogForever.

Is there a particular blog or blogger you would like to see included in the BlogForever survey? We invite you to use this form to nominate them, and we will try to ensure that the blog is reviewed or the blogger contacted to participate in our survey.

Susan Corrigall(NRS), Patricia Sleeman & Ed Pinsent (ULCC).

We gave a 2 day in house workshop to the National Archives of Scotland last week.  As Ed Pinsent has noted in his post about legal admissability, our timing was quite interesting; we arrived on the Monday the week after NAS had merged with the General Register Office, to become  the National Records of Scotland.  Interesting times for all and while we couldn’t have known in 2010 what a pivotal day it would be for NAS, we hoped  our training session on digital preservation would be in some way timely.

The 20 staff attending were very varied in their reponsibility and roles in the NAS/NRS, however wisely it was felt that all needed to have a entry level awareness of the issues relating to digital preservation.   NAS has been engaged for some time in the management of its digital resources and it is looking at cost effective ways of doing so.  Staff are being skilled up in relation to digital preservation, in this way NAS are ‘maximising their current resource potential’, to use a buzz word.

Working with a group from one organisation in situ is quite different for us as normally the DPTP is an open course which sees a variety of delegate hailing from many very varied organsiations. As a result we cover many topics to give a good grounding in what we feel are the most important aspects fo digital preservation, while trying to accomodate varying levels of expertise and knowledge.  This type of course is sucessful in many ways – it can often lift spirits, particularly for peope working in a solitary environment to see how much in common they have in terms of digital preservation and its challenges. The social aspect of the course is important as people recognise that digital preservation affects all communities involved in any kind of management of digital content in the long term.  A lot of what we aspire to do is build confidence in people to tackle the issue.

Working in an organisation who has commissioned the DPTP is quite a different experience. We see the coming together of a group already known to each other to a greater or lesser degree. However very often they come from different sections and this ‘time out’ together should be of value in many ways:  as a collective learning opportunity, socialisation and in a time of change an opportunity for supporting each other.

We also get the chance during a class project to investigate a particular issue of note for the organisation.  This hopefully enables the group to leave with a finished product.  In the ‘open to all’ DPTP we want our group to leave with a better understanding of everything but more precisely to have gone some way regarding the implmentation of theory to real life work situatsions through our classwork.  To leave the course with soemthign immediately implementable within the workplace we feel is important. Our class session at NAS/NRS remains confidential but we thoroughly enjoyed it and felt that it translated a lot of what was hitherto theory into a an enhancement of an existing practise to enable improved digital object management.

Interesting times for all and I feel perhaps it was timely that NAS are skilling up their staff to embark on digital preservation now.  Digital preservation is a concern for all government departments but who can manage it? Archivists as a profession have a good set of tools and concepts from their profession which translate well to digital preservation.  In fact OAIS tells many archivists nothing new – they are doing most of it already in the analogue world.

Thus archivists hold one major key on our large key set which we need to unlock digital preservation.

At an atomic level, a “blog” comprises “blog posts”, which are continually added to the blog corpus: that is the dynamic essence of a blog, and distinguishes it from old-fashioned, largely static Websites and hypertexts in which little content changed between major update iterations, which process was probably more akin to “publishing a new edition” in the world of non-digital publications.

The blog also displays, as part of its frame, other graphical and functional elements (sidebars, widgets, “blogrolls”, etc) which may themselves contain dynamically updated, constantly changing information. These can be added, removed, amended and rearranged at will by the blog author/editor. Blog posts that were “published” in the context of one set of framing elements, will persist through subsequent versions of that framework.

Similarly with design (layout, colours, mastheads, etc), though the persistence tends to be longer, the informal nature of blogs means that these may be easily changed by the blog editor/author, and are thus more volatile than a typical “corporate” website. Again, blog posts may persist, unchanged in themselves, through many iterations of the blog site design and layout.

A simple view of blog elements and their temporal relationship

This very simplified visualisations suggests where we might start conceptualising key elements of a blog. It indicates that they iterate over time, but in the cases of Design, Posts and Widgets (as we’ll call them for brevity), according to independent schedules. While Posts and Comments persist in the online view of a blog, designs and widget arrangements are overwritten.

With my earlier ArchivePress project we deliberately overlooked preservation of the blog’s framing elements, and (given the much smaller scope of that project) established an acceptable rationale for doing so. The challenge for BlogForever is to find a solution to  precisely these issues. Unless we were simply to adopt the snapshot approach of Heritrix-based web archiving initiatives (e.g. Wayback/archive.org, UK Web Archive), we need to ensure the BlogForever repository supports a degree of granularity that can capture, describe and preserve atomic blog objects in a way that reflects the particular interdependencies, in order to understand and preserve them authentically, and permit the many possible authentic and valid “time slice” views and analyses that users of the archive will need.

(I appreciate, by the way that these objects themselves are compound objects, so not strictly “atomic”: but the same is also true of atoms, as our CERN colleagues can attest!)

Our brief was to introduce digital preservation topics to traditional archivists. Among other things I was asked to deliver a module on BIP0008 and the legal admissibility of electronic records. In preparing this I discovered that the standard is very comprehensive, requiring written policies for classes of records that are in scope, a stated “duty of care” and written procedures for what staff should be doing, an extremely meticulous and well-documented methodology for the production of scanned versions of records, plus a reliable IT framework in which this can work. And of course, an audit trail marked with date and timestamps generated at every possible link in the chain of custody. That’s a lot of boxes to tick, but the payoff is a scanned document (or born-digital record) which is regarded in the eyes of the law as an authentic unaltered copy, hence legally admissible.

My personal take on the standard it goes a long way to satisfying the requirements of auditors who seem to take legal admissibility to rather extreme lengths; to my mind, any good EDRMS or records management system ought to be providing enough audit trails to keep them happy. Nonetheless the archivists at Scotland seem to have a requirement not only to observe this standard, but also to ensure they continue the chain of custody from current records management into archival storage. In other words, legally admissible records must also be legally admissible archives; and any preservation actions performed on these objects while in digital custody must not compromise that authenticity.

This may also have been reflected in their IT manager’s interest in use of the checksum and file format validation tools which can be used in the repository; he seemed to be wondering if such tools could verify authenticity, perhaps by indicating whether a file had been tampered with at any stage between leaving the EDRM system and entering the archival repository. It’s an interesting line of thought, and an approach that would probably involve a heightened degree of audit trailing for any organisation that wanted to work this way. My off-the-cuff contribution to this discussion on the day involved something about providing evidence that “best effort” had been made, but I suppose the real proof would be in a court of law and a legal precedent.

Rory and I met with Richard Gartner and Gareth Knight at CeRch today, to catch up with their investigations into using GATE and OpenCalais to process the EAD outputs from AIM25.

Results look very encouraging. OpenCalais, in particular, generates a post-processing set of identified entities (personal names, place names, corporate names) which Richard G has then created regular expressions to locate these in the body of the EAD and wrap in appropriate EAD tags (<persname> etc).

This suggests that the way forward for enhancing the existing data entry processes for AIM25 will involve dispatching the EAD-compliant data entered by collections manager to OpenCalais, and returning the data, with enhanced markup, for checking by the submitter. This hook should be easy enough to insert for manual, form-based entry; for batch entry processes we will need to assess whether any significant delays are introduced.

We’ve also started to consider ideas for a URI scheme for the entities identified. Our current working hypothesis is that this will involve defining a “data” namespace for AIM25, binding to http://data.aim25.ac.uk/. Within that we can develop a structure along the lines /person, /place, /corporate_body, and append our unique IDs for each entity. Further research is necessary, particularly into the recommendations of the Cabinet Office recommendations for Designing URI Sets for the UK Public Sector.

These URIs can then be used in identifier attributes for our EAD elements (<persname>, etc.), and thence easily transformed into an RDFa format for the Web-based HTML rendering of the AIM25 catalogues.

Next steps include further investigating how to implement and assert relationships between our entities and other open datasets (e.g. our_entity is_the_same_as your_entity). And how to make the authority data, duly marked-up, available as open metadata.

Rory and I can now start to consider suitable approaches to embedding this in our development copy of the existing AIM25 system, and we’ll continue to liaise closely with CeRch for advice on  the relative merits of Gate and OpenCalais processing, and guidance on URI implementation.