Fáilte go Sráid na Banrighinn

The whole purpose of education is to turn mirrors into windows.  ~Sydney J. Harris

Football and digital preservation seem an unlikely combination but on May 15th Ed and I arrived to a slightly damp Glasgow celebrating en masse the end of the football season, to deliver our DPTP north of the border.  There, I am afraid the comparison between football and digital curation ends but it is worth noting that on Monday our students did not consist, in the main of bleary eyed Scots.  As I am myself half Scottish I was looking forward to hearing about the digital state of play in my mother’s country but curiously we had a very international group, many had travelled far from places such as the European Central Bank,  European Commission, PRONI,  and near  such as from the National Records Scotland (who clearly cannot get enough of a good thing as we were with them 4 weeks previous).  The group was very creative who seemed to work well together. I know  I know, we say this a lot but this group seemed to shine for many reasons.  They were keen to work together and shout out about their ideas right from the start. And though being a disparate international group they managed to work together very neatly and develop some great case studies for us.

Internationally (and sometimes nationally!) language has always been a barrier  for communication, but  it is also an enabler  and this is again where we see something such as the OAIS being a useful way of empowering people to be able to communicate with each other, and other professionals.  The DPTP rarely (and rightly so) has students just from the traditional information management world (libraries/archives/etc).  Many people now being redeployed from different parts of their organisation and as such are often unfamilair with the idea/concept of an archive or indeed of the traditional notion of the lifecycle of a document/record. This is where OAIS is very valuable due to the way it which it expresses the way a digital object should be ideally kept and uses a language describe this.

In many of our case studies with OAIS, we often find that many organisations are actually mapping very well to the OAIS which indicates that it reflects  good and real practise when it comes to managing our digital repository. Our job in the DPTP is to take someone who has never heard of the OAIS (suprisingly more often than not) and by the end of the three days have them fluent in the OAIS concepts both through listening and through application to their own environments.  Our feedback from the course has proven that this is a job well done….DPTP abú!

Rating a JISC bid

Marking

Markers – at least those outside the JISC Executive – get anywhere between 1 and 10 bids to look at, and something like 2 weeks in which to do the marking. As well as the bids, they’ll get some guidance for markers and the original text of the call. They’ll also see a log of all bids received, which as well as assigning each bid a unique ID, tells them how much each strand is over-subscribed (that is, how much is being asked for against how much is available.) And they’ll have access to a closed email list to clarify issues with other markers and the Executive. In my experience these lists are not much used these days, although they can prove useful to clarify ambiguities in the call or the marking criteria. Some are probably using Twitter for this now, which is unfortunate as it means that not all the markers will be aware of the discussions.

At an early stage, markers are meant to double-check that they don’t have any conflicts of interest in the bids assigned to them. A bid in which your own institution is a partner definitely constitutes such a conflict, and the Executive usually spot these in advance. But they arise for other reasons, and every marking episode I’ve been involved with has had at least one person realising that they have a conflict, often very late in the process.

Then we begin the process of reading, evaluating, and assigning marks. I suspect everyone has a different approach to this, but the outputs are the same. We score each criterion on a 5-point scale that’s really a 10-point scale (because it has half marks.) The criteria include such things as “Appropriateness to the call” and “value” and the lowest rating implies that the bid fails that criterion in a way which just can’t be fixed, whereas the highest implies that it exceeds expectations significantly in a number of ways. Markers also make comments on each criterion to explain the thinking behind their marking. These comments will form the bulk of the feedback that you will receive if you ask for it.

It’s worth noting that you can (and should) ask for this feedback even if your bid is successful. Sometimes your programme manager will offer it to you unasked. Few bids are perfect, and most bids contain something of value. The feedback can tell you what you need to improve but it will usually also tell you what you did well. Both are good to know!

As well as the criteria-based marking, we are also asked to rate the bid overall as A, B or C. As the picture above shows, this means that we strongly recommend funding, weakly recommend it, or do not recommend funding. These marks are the most significant ones once all the bids are considered at the evaluation stage.

The marking process is now all done via the web, although some find this frustratingly slow if they have to copy their marking information from some other source to the web forms.

The Evaluation Panel

The evaluation process usually involves a face-to-face meeting of all the markers, or just those from the Executive. The exact conduct of the meeting will depend on the size and complexity of the call, the number of bids and the number of projects to be funded. Every bid will have been marked by at least 3 people. Typically, one that scores AAA will be approved without further discussion, and one that scores CCC is likely to be rejected without significant discussion, although the panel will make sure that there’s sufficient information in the comments to provide feedback for the CCC bid.

What happens next depends to a great extent on the degree of competition, the quality of the proposals and the way the evaluation panel chair chooses to work. If there are many more proposals than can be funded, it’s not unusual to try to pick off further outliers – ones that stand out from the rest as being particularly strong or weak – before examining the rest in detail. The marks in each criterion will often come into play here, either to choose between two bids with equal recommendations, or to compare (say) an ABC with a BBB. Markers may be asked to justify or clarify their comments, and opinions do change as the result of discussion at this stage. The Executive will also want to bring other considerations into the process – either to ensure a range of different types of project are funded, or to ensure that funding goes to a range of institutions. Similarly, bids are sometimes approved subject to (agreed) change. Scores of 3 or below for any criteria imply that the marker sees problems that can be, and should be, corrected before funding is approved. If one institution receives funding for a number of related projects, they may be asked to look for economies of scale between them.

When there’s less or even no competition (with the number of bids being less than or equal to the number of projects desired) then the evaluation will have a different focus. It will still be necessary to eliminate projects that are too weak to receive funding, even if that means that some funding will be unspent. (Sometimes that funding can be reallocated to another stream where it can be better spent.) For those that can be funded, any concerns that the markers had need to be turned into guidance for the programme managers or the projects themselves. The budget may need to be made clearer, the dissemination plan improved, or the project may need to take account of the work of a related project, for instance.

In fact, as I was relieved to note, this group of standards isn’t entirely new to the digital preservation community, and the authors are also aware of general DP standards such as OAIS. They derive from a group of standards known as STEP (Standard for Exchange of Product Model Data), codified in ISO 10303. STEP came up in a presentation and discussions at PresDB’07, although it’s been in development since the turn of the century at least. But STEP is a huge family of standards, and this particular work appears to have emerged from specific work going on in the aerospace industry, which is known to have had concerns about long-term survival of CAD data for some time.

LOTAR is one of the results of this concern, and a presentation by Pierre Duchier of AIRBUS at an Aristote conference in April this year gives a clear picture of the concerns of industry and the approach they are taking. LOTAR and related work were also covered at a invitation-only event at Bath in 2007, Atlantic Workshop on Long-Term Knowledge Retrieval whose attendees included many names familiar from the DCC and the repository community.

Overall, this is reassuring. Here is a set of digital preservation standards being developed and driven by concerns in industry, but where the work is taking place in dialogue with with the academic research and development community. CAD data in particular has long been a concern at the Archaeology Data Service, which has significant holdings of Autocad files, and it will be interesting to see to what extent LOTAR has relevance for such activities. In the meantime, I can see I’ve got a lot of reading to do to catch up, but it would be interesting to hear from others who have more insight into the possible use of these standards outside the aerospace industry.

Incidentally, BS EN 9300 is not to be confused with BS EN ISO 9300, an entirely different standard concerned with “Measurement of gas flow by means of critical flow Venturi nozzles“.

Authenticity is a key requirement in any digital preservation application, and it’s not a particularly novel one either as this post by Maureen Pennock makes clear. One of the six mandatory responsibilities which OAIS places on an archive deal with authenticity and traceability, and this was reflected virtually unchanged in the TRAC methodology, in which one of the audit checkpoints was that the repository

… enables the dissemination of authentic copies of the original or objects traceable to originals.

That choice of words is important: most repositories aren’t dealing with use cases that involve legal evidence on a daily basis, and for most users it isn’t necessary to provide dissemination copies of material that can be taken straight into a courtroom. Sometimes, doing so places extra costs on the dissemination process that aren’t needed. Other repositories may find it easier to make all copies ‘authentic’, or ‘capable of being traceable to originals.’ But we simply wanted to ensure that repositories were capable of doing this when it was required. If they could not, it called into question their own confidence in the integrity of their holdings.

BS 10008 deals with a subset of the problem, one suitable for document-oriented problems, and it isn’t at all specific about the mechanics of how it is achieved. The standard is going to be useful and it’s likely to have more weight as a standard (to which it’s easier to force compliance) than as a code of practice. But there’s areas that are worthy of comment from the DP community. Take, for instance, this definition of migration:

transfer of electronic information from one storage media to another

That’s certainly not the usual use of the term – I would use either ‘refreshing’ to refer to this action, or ‘media migration’ if I wanted to make it clear that no change in format or encoding was involved but still wanted to talk of ‘migration.’

The document is open for public comment until June 30th, 2008. You can view the draft and make your comments using BSI’s online review website (Incidentally, this is the first time I’ve used the online review process, and it works a treat. I last reviewed a standard about 10 years ago, when it was still a very paper-based process, although one could at least submit comments by email then. This is a great improvement. And while you’re there, you can take the opportunity to review such upcoming standards as BS 8507-1 for “Close Protection Services” – otherwise known as bodyguards, methinks)

Does this mean that OAI-PMH is a standard that’s had its day? Some commentators have gleefully rejoiced in the possible demise of the standard. However the discussion on Paul Walk’s blog is more balanced, lucid and informative, and covers well the many areas where OAI-PMH is more and less valuable. We shouldn’t forget that Google Sitemaps and OAI-PMH were created for different purposes, though both use XML, and inevitably there is potential overlap in their application.

NDAD has an OAI-PMH interface: I don’t know whether this has ever been of any use to anyone. NDAD’s holdings are markedly different from the majority of Open Access repositories running Eprints or DSpace (see ROAR for a full list).

Interesting what other things you learn too – for example, courtesy of a comment on Lorcan Dempsey’s blog, that “[t]here’s been an unofficial Google Sitemap hack for DSpace for half of forever. The funny thing was when its use was recommended to keep Google crawlers from crashing DSpace installs.” We have a running battle with Google crawlers triggering errors on our DSpace server, so this might be worth knowing when we start our next cycle of maintenance and upgrade.

A propos of how this affects Eprints, Tim Brody points out on Eprints-Tech:

You don’t need to change anything.

Sitemaps can be used to expose hard to find Web pages or to improve indexing by focusing the crawler at particular parts of the site. The EPrints 3 Web interface has already been designed with Google in mind and is fully compatible with their normal crawler without the need for sitemaps. That doesn’t preclude the addition of a sitemap to EPrints but it seems unnecessary at the moment.

(EPrints 3 has some Google-requested additions, EPrints 2 has much the same crawler-friendly interface)

One interesting aspect of this event is Google’s approach to the change:

In order to move forward with even better coverage of your websites, we have decided to support only the standard XML Sitemap format by May 2008. We are in the process of notifying sites using OAI-PMH to alert them of the change.

An interesting contrast to the more cavalier approach of Microsoft in its recent MS Office 2003 format debacle.