Chris Awre, William Nixon, Rory McNicholl at the Texas Longhorns stadium

The conference kicked off with a keynote from Jim Jagielski of the Apache Software Foundation, describing the history and organisation behind Apache and its projects. It was observed by some in the Twitter backchannel that the talk could as easily have been from 2001 as 2011, but for all that it was a worthwhile reminder that, in all our efforts, we stand on the shoulders of the giants who created and maintain the infrastructure of the Web and the Internet. And also that many our endeavours benefit from a little more dedication and commitment than you can usually squeeze between 9-to-5.

The closing keynote was by repositories stalwart Clifford Lynch, who managed to touch on so many perennial repository themes, I won’t attempt to summarise them. There is a handy anthology of tweets about his talk on Storify.

In between were plenty of presentations and opportunities to meet friends old and new from the United States of Repoland – some we have worked with, some we would like to work with, and many with challenging ideas and insights into the many facets of working with repositories.

The OR conference hops back and forth across the Atlantic (I’ve previously attended OR08 in Southampton, OR09 in Atlanta and OR10 in Madrid). Unfortunately when the conference is held Stateside, the representation of the EPrints community tends to be noticeably smaller. Not that there aren’t EPrints users in the USA (we were particularly pleased to meet the team from Cal Tech Library, very happy users and advocates of EPrints), but the distribution of software platforms is significantly different from Europe in general, and the UK in particular (if you are interested in such things, you can check out the statistics at OpenDOAR). And of course travel logistics (and costs) are non-trivial. Luckily Rory and I had been saving our prize money from last year’s Developer Challenge!

While it eluded me in previous years, I think at last I am starting to grasp at least some of the salient points of the thing they call Duraspace (launched, if I recall, in Atlanta)! I’m certainly hoping to find time to take my free Duracloud trial. However other aspects still remain opaque to me. At one panel discussion about the prospects for implementing DSpace over Fedora (or Fedora under DSpace, depending which way up you look at it), I was surprised to hear a description of ongoing DSpace-Fedora alignment efforts as “more about the journey than the destination”. An enviable luxury: for the time being we need tangible outcomes for our repositories and customers, and that’s one reason why we’ll be sticking with EPrints for the foreseeable future.

Personal highlights for me are described elsewhere: the Developer Challenge, which we enjoyed immensely, and Changing Platforms the talk that I presented with Imma Subirats, of the UN Food & Agricultural Organisation, where we discussed migrating between repository platforms. Rory also had a chance to meet developers from Yale, who had worked on the other end of the SOAS-Yale Islamic Manuscripts collaboration, and show off some of his work for the SOAS repository. We were also hugely appreciative of the generosity of the EPrints team, who kept us generally amused and amazed, and kindly included us in their group dinner on the last evening.

Adrian Stevenson's got them all-night late bar open repository blues...

The facilities at UT’s AT&T Conference Centre were outstanding, as was the surrounding campus generally, including the Longhorns football stadium (with its insanely massive west stand) where the conference dinner was held. Austin has far more attractions than we could see in such a short time, and it is an impressive and vibrant city, from the spectacular grandeur of the Texas state capitol, to the noisy entertainment on 6th Street, where virtually every bar has some kind of rock or blues band playing. We were most impressed by UKOLN’s Adrian Stevenson who jammed on a borrowed guitar with the blues band in the Blue Moon bar at 2am. As if that wasn’t enough, our visit also coincided with the massive Republic Of Texas biker rally – an insanely noisy procession of up to 50,000 bikers through the main streets of the city. Our ears won’t forget OR11 in a hurry.

In addition to her work at FAO, Imma is Chief Executive for the E-LIS repository, a major international and multi-lingual repository of articles about Library and Information Science. E-LIS has operated since 2003 on EPrints, but last year migrated to DSpace, because CILEA in Italy, who generously donate support and hosting, now focuses exclusively on working with DSpace. The E-LIS migration has been largely successful, however a number of EPrints features on which the E-LIS editors and users depended, have been difficult to replicate in DSpace, or had to be put on ice. This is no reflection on the specialists at CILEA, but perhaps indicative of more profound differences between EPrints and DSpace, that aren’t always reflected in the usual comparisons of repository platforms, such as the otherwise informative JISC RSP Repository Software survey.

ULCC of course has just completed a repository migration from DSpace to EPrints for the School of Advanced Study. Our motivation was in many respects the same as that of CILEA – our expertise lies firmly in the EPrints camp. But I think the outcomes for our end-user community are more demonstrably positive: in fact I don’t think there’s a single feature of the new SAS-Space-on-EPrints that isn’t a major improvement over its previous incarnation.

Migration of metadata and data (at least from DSpace to EPrints) presented few issues (that weren’t of my own making!) – export, transform, import. Here the similarities between the models of the two platforms was extremely valuable. But we did encounter other significant differences, some of which are set out in more detail below.

Richard presenting Changing Platforms at OR11

Issues in EPrints

Perhaps the most significant issue we encountered with re-implementing SAS-Space on EPrints was the absence of built-in support for Handle persistent identifiers. Handle support comes out-of-the-box with DSpace, but not with EPrints, so the choice we faced was between re-implementing Handle support, or dropping it. We chose the latter, since the benefits of Handles to a relatively small IR like SAS-Space were not obvious, and so it was hard to justify the extra cost and effort. By ensuring that items kept the same ID when migrated from DSpace to EPrints, and implementing a simple rewrite rule, we have ensured that Handle URIs created while DSpace was operational continue to point to the same item – but for items added since EPrints went live, no new Handle URIs are coined.

(Shortly after we returned from OR11, an extended discussion broke out on Twitter, amongst several well-respected gentlemen in our field, about the benefits of using Handles. A considerable amount of scepticism was expressed about their usefulness.)

Issues in DSpace

Imma described some workflow issues encountered with the new implementation of her repository. The E-LIS team is accustomed to a very flexible EPrints-based workflow that allows items to have their workflow status changed quite freely. DSpace, by contrast, has a unidirectional workflow model, so that items cannot (for example) be reverted from Live to Pending, if some kind of error is spotted, but effectively need to be deleted and resubmitted. This is obviously a significant divergence between the superficially similar repository platforms.

Another example Imma gave of a perplexing feature of the default DSpace UI is the button on each abstract page that says “View Full Item Record”. It leads to a rather intimidating web page displaying the item metadata as Qualified Dublin Core. It’s not a very attractive display, nor is it actually a “data” rendering of the metadata (as you would get by explicitly choosing to Export As XML, or from some new-fangled Linked Data features). It’s not clear why this view would be of interest to general users of the repository: why is it there?

At OR11 I talked to several people working with DSpace, and all agreed that there’s room for improvement in the default Web UI. In some cases they have completely reimplemented the web templates. It’s also worth noting that the page layout in the default JSP UI is entirely implemented using HTML tables, and doesn’t pass W3C validation. For a Web application that’s nearly 10 years old, this is disappointing. (The alternative Manakin XML UI implements an attractive vision of UI abstraction using XSLT, but reports suggest that configuring/maintaining it is not for the faint-hearted.)

Quite a few Web design infelicities are perpetrated in the default Community, Collection and Abstract page templates. (During the conference, many of us enjoyed and applauded Simeon Warner’s timely rant,”Don’t bold the field name”.) Of course we can change them – it’s Open Source, isn’t it? – but is it unreasonable to expect default Web templates that are at least potentially usable as is? Of course the natural and reasonable response of the DSpace community is to ask that we report the issue as a bug or feature-request to the development team. Or fix it ourselves and share the fix. But where an absent feature is really important to a user (by which I probably mean a repository manager), then the choice faced is between “getting by” until it’s implemented in the core distribution, or doing it themselves (which probably means hiring a specialist developer to implement it for them).

Out-of-the-box

At the DSpace User Group meeting in 2007, I described how we considered that, back in 2005, DSpace offered a better “out-of-the-box” experience than EPrints. I never thought it was anything to write home about – in fact I remember being disappointed by the very UI issues I’ve described above – but to my untrained eye it did seem better than EPrints, at the time. But, as I’ve mentioned elsewhere, EPrints has improved remarkably since.

Of course a lot of people we admire have proved that you can create impressive repository systems using DSpace. It performs and provides a lot of essential repository functionality. Its Lucene search engine is certainly better than anything EPrints currently offers. But I’m still surprised how much more work seems to be necessary to make a DSpace installation as readily useful and usable as EPrints, and this seems to represent considerable additional cost in setting up DSpace.

I’ve heard it sometimes argued – in both EPrints and DSpace camps – that Repository setup shouldn’t be too easy, lest repository managers get in a mess and endanger the integrity of their system. In my opinion, as developers and solution providers, our job is to provide as many features and tools as possible to enable Repository Managers to manage their collections effectively and easily – not act as as gatekeepers to their systems and data.

By way of contrast, we have recently supported the Institute of Education (IOE) in setting up an EPrints repository of UK government publications, and we were pleased to see the repository manager called on us very little, other than to answer some questions and apply a few small configuration changes. The experience with SAS-Space has also confirmed to me that EPrints now has strong out-of-the-box appeal, and a rich set of features available through the Web UI, that enable a reasonably confident repository manager to get to work without needing to initiate a major technical project.

In the current climate, of straitened library budgets, this could make a considerable difference to the viability of a repository startup project. For a growing number of libraries and information services – not least at smaller research institutions, or in developing countries – that could be the difference between having a repository, or not.

Excitement at the OR11 Developer Challenge Show-and-Tell (Photo by @sparrowbarley)

An event that asked developers to demonstrate the Future of Repositories can only be considered a great success when it receives entries that include:

• Multiple real-time examples of using “Repositories As A Service (RaaS)”, not only exchanging data but also sharing sophisticated functionality between EPrints and DSpace – and even including an Android application
• A tool for bundling and depositing a whole raft of research related outputs from the Web via RDF
• A tactile repository search interface with dynamic search suggestions, specifically designed for tablets and smartphones
• A complete gesture and voice-driven system for depositing and searching in repositories

All these – and other great entries too – were achieved in a couple of days’ work during the course of the conference, for the annual OR Developer Challenge, and presented at a packed Show-and-Tell session on Thursday afternoon (true, there was free beer).

Stuart Lewis’s team were worthy winners with their RaaS project, particularly as they showed a genuine commitment to a cross-platform approach – something which, sensibly, backgrounds the individual software platforms, that often receive too much attention, and focuses on the Repository as an application and entity in its own right.

We were also really pleased to see a prize go to Patrick McSweeney and Matt Taylor. And enjoyed seeing Dave Tarrant stealing the show (again) with his live demonstration of using a Microsoft Xbox Kinect to submit items to a repository.

Our own entry may not have won, but several people liked it, and you may see more of it in future. For the second year running, the Developer Challenge was a great opportunity for Rory and me to concentrate on an idea that we’ve been kicking around, without having found a home for it in existing work (yet). This was true for our Semantic Metadata popup tools that won the challenge with last year.

This year we set about achieving our  longstanding desire to take the very tactile and dynamic search interface that Rory created for the MERLIN project, and turn it into a touchscreen app, for smartphones and tablets.  The results were pretty effective.

The MERLIN Mobile interface ("TEXAS") as demonstrated at OR11

The MERLIN interface on LASSO is quite complex, but at the heart of it is the tag cloud of related terms that the Termine text-mining suggests. This always looked like it might be good on a touchscreen, so we stripped it down, rearranged and tweaked it to make it viable on an Ipad screen. If you’ve got an Ipad, you can try it out by pointing your Safari browser at http://is.gd/texasweb. (It will work on desktop browsers too, but it looks best on a portrait oriented screen.)

If you’ve got an Android device, you can even download an app-based version of it from http://is.gd/texasapp. (It’s a bit cramped in a smartphone display, but still essentially working.)

If you’ve got neither we created this page to give you a rough idea what it looks like.

It’s worth mentioning that this is a real, live, working application: enter a search term (“logical positivism”, “climate change”, “Jeremy Bentham”,  …) and it searches over the full-text corpus of all articles in University of London Open Access repositories (SHERPA-LEAP consortium members), and makes suggestions about additional or alternative search terms, based on the results of the text-mining analysis of the articles

The hackathon approach of working closely together to create something quickly worked well again: Rory hacks, and I test and review each time he hits ‘Save’! All very agile and iterative.

In honor of our hosts in Austin, we decided to call the new interface Touchscreen Enhanced Cross-Search with Augmented Serendipity – or TEXAS for short.

Kudos to Mahendra Mahey, Peter Sefton and the DevCSI project for putting (and keeping) it all together, and to the awesome panel of judges (even though they didn’t pick us)!

Over the last year or so we’ve installed and configured (in some cases reconfigured) the IRStats package for several of the LEAP repositories, including those hosted by ULCC. It seemed a good moment to share a few thoughts about the process of getting “all statted up” with EPrints.

By default, and without any further action, IRStats provides a kind of smorgasbord control panel, demonstrating the many optional graphs, charts and list available. You can see an example on our own ULCC Publications repository.

More recently we’ve seen growing demand among repository managers to share data on downloads with both their depositors and users at large. It’s really important for repository managers to select carefully which statistics views they actually want or need to display – we can only suggest things we think might work. Once you’ve decided on the views you want, we can look at the most effective ways to display them: and this is why I’ve been having fun souping up some of the displays already offered by IRstats.

The first display we’ve been working on is the Statistics digest. These are common enough and we’ve used the example of UCL Discovery repository as the basis of work for both SAS-Space and SOAS institutional repository.

The second approach has been to re-style the IRstats “dashboard” view to lay the graphs on top of each other and then use some Javascript to handle the tabbed navigation. This seemed a more elegant approach than inserting lots of charts in the abstract page itself (as, for example, at ECS EPrints). I’ve used this display technique to display statistics for individual eprints for the School of Pharmacy, as well as SAS and SOAS.

The tabbed display of graphs and tables was also combined with a ‘modal box’ display that keeps the height of page the same (for example on this Abstract page at SOAS. At the bottom of the Abstract page I’ve added a statistics section showing the number full-text downloads, and a link that displays detailed stats in an overlaid box.

This method doesn’t just work for individual items, but can be used on other datasets in too. For example, on SAS-Space we have added it to the bottom of their Collection browse pages, so that at the bottom of each Collection view there is an opportunity to view download statistics for that collection as a whole.

Additionally in SAS-Space, since it is a repository for a number of discrete institutes, there was a requirement for institutional editors to have access to their own institute’s statistics. To achieve this, I allowed access to a constrained version of the IRStats control panel for editor-users who had the appropriate editorial permissions for the institute in question. (Unless you are a SAS-Space editor, you won’t be able to access this.)

Which statistics views to insert as tabs is the decision of the repository manager. Views we’ve used include:

• Monthly downloads
• Daily downloads
• Unique visitors
• Referrers
• Search Engines
• Top 10 items downloaded (only for a Collection, Repository or Division)
• Top 10 search terms

From a technical point-of-view, we will have to review these configurations when we upgrade to EPrints version 3.3, possibly later in the year (if it’s released!!), in conjunction with our VM infrastructure migration, and start doing things with EPStats rather than IRStats. But we now have an effective framework for adding statistics quickly to any EPrints installation.

Some repositories have reported issues with Microsoft “DOCX” files, which IE8 in particular may treat as a ZIP file. This is a potential problem with all the current slew of MS file types. The solution is to add the following entries to your web server configuration.

Exactly how you (or more likely your system manager) achieve this depends on your Web platform (e.g. Apache, Tomcat, IIS) but whoever runs it should be able to make the necessary changes, and once the Web server is restarted, the new types should be picked up. (We’ve just done this for the ULCC-hosted repositories.)

“MIME-Types” have a long and chequered history as a way of identifying file types to internet applications. To some extent IE8 is correct to infer (in the absence of better information from the Web server) that .docx files are ZIP files, because MS Office Open XML formats are bundled using the ZIP compression tool. But in general what one really wants the browser to do is pass the file to an Office application, not WinZip.

Ironically, it seems other browsers do correctly infer MS OOXML file types.

Friday’s CeDAAME dissemination event was also an opportunity to be reminded that ULCC’s Digital Archives team has contributed in other ways to the success of SOAS’s team, directly and indirectly. Julie Makinson described how SOAS used the AIDA digital asset assessment toolkit in developing their strategic approach; and many of the SOAS team are alumni of the DPTP: so Ed and Patricia have also had their part to play in supporting SOAS’s digitisation efforts.

The presentations at SOAS were extremely interesting, describing the full range of activities of a multi-faceted digitisation programme, from the development of the strategy (using the aforementioned AIDA) to the many challenges of digitising Islamic manuscripts and related materials.

How, for example, do you reliably OCR pages of centuries-old text with mixtures of Arabic and Latin/English/French? The answer is that sometimes rekeying is unavoidable. We learned, too, that Yale used UKOLN’s DC Dot Dublin Core editor to create their metadata for Islamic collections (and then convert to TEI). Thanks to the native DC and Unicode support in EPrints, SOAS metadata (in English and Arabic) was created and managed directly in the repository. Metadata exchange between Yale’s Fedora-based system and SOAS’s EPrints system seems to have been achieved effectively – I know Rory worked closely with SOAS and Yale on this.

And I sensed genuine excitement in the room when the page-turning interfaces for viewing the books online were unveiled: both very impressive. (For SOAS Rory has been working long and hard on adapting the open source book viewer used by the Internet Archive, and ensuring that the right-to-left reading and page-turning functionality works smoothly.) We also learned about a variety of different approaches to the issues of managing and funding digitisation and cataloguing activities: with my work on the Mediawiki-based Transcribe Bentham project in mind, it was particularly interesting to hear about University of Michigan’s Collaborative Cataloguing initiative.

All in all an exciting day, and particularly satisfying to see close-up the kind of synergies that exist across all of the activities of ULCC’s Digital Archives and Repositories Team. In addition to further enhancing the SOAS Digital Archives system, we are also looking forward to working with them on their JISC-funded Engaging Overseas Communities project, which is going to involve hooking EPrints up to mobile phones in Africa and Asia.

As if that wasn’t enough, at lunchtime I also dashed over to the School of Pharmacy, where Jean, Neroli and Michelle had kindly organised a lunchtime meeting for the University of London repository managers in the LEAP consortium. It was an opportunity for me to unveil a preview of the new SHERPA-LEAP website (with added social networking goodness, courtesy of WordPress/BuddyPress) that we expect to launch very shortly.

It was a nice way to round off a week in which the Team also achieved significant milestones in our work on preservation metadata for the Parliamentary Archives and strategic development for The Women’s Library, began planning for the next DPTP course, and we received news that the FP7 BlogForever project, which will see us collaborating with Warwick, HATII, CERN and others until 2013, has received its final sign-off from the European Commission.

It was great to take part in last week’s Repositories Support Project event at Sheffield Cathedral. The theme of the day, organised by Jackie Wickham and the RSP team, was “Doing It Differently” and it covered a wide range of repository-related themes. I took along an updated and expanded version of the presentation I made to SHERPA-LEAP repository managers. I covered the same topics, but in preparing the presentation, I was amazed how many more things there were to talk about a year on.

Stephanie Taylor gave an excellent overview of the repository scene, and I hope I followed it up with useful ideas about making repositories more user-friendly or just generally useful to users. Other talks went off into less well trodden areas, though no less interesting: Pat Lockley impressed again with his enthusiastic description of Xpert; Joss Winn described his further adventures in WordPress land; and Stephanie Meece described the challenges of non-textual repositories at UAL. My ears pricked up when Jason Hoyt of Mendeley mentioned that an imminent upgrade to Mendeley will be able to identify OA sources for papers, which might signal it’s time for me to finally catch up with Mendeley (dissertation starts next year!). I didn’t catch the final speakers as I had to catch my train, but I commend to you Vicki McGarvey’s post on the SHARE project blog at Nottingham Trent University.

I tried to keep things simple by steering clear of all the complicated issues in repository management – OA, OAI-PMH, copyright, advocacy, REF, RIM, etc – and just focus on simple UI enhancements that might improve a user’s experience of the repository, and effective use of features like RSS feeds and statistics, with examples from all over the world of institutional and specialist repositories. Which features a repository manager might choose, if any, is up to them and their own circumstances, but my aim was to ensure they are at least aware of what’s possible – as evidenced by what’s been done in many repositories around the country.

Although I focused on EPrints installations, I think nearly everything I demonstrated ought to be feasible in other platforms. Overloading an abstract page with features like “Share this on Facebook/Twitter”, QR Codes, or metadata export in RSS/JSON/CSV and more, should be a very easy way to enhance the user experience of repositories. As I suggested, adding buttons to support “the latest thing” users may be finding useful, is generally not difficult. A “Send This Paper To My Kindle” button, for example, seems so trivial I might even try it myself.

I had a long list of ideas/examples to show: for anyone who didn’t have time to copy down the small print, they were:

• SNEEP
• Lincoln EPrints
• Language Box
• MePrints
• Hum Box
• ULCC Publications Archive
• UCL EPrints
• IRStats
• Repository Stats using Google Analytics (presentation by Graham Triggs at OR10)
• E-Space at Manchester Metropolitan University
• Framework for Linking Inline Semantic Metadata
• Oxford University Research Archive
• MERLIN

We’ve brought along a smashing poster of our work for SOAS, which attracted a considerable amount of interest for its distinctive and attractive Web user interface.

Of the themes of the conference we found most interesting, more in due course.

But we also decided it was about time we entered the annual Repository Developer Challenge. And at the conference dinner on Wednesday we learned that our entry was awarded first prize. La Roja weren’t the only ones with something to celebrate; and we’r'e very pleased to have our names on the same (virtual) cup as such giants of the reposphere as Tim Donohue, David Tarrant, Ben O’Steen and Tim Brody!

The challenge set by the JISC funded DevCSI project, managed by UKOLN, was to “Create a functioning repository user-interface, presenting a single metadata record which includes as many automatically created, useful links to related external content as possible.”

Our idea began by simply imagining a typical repository abstract page overloaded with functional, contextual popup menus, just like a Google Docs page, for example. We could add a menu to each metadata element, and populate the menu with links pertinent to the element.

In order to do that we needed:

• first, to be able to automatically identify the metadata values (title, author and the like) in the page, and,
• second, to manage a list of the web services that are appropriate to each metadata element. The result would automatically generate, on each item’s abstract page, links such as “Find this title at Amazon”, “Find this author at the BL”
• third, some clever scripts that put all this stuff together

The first task was achieved by editing the EPrints templates to ensure each metadata value was wrapped in RDFa semantic tags (spans with a ‘property’ class) that we could identify programmatically. The semantic schemas used are formally declared in the page header.

The second aim achieved by identifying appropriate target services, and the exact URL required to activate them. As a simple example, we can easily create a link to search Google for VALUE using the URL http://www.google.com/#q=VALUE. By way of a data source for these services, I created a spreadsheet in which each row linked a metadata element (e.g. dc:title) to a service (e.g. Google).

The third part was the tricky bit that Rory deftly dispatched. He created scripts that scour the Abstract page looking for RDFa tags, then look them up in the services data table, and dynamically create links as appropriate.

We chose to create an example using the test server that we maintain for the Linnean Online collection. Linnaeus’s botanical specimens make a nice change from the usual ETDR fare of most repositories. This also allowed us to demonstrate using metadata schema from a domain other than the ubiquitous Dublin Core: life sciences have developed a schema called Darwin Core, which defines necessary metadata in that domain, for example as Genus and Species. What’s more, there is a wide range of resources in the field that might usefully be linked to, such as the collections at Kew or the Natural History Museum, the International Plant Names Index and the Encyclopedia of Life.

The results were pretty much as we’d hoped. It’s worth noting that, while we implemented it in EPrints, the technique could be applied to any template-based repository platform, or, for that matter, virtually any web application. Once the RDF templates and code are in place in the templates, it is only necessary to edit the table of data services in order to add or remove links. With only a bit more polish than we had time for during the conference, we think that this could be a useful addition to the toolkit of any repository developer or repository manager. We’ll keep you posted!

UKOLN filmed the proceedings so you can see me presenting our entry at the conference if you want.

Winner of the Developer Challenge at OR10 (Madrid) – Richard Davis and Rory McNicoll from UKOLN on Vimeo.

The AIDA project (Assessing Institutional Digital Assets) has completed its official, funded phase, but it’s gratifying to see interest emerging in the toolkit. We possibly could have done more at ULCC to publicise and sell our work, but our ongoing partnership with the DCC on the current Research Data Management project for the JISC gives us an opportunity to make up for that. One of the planned outcomes of the RDMP work will be an integrated planning tool for use by data owners or repository managers (or indeed anyone who has a digital collection to curate) that will offer the best of DAF, DRAMBORA, LIFE2 and AIDA without requiring an Institution to compile the same profile information four times over. We have already massaged the toolkit into a proof-of-concept online version of AIDA, using MediaWiki, and this clearly signals the way forward for this kind of assessment tool.

I was recently invited to contribute a module about AIDA to Steve Hitchcock’s Keep-It programme in Southampton – encouragingly, he is someone looking into the detail of how repositories could be used to manage digital preservation, and wants input from as many current toolkits as he could get his hands on. My experiences of the day have already been blogged. I thought I would add two other little incidents from the day that I found interesting.

The first was the repository manager whose perception was that assessment of the Institution’s workings at the highest level (for example, its technology infrastructure, business management planning process and implementation of centralised policies) was not really part of her job. So why work with AIDA at all? The main purpose of AIDA is largely to assess the Institution’s overall preparedness to do asset management, and the task of assessment can take an individual staff member (repository manager, records manager, librarian) to parts of the organisation they didn’t know about before. I try and make this sound positive when I encouragingly suggest that an AIDA assessment has to be a collaborative team effort within an organisation. But our friend at Southampton reminded me that people do have these sensitivities and that very often, merely having a repository in place at all represents a hard-won struggle.

The second incident relates to my AIDA exercise, where I asked teams to apply sections of the toolkit to their own organisation. The response fed back by Miggie Pickton was memorable – her team had elected to analyse three separate organisations, applying one AIDA leg (Organisation, Technology and Resource) to each. My initial feeling was that this makes a complete mockery of AIDA, subjective and unvalidated as it might be; what better way to cheat a good score than by cherry-picking the best results across three institutions? However, Miggie’s observations were in fact very useful – and the scores still resulted in a wobbly three-legged stool. It seems that even if they collaborated, HFE Institutions still would not be able to achieve that stability that is the foundation for good asset management.

“Would you want to put that in your repository?” asked Les. And this is a particularly extreme example where we can do some calculations to give us a fairly good answer. Scientific data centres and the researchers that use them have been considering this question for many years, and one way of looking at it is to see if the cost of recomputation exceeds the cost of storage over a particular time period. We’re assuming here that the initial question – is this worth keeping at all – has been answered at least vaguely positively.

Let’s look first at the cost of recomputation. Fabrice says the equipment used for this task cost no more than €2000. If we assume that it has a life of 3 years, that gives us a cost per day of €1.83. I’m avoiding the usual accounting practice of allowing for inflation, or lost interest on capital, in calculating the true depreciation value of the asset – there’s a number of different schemes and they all give similar results. I’ve just dividided the capital cost by the number of days of use we’ll get. But computers use electricity, and that costs money as well. Let’s assume this is a power-hungry beast that draws 400W and that power costs us 13.5¢ per kwH (which is what my domestic tarrif is if we assume a euro/sterling rate of €1.10 = £1 and 5% VAT.) That adds €1.30/day to the cost of running the system, for a total cost of €3.13/day.

Fabrice’s announcement says that it took 131 days of system time to calculate and verify his results, which gives a computational cost of €410.03 – which I’ll round to €410 since I’ve only been using 3 significant figures so far in the computations, and because there’s a lot of hand-waving involved in lots of these figures. So, we know how much it would take to recompute this result given the software, machine and instructions. (And the computational cost is likely to decline over time in the short term.)

The answer needs a Terabyte of storage. What will it cost to keep that in a repository? That’s a slightly more difficult question to answer, but we can give a number of figures that provide upper and lower bounds. SDSC quote $390/Tbyte/year for archival tape storage (dual copies), excluding setup costs and assuming no retrieval. Moore et al quote $500/year as a raw figure, obtained by dividing total system costs by usable storage within it. At current rates of $1 = €0.67, that gives us a cost of €261/year or €335/year. SDSC are likely to be at the cheap end of the scale. ULCC’s costs, given our lower total volumes, would be closer to €1500/year for a similar service (dual archival tape copies on separate sites) although that does include retrieval costs. Amazon’s AWS would be about €100/year for a single copy. You would want two copies, so it’s twice that, and the cost of transferring the data in would be about 25% more than the storage cost. Since I haven’t factored in ingest costs for any of the other models, I’ll ignore it for AWS as well. (And yes, AWS isn’t a repository, and there’s no metadata, and… This is a back-of-the-envelope calculation. It’s a small envelope.)

Which means, at a very rough level and ignoring many pertinent factors, that after about two years of storage in the repository, we would have been better off recalculating the data rather than storing it. There’s a lot of assumptions hidden there, however. For one, we’re assuming that this data will rarely, if ever, be required. If many people want it, the recalculation cost rapidly becomes prohibitive (and so does the 131 days they have to wait for their request to be satisfied!)

One of the other problems is more subtle. I said that, in the short term, recalculation costs would be likely to fall as computational power becomes cheaper. The energy costs involved will rise, of course, but there’s still a significant downward trend. But after a sufficient period of time, it becomes non-trivial to reconstruct the software and the environment it needs in order to allow the computation to happen. Imagine trying to recalculate something now where the original software is a PL/I program designed to run under OS/360. It’s not impossible by any means, but the cost involved and expertise required is non-trivial. At least with our example we won’t have any doubts about whether the right answer has been produced – the computation of π produces an exact, if never-ending, answer. Most scientific software doesn’t do this and the exact answers produced can depend on the compiler, the floating-point hardware, mathematical libraries and the operating system. Over time, it becomes harder and harder to recreate these faithfully, and we often don’t have any means of checking whether or not we have succeeded. (Keeping the original outputs would help in this, of course, but that’s exactly what we’re trying to avoid.) That’s part of the problem that Brian Matthews and his colleagues examine in the SigSoft project and there’s still a great deal of work to be done there.

So have we answered Les’s question ? My feeling is that in this case we have – there’s a fair amount of evidence that suggests that keeping this particular data set isn’t cost-effective. But in general, the question is far harder to answer. Yet we must strive harder for more general answers as the cost of not doing so is not trivial. Even if money did grow on trees, it still wouldn’t be free and at present we need to be very careful how we use it.