As an outsider sometimes it feels that reading/hearing about digital preservation research is a bit like reading an article on particle physics.  In fact sometimes lectures on particle physics might be clearer.  The number of acronyms, the amount of jargon per project and also the amount of assumed knowledge can, at times make our subject inpenetrable to many, not to mention boring.

So with this in mind I steeled myself for a few days at the PLANETS workshop in London recently.   At the DPTP we talk a lot about OAIS and I was keen to hear about PLANETS’  ’suite’ of tools developed with the OAIS model and digital preservation in mind. The OAIS model implies a number of flexible preservation workflows so I was looking forward to hearing about what the  PLANETS team had done.

Ross King’s  wonderfully clear session really drew the crowd in.  He set out the business case for digital preservation, provided explanations of the technical elements that make up digital preservation, the technical problems behind preservation; the options available and the decisions that need to be taken. While not taking away from the sheer volume of ’stuff’ being created and which will be created as well as the challenges, it was a very motivational session. Next Clive Billeness drew our attention to the fact that risk is the only real motivating factor when it comes to getting the powers that be in organisations to take digital preaservation seriously. He says that we need to talk to financial planners, CEOs in terms of risk, using the language of risk. Companies are now being asked to keep a ‘living will’ so that they can garantee that they can operate in case of a disaster, ensuring that records needed to operate are kept in a manner which can ensure operations.

So what are these PLANETS tools? I am not going to cover them all but Hannes Kulovits introduced us to PLATO,  PLANETS preservation planning tool.  Preservation planning for digital preservation includes preservation policies, legal obligations, organisational and technical constraints, user requirements and preservation goals. It also describes the preservation context, ‘the evaluated alternative preservation strategies and the resulting decision for one strategy, including the rationale of the decision.’ During the workshop we looked in depth at how to decide on preservation strategies and factors which influence decision making.  Input is needed from a wide range of persons, depending on the institutional context and the collection. PLANETS have also pinned down some essential characteristics for digital objects with regard to their preservation.

I was a bit narked that I didn’t get to actually play with it on my laptop there and then but I have done so since the workshop.  How long this planning process takes place is another issue. This was a bit of a drop for us all, yes planning takes time but that much time?

A quick survey of the crowd by me during and after revealed that some people felt that while they enjoyed initial content they felt that the workshop  wasn’t for them, too technical or too highbrow. I consider myself to be a low to middle brow kind of girl (dp-wise I hasten to add) so was suprised by this as I felt that it was one the clearest events I had been to on digital preservation.  I felt this response was a real pity, perhaps there needs to be more interaction with the groups attending these workshops (one remaining) to gauge receptivity.  I struggle with projects who really cannot communicate their products. I didn’t feel this case here. The Planets Framework is really good and may need definining but I really enjoyed it and look forward to next developments.

Obviously, transcription of manuscript materials is an important digitisation activity that can rarely, if ever, be left to computers, in the way that printed texts can be, using OCR. But it’s painstaking and laborious work, and anything that eases the burden is welcome.

The project is already throwing up some very interesting conversations about transcription.  At ULCC we have thought about transcription before, particularly with regard to our ongoing work for the Linnean Society archives, and we hope that there will yet be synergies to exploit. It is a great feeling to be so closely involved with disseminating the work of two such seminal figures as Linnaeus and Bentham.

We’re not naïve enough to think that collaborative web-based transcription is new, but we’ve yet to find any substantial comparable examples. A comment on UCL’s Digital Humanities blog teases us with the prospect of information about other similar projects, but fails to provide even a single link or hint, so is effectively useless: hardly in the collaborative spirit! A more useful lead was Joanne Evans’ link to the National Library of Australia’s Australian Newspapers project, which is crowdsourcing the proof-reading and correcting of OCR outputs, and has an impressive-looking site – I’m sure we’ll be borrowing some ideas from there.

Another useful lead has been from Ben Brumfield of Austin, Texas, directing us to his blog about collaborative manuscript transcription which has been going even longer than DA Blog, and looks like it’s going to make interesting reading. Ben’s recent blog post about a distributed transcription exercise of the US Geological Survey’s Bird Phenology Program includes a link to a training video for volunteers (it even sounds like it’s been recorded in a birdhouse).  In the video we can see a database-form approach to transcription, which is particularly appropriate for transcribing data already entered on structured forms.

For more heterogeneous and free-form texts, such as the Bentham manuscripts, wikis seem to me much more appropriate, being in essence discrete hypertext engines. As for collaborative features, MediaWiki in particular has strong and proven features: there can be few better advertisements for effective virtual, global collaboration and crowdsourcing than Wikipedia.

One thing that is particularly compelling about the BPP video is that it is an excellent example of a thorough approach to online collaboration, giving clear and unequivocal guidance to contributors. Now that screencast tools are so readily available, it’s clear that for many activities like this, video-based instruction is the ideal tool, and often preferable to any number of written instructions. No less than for online teaching and learning environments, the need for effective induction and inclusive management of the online community must never be overlooked.

The Digital Preservation Coalition are also generously giving three, yes THREE scholarships to attend the DPTP. Once again open to DPC members. Remember folks, this can include your institution or your professional association.

Hope to see some of you there!

I was recently invited to contribute a module about AIDA to Steve Hitchcock’s Keep-It programme in Southampton – encouragingly, he is someone looking into the detail of how repositories could be used to manage digital preservation, and wants input from as many current toolkits as he could get his hands on. My experiences of the day have already been blogged. I thought I would add two other little incidents from the day that I found interesting.

The first was the repository manager whose perception was that assessment of the Institution’s workings at the highest level (for example, its technology infrastructure, business management planning process and implementation of centralised policies) was not really part of her job. So why work with AIDA at all? The main purpose of AIDA is largely to assess the Institution’s overall preparedness to do asset management, and the task of assessment can take an individual staff member (repository manager, records manager, librarian) to parts of the organisation they didn’t know about before. I try and make this sound positive when I encouragingly suggest that an AIDA assessment has to be a collaborative team effort within an organisation. But our friend at Southampton reminded me that people do have these sensitivities and that very often, merely having a repository in place at all represents a hard-won struggle.

The second incident relates to my AIDA exercise, where I asked teams to apply sections of the toolkit to their own organisation. The response fed back by Miggie Pickton was memorable – her team had elected to analyse three separate organisations, applying one AIDA leg (Organisation, Technology and Resource) to each. My initial feeling was that this makes a complete mockery of AIDA, subjective and unvalidated as it might be; what better way to cheat a good score than by cherry-picking the best results across three institutions? However, Miggie’s observations were in fact very useful – and the scores still resulted in a wobbly three-legged stool. It seems that even if they collaborated, HFE Institutions still would not be able to achieve that stability that is the foundation for good asset management.

Beacon cited through fog

Who would have thought that reference management could be so interesting? We spent a  very informative and enjoyable Thursday in snowy Milton Keynes, at the Innovations in Reference Management (#IRM10) event (part of the OU/JISC TELSTAR project). All thoroughly blogged by Owen Stephens, and tweeted by many.

Owen Stephens and Jason Platts of OU described the outputs of the TELSTAR project, which integrates the OU’s Moodle VLE with Refworks. This means that students using the VLE can move seamlessly between their reading lists and Refworks, locating resources, maintaining consistency of style and generating bibliographies easily.

Paul Stainthorp of Lincoln University described some exciting, bleeding-edge uses of Yahoo Pipes to mashup data from Refworks, OPAC, and Amazon. Arguably even more bleeding-edge was the presentation by Euan Adie from Nature Publishing, who showed us Help Me Igor, a reference manager plugin for Google Wave. Speakers from CiteULike and Mendeley also gave us fascinating insights into their respective social-tinged bibliographic management offerings.

Perhaps unsurprisingly, Kevin and I brought to the table the theme of web preservation. With reference to our work with JISC-PoWR, UKWAC and ArchivePress, we reminded anyone who hasn’t heard our spiel already that there are many important, valuable and eminently citable web resources, notably blogs by academic researchers, that are at risk of disappearing – making references to them virtually useless.

Authors may not be responsible for ensuring their readers can access the resources they reference, but we think they should at least give them a fighting chance of doing so! We  therefore proposed that students and researchers should be encouraged to locate and cite copies of web resources in stable web archives (such as the UK Web Archive) rather than “in the wild”.

We also discussed the idea that persistent collections of web resources could be created at the institutional level, whether that were an open archive of blog posts by a university’s researchers, or a closed repository where researchers can store copies of the web resources they cite.

One of the strong themes that emerged in discussion was the need for information literacy/digital skills training at all levels to address current tools and trends in reference management; and to re-assert the purpose, value and nature of citation in online digital environments

An interesting suggestion also made was that reference management tools are becoming a natural part of the environment, just as email has: is provision of specialised applications by universities an “aberration”?

I’m inclined to think not, after all it was clear from the workshop that there’s still a need to support ongoing study and research effectively, and scope to develop and validate new approaches.  Microsoft Word may now include reference management features, but that doesn’t obviate the need to educate people in how to use them effectively, and why.

We’re very grateful to Owen for including us in his programme: this is a fascinating area, where e-learning, libraries, preservation and publishing collide, and I’m sure we haven’t heard the last of it.

“Would you want to put that in your repository?” asked Les. And this is a particularly extreme example where we can do some calculations to give us a fairly good answer. Scientific data centres and the researchers that use them have been considering this question for many years, and one way of looking at it is to see if the cost of recomputation exceeds the cost of storage over a particular time period. We’re assuming here that the initial question – is this worth keeping at all – has been answered at least vaguely positively.

Let’s look first at the cost of recomputation. Fabrice says the equipment used for this task cost no more than €2000. If we assume that it has a life of 3 years, that gives us a cost per day of €1.83. I’m avoiding the usual accounting practice of allowing for inflation, or lost interest on capital, in calculating the true depreciation value of the asset – there’s a number of different schemes and they all give similar results. I’ve just dividided the capital cost by the number of days of use we’ll get. But computers use electricity, and that costs money as well. Let’s assume this is a power-hungry beast that draws 400W and that power costs us 13.5¢ per kwH (which is what my domestic tarrif is if we assume a euro/sterling rate of €1.10 = £1 and 5% VAT.) That adds €1.30/day to the cost of running the system, for a total cost of €3.13/day.

Fabrice’s announcement says that it took 131 days of system time to calculate and verify his results, which gives a computational cost of €410.03 – which I’ll round to €410 since I’ve only been using 3 significant figures so far in the computations, and because there’s a lot of hand-waving involved in lots of these figures. So, we know how much it would take to recompute this result given the software, machine and instructions. (And the computational cost is likely to decline over time in the short term.)

The answer needs a Terabyte of storage. What will it cost to keep that in a repository? That’s a slightly more difficult question to answer, but we can give a number of figures that provide upper and lower bounds. SDSC quote $390/Tbyte/year for archival tape storage (dual copies), excluding setup costs and assuming no retrieval. Moore et al quote $500/year as a raw figure, obtained by dividing total system costs by usable storage within it. At current rates of $1 = €0.67, that gives us a cost of €261/year or €335/year. SDSC are likely to be at the cheap end of the scale. ULCC’s costs, given our lower total volumes, would be closer to €1500/year for a similar service (dual archival tape copies on separate sites) although that does include retrieval costs. Amazon’s AWS would be about €100/year for a single copy. You would want two copies, so it’s twice that, and the cost of transferring the data in would be about 25% more than the storage cost. Since I haven’t factored in ingest costs for any of the other models, I’ll ignore it for AWS as well. (And yes, AWS isn’t a repository, and there’s no metadata, and… This is a back-of-the-envelope calculation. It’s a small envelope.)

Which means, at a very rough level and ignoring many pertinent factors, that after about two years of storage in the repository, we would have been better off recalculating the data rather than storing it. There’s a lot of assumptions hidden there, however. For one, we’re assuming that this data will rarely, if ever, be required. If many people want it, the recalculation cost rapidly becomes prohibitive (and so does the 131 days they have to wait for their request to be satisfied!)

One of the other problems is more subtle. I said that, in the short term, recalculation costs would be likely to fall as computational power becomes cheaper. The energy costs involved will rise, of course, but there’s still a significant downward trend. But after a sufficient period of time, it becomes non-trivial to reconstruct the software and the environment it needs in order to allow the computation to happen. Imagine trying to recalculate something now where the original software is a PL/I program designed to run under OS/360. It’s not impossible by any means, but the cost involved and expertise required is non-trivial. At least with our example we won’t have any doubts about whether the right answer has been produced – the computation of π produces an exact, if never-ending, answer. Most scientific software doesn’t do this and the exact answers produced can depend on the compiler, the floating-point hardware, mathematical libraries and the operating system. Over time, it becomes harder and harder to recreate these faithfully, and we often don’t have any means of checking whether or not we have succeeded. (Keeping the original outputs would help in this, of course, but that’s exactly what we’re trying to avoid.) That’s part of the problem that Brian Matthews and his colleagues examine in the SigSoft project and there’s still a great deal of work to be done there.

So have we answered Les’s question ? My feeling is that in this case we have – there’s a fair amount of evidence that suggests that keeping this particular data set isn’t cost-effective. But in general, the question is far harder to answer. Yet we must strive harder for more general answers as the cost of not doing so is not trivial. Even if money did grow on trees, it still wouldn’t be free and at present we need to be very careful how we use it.

As regular readers will know, we have been working with repositories for quite a few years now. In 2005 we began working with the School of Advanced Study on their requirements for an Institutional Repository, and since then we have installed, configured and maintained several repositories, including some highly customised, specialist systems.

In most cases we have used EPrints. This is partly because we are familiar with the stuff it is built with (Perl, MySQL and XML have been at the heart of the NDAD dataset repository we have operated for The National Archives since 1997). But also because we like the ever-expanding set of features and options EPrints provides. I’ve watched its capabilities grow, thanks to the seemingly limitless energy and initiative of the EPrints team at Southampton. (For an interesting, user’s-eye perspective on the relative merits of DSpace and EPrints, I recommend reading some of the posts tagged DSpace in Dorothea Salo’s Caveat Lector blog).

It’s three years almost to the day since Rory and I attended the pre-launch briefing on EPrints3 and came away convinced that, with its AJAX UI and evolving plugin architecture, EPrints 3 was likely to play a big part in our future plans.

And hardly a day’s gone by since, when we haven’t had some EPrints-related work on our plate. In 2007 we began developing Linnean Online for the Linnean Society, and PRIMO for the Institute of Musical Research. Out of this, and the snowballing Web 2.0 zeitgeist, we also honed the idea that became SNEEP (Social Networking Extensions for EPrints), one of the first JISC Rapid Innovation projects. Most recently, we’ve scaled new heights of EPrints customisation with the SOAS Fürer-Haimendorf collection, with its user-defined albums and searching enhancements, all wrapped up in 9Web’s impressive graphic design.

We’ve tweaked config files and hacked templates and for the most part enjoyed doing stuff with EPrints. (All credit is due to Rory and Ben, by the way. My role is chiefly to say “We could make it do that couldn’t we?” And, lo and behold, usually “we” can.)

Over the years I’ve also talked to many repository managers, and potential repository managers, about their requirements and expectations. I’ve spoken and networked at DSpace User Groups , Open Repositories conferences and many excellent events organised by the JISC, particularly the Repositories Support Project – and I’ve met a lot of smart and insightful people in the repo biz. Some of it must have rubbed off – I think my own understanding of what’s needed, and what’s feasible has grown considerably.

But what we’ve never done is run our own repository, and experienced these things day-to-day for ourselves. As Atticus Finch said in To Kill A Mockingbird,

You never really understand a person until you consider things from his point of view . . . until you climb into his skin and walk around in it.

That’s why, in the gaps between everything else going on round here, Annemarie has been putting together the ULCC Publications Archive, which I hope will become a canonical home for our published outputs. It’s not big and it’s not clever, it’s certainly not perfect, but it is something we can use to improve our understanding of what it means to run a repository. We will also no doubt use it to explore some of the tools and techniques emanating from the EPrints developer community.

And now I can really start to empathise with the repository managers I know: their agony – clarifying copyright and licenses, ambiguous form fields, disappearing diacritics – and their ecstasy – a well-formed subject tree or citation, a successful search. I’ve also an insight into the needs of authors/submitters, since several articles are mine – and I naturally want to get the citations looking just right, so that I can embed some of the nice feeds EPrints provides into my blogs, e-portfolios and who knows what other mashups. Self-interest is a great motivator, as many Open Access advocates have observed: before long I’m sure I’ll be wanting download statistics, author profiles, and most of the other things I described in 1001 Things To Do With A Live Repository.

For me it’s an invaluable experience – no less so than when, a couple of years ago, I became an actual user of a VLE, through my MSc course at Edinburgh. There’s a world of difference between being a developer or implementer of this kind of online system – thinking your job’s done when it seems to be up-and-running – and being the poor end-user who doesn’t care about PHP, JSP, Maven, Apache, etc, but  just wants to get something done.

Among the things you’ll find in pubs.ulcc.ac.uk are: papers and articles from events we have contributed to over the years, such as iPRES, Open Repositories, and DLM-Forum; published reports, like last year’s JISC-PoWR web preservation report; presentations and posters from other events, mostly in the field of e-learning or digital archives; and even the swish product sheets produced by our ace marketing department, Tim and Frank!

As well as our most recent UK activities, we’ve also unearthed some other curios, such as Patricia’s article for the Catalan Archivists’ Forum, in Catalan, and a piece by Kevin in La Vanguardia, in Spanish. Also of interest is a brief account of ULCC’s first 30 years, in the form of a brochure for a small exhibition that was held at Senate House Library in 1999.

No doubt as we delve through our own digital archives we’ll find more goodies. Having a repository is an excellent opportunity to locate and appraise these things, and share those that seem interesting and informative enough. No less than this blog, and our E-learning colleagues’ El Blog, it should be an attractive and effective shop-window – just like any good Institutional Repository.

My naïve thinking in the area has been characterised by the assumption that the process is rather static or linear, and that the problem we’re facing is broadly the same every time; migrate data from a format that’s about to become obsolete or unsupported, onto another format that’s stable, supported, and open. MS Word document to PDF or PDF/A…now that, I can understand!

In fact, I learned at least two ways of thinking about formats that hadn’t occurred to me before. One simple one is costs; some formats can cost more to preserve than others. This can be calculated in terms of storage costs, multiplied over time, and the costs associated with migrations to new versions of that format. For example, we’ve tended to pin our faith on the TIFF format for images for many reasons, but there’s a high storage price to be paid for all that wonderful losslessness. This may be one reason why the DP world is looking with more favour on the JPEG2000 format, which is ‘virtually’ lossless and smaller in size.

Secondly, the problems of preserving digital data which doesn’t actually have a specified stable preservation format. Chris Puttick of Oxford Archaeology gave a vivid description of the problems he’s facing with CAD and GIS files, where the data can’t easily be tied to a single format in the first place (nor can a stable format for migration be identified). As the NLA put it on their PADI page, “At present there is little dealing specifically or comprehensively with the preservation of this particular type of data, although some aspects of database preservation are applicable to GIS. Some long term preservation issues include a lack of open source formats and metadata standards, large data volume and complex data objects.” Puttick suggests that his data doesn’t really perform at all unless it’s operated within a very specific environment of hardware and software. How do we preserve an environment? This appears to be quite a distinct preservation problem and much harder to solve than Word to PDF, to put it mildly.

William Kilbride suggested that such cases (and websites too, arguably, because they are time-based) are more like a stream of data – a handy image which conveys something about the dynamic of such information packages, and showing us that it’s much harder to nail them down into a single format. You can never step into the same river twice.

We had a brief preview of the new DPC website, which promises to be a much more effective mechanism for the membership to engage with each other and the wider world, and confirmed recommendations emerging from a planning day earlier in November which should keep the DPC busy (and financially secure) for a few years to come.

Finally, we had an entertaining and thought-provoking talk from Professor Michael Anderson. Professor Anderson touched on many issues relating to digital preservation from his research career, past and present. He mourned the loss of Scottish census microdata from 1951 and 1961, painstakingly copied to magnetic tape from round-holed punch cards for 1951 and standard cards for 1961, which had to be destroyed when ONS realised the potential for inadvertent disclosure of personal information. But more tellingly, he described the loss, or partial loss, of data closer to home for him, and which has implications for research publications underpinned by data. For a spreadsheet of fertility data for Scotland, no copy survives of the raw underlying data although many derived and aggregated copies exist. For other material, the data is still available but he is slowly losing the ability to do what he wants with it, as applications designed for 1980s-era Macintosh systems find it more and more difficult to operate properly under contemporary operating systems.

These problems are frustrating for specialists to hear about because we know they could have been resolved easily and cheaply if intervention took place early enough. The spreadsheet has been lost, it appears, because it seemed to difficult to access after one change of technology, so its owner simply destroyed it. The primary problem here, then, is one of advocacy rather than technology.

Professor Anderson also talked of two familiar barriers to the deposit of research data: the unwillingness to deposit data seen as being insufficiently polished for re-use, and the lack of reward or recognition for data depositors, as opposed to those who publish papers based on the data. Although he didn’t offer answers, he did make clear that this is a problem which has been with us for a long time. He ended with a plea to librarians and others to ensure that the type of publication he wants to produce now – such as publications containing actionable data – can be preserved and accessed in future more easily than some of the data that’s already been lost.

I had to leave shortly after a lively discussion had begun, which was a shame, As well as reassuring Professor Anderson that the 1961 census microdata were safe, I would have liked to thank him for his work in initiating the Research Support Libraries Programme (of which Ronald Milne was the director) which provided the funding to establish AIM25 – a service that’s still going strong over 10 years later, and which we’re proud to be involved with.

Following up his excellent interview with William Kilbride at DPTP in May, this time Frank Steiner conducted video interviews with some of the DPC students and scholarship winners – hopefully you can see them below:

This is just the first phase of the rollout, not only of Fürer-Haimendorf’s pictures, but also of many other valuable collections at SOAS. We are pleased and excited to have been able to assist with this endeavour, by customising EPrints to meet the extensive requirements set out for the system by Susannah Rayner and Malcolm Raggett, who are leading the project at SOAS. No less than with Linnean Online, it is a rare privilege to be associated with a project giving a new impetus, and worldwide access, to such invaluable historically important, archival collections.

SOAS organised a fascinating series of lectures on Friday to launch the collection. In one talk, Albert Alban von Stockhausen spoke passionately about the importance of sharing the collection not only with researchers, but also with the communities that Haimendorf visited. Surviving subjects in Nagaland, and their descendants, have been deeply moved on seeing these photos again. Some even refer to their own age in terms of how old they were when Fürer-Haimendorf visited.

Stuart Blackburn talked about Fürer-Haimendorf’s work in the Apatani valley in the 1940s, where he got a rapturous reception, if not as the first westerner to visit them, then probably as only the 5th or 6th, when he acted as an official representative of the British government in India, trying (not always successfully) to resolve tribal disputes. In the photos we can see valuable records of rice-growing, forest-clearance, village-settlement patterns, ritual and warrior practices, and the Apatani villages, with their densely populated, crowded lanes. There are also many individual portraits.

We are particularly pleased with the results. It is an EPrints-based repository which implements the very exacting requirements set out by the SOAS team. It has been an opportunity to develop even further the work Rory has done with EPrints, drawing on our experience adding plugins for bookmarking, tagging and commenting, which began with Linnean Online and continued with SNEEP. We also particularly enjoyed, and benefitted from, working closely with the team from 9Web, who provided graphic designs and meticulous usability testing.

I hope to enumerate the new features and describe the development work in a future post; meanwhile you can see it in action at digital.info.soas.ac.uk.

As you’ll know from my previous post, the CLASM project is developing a plugin to enable direct deposit from Moodle to any SWORD-compliant repository. Our specific use case is to support management of CLA materials for teaching, but there is no reason why the Moodle plugins that James has developed couldn’t be adapted to deposit pretty much anything available in a Moodle VLE into a repository.

Luckily I don’t even have to write it up, because James has already written an excellent account on EL Blog, so please check his report out if you want to know more.

FOTE 2009 in Second Life

We used the opportunity to include a short presentation about our CLASM project, and I shared the platform for one session with James Ballard, our resident Learning Technologist and ace Moodle hacker. The full video and slides (with audio) of our talk are available from the FOTE website; the slides are also on Slideshare.

I was particularly pleased to make contact with Jane Secker of LSE, who knows more than most about CLA, and I am looking forward to discussing some issues with her, as we try to refine the work done on the CLASM plugins and produce a finished package. Jane also published an excellent account of the day’s events on her blog

The audience was a bit different from the JISC Information Environment crowd I’ve made presentations to before, so my talk was a very high-level overview of repository work in the sector, with a few ideas about where trends and technology seem to be leading us. One particular advantage I see is that interoperability between web applications should enable us to focus on using the “right” tools – portfolio, VLE, blog, repository, etc, maybe even VW – at each stage of the institutional/educational workflow, rather than using over-ambitious and over-complicated systems that try to do everything. “Small pieces loosely joined,” and all that.

Unfortunately, while the slides on the FOTE website include audio, the video there doesn’t include the slides, which robs the talk of some context. I have, by some dark means, managed to create a new version which combines the video and slides and upload it to YouTube. (To keep it short and relevant to DA Blog readers, I’ve only included my part of the presentation.)

We’ve had to lose a lot of material relating to past computing technologies that is now of limited or no value to us – stuff we were keeping really just because we had the space. That includes extensive documentation on IBM and CDC systems of the past, as well as DEC systems and a huge variety of micros. We’ve kept a few gems (the reference card for the SNUFF editor is a particular personal favourite) and discovered a few as well. They include what’s probably the earliest evidence of ULCC’s web presence in 1994. I’ll be writing about that soon over on the JISC-PoWR blog.

Many interesting discussions about repository approaches for the arts followed, both in the workshops, plenary debates, and over the rather smashing buffet lunch. One interesting direction the discussions took was in suggesting that while endeavours like KULTUR and PRIMO provide examples of ways to develop repositories for visual and performing arts, they don’t offer any kind of ready-made application for institutions wanting to create their own repositories with a minimum of fuss and cost. Is it possible that the benefits of such projects (particularly JISC-funded projects) would be greater if the outputs generated a reusable product rather than just a script or a recipe? Bill agreed to discuss this idea further within RSP, and I look forward to following it up soon.

But I digress. I’m here to write about some outcomes from Friday’s DPC board meeting. Encouragingly, it looks likely that the digital preservation award will return in November 2010, although some hurdles remain to be overcome. It’s quite possible that some aspects (such as eligibility or marking criteria) may change. Watch out for news late this year or early next. In the meantime, if you have views on what would make the awards more interesting or relevant to you, and particularly on what might encourage you to enter, do let me or the DPC know.

The joint Society of Archivists’s digital preservation roadshows (supported by DPC, TNA, Planets and Cymal) have been extremely popular, with some events over-subscribed. They are proving a great way to get basic, practical information about digital preservation tools and methods into the hands of working archivists and records managers. The problems, and the reception, sound reminiscent of similar work I did for the SoA about 10 years ago, as part of their occasional training days for newly-qualified archivists.

I’m also pleased to say that the Board approved a proposal to allocate more money to training scholarships in 2009/10, which can be used to support attendance at DPTP or other member-provided courses such as DC 101 (which is currently free.) We’re also looking forward to a joint training showcase in Belfast with the DCC’s DC 101, facilitated by JISC and PRONI, in September. More details will appear here and elsewhere when we have them.

We’re expecting an increased number of DPC techwatch reports in the coming year. The latest, released on preview to DPC members yesterday (2009-07-10), covers geospatial data, and there’s a long list of candidate topics for the next couple of years.

Finally, the board said thanks and farewell to its current chair, Ronald Milne, who is taking up a new post at the National Library of New Zealand next month. The Vice Chair, Bruno Longmore, will act as DPC chair until elections are organised for the AGM in November.

The JISC funded SNEEP project (Social Networking Extensions for EPrints) – part of the original JISC rapid innovation programme – aimed to provide a set of social networking tools for EPrints repositories. It ran for 6 months and ended in May 2008. Since the rather low key publication of the resultant EPrints plugin interest and uptake has been slowly but surely gathering momentum.

Today I am pleased to announce a couple of significant SNEEP related developments. Firstly , thanks to my colleague Ben Wheeler here at ULCC, SNEEP 0.3.2 released this week offers an automagic installer. This does away with the (slightly tortuous) manual install procedure that we suspect discouraged all but the hardier EPrints hac… I mean administrators.

You can download SNEEP 0.3.2 and/or read Ben’s post to the EP-tech mailling list. The download page is also a good place to see SNEEP in action.

PICT

I am also pleased to announce a new project (funded as part of the 2009 JISC rapid innovation programme) that aims to build on the SNEEP work to provide SNEEP-ish services to a broader range of web resources. The goal of the PICT project (Platform Independent Community Toolbox) is a lightweight javascript tool that can be deployed across an number of web resources (not just a repository) to encompass the web-based real estate of a given research community and provide that community with collaborative tools available at the on-line research coalface.

Effectively PICT will allow resource owners to offer

• tags
• comments
• notes
• other goodies

from their web page. The data gathered by these tools will be managed by a PICT server (probably run by a community-minded resource owner) and be available for cross referencing with other resources in a PICT community.

If all that is a bit difficult to picture, rest assured that demos will appear throughout the course of the project that should help to clear the murk.

Frank Steiner: I understand you’ve just recently taken on the post at DPC. What is your background in the field of digital preservation and how did you end up at the DPC?
William Kilbride: After my archaeology studies at Glasgow University and an MSc in computer applications I worked for the Archaeology Data Service at University of York. I started there in 1999 – which were early days in digital preservation – at least within archaeology. Part of our work involved raising awareness as well as ensuring long-term provision and access to archaeological research data. Because fieldwork can be very destructive, archaeologists have always had a close relationship with and respect for archives. The DPC came into existence at that time – also based in York. We had a lot of shared interests and quickly developed a close working relationship.

FS: So you basically switched camps and went to the other part of town to get into work in the mornings. (N.B.: DPC is based in York)
WK: No, I actually took a slight detour. For family reasons and because of some interesting work they were doing I moved to Glasgow in 2006 to work in Glasgow Museums as Research Manager.

FS: This sounds like there is more than one?
WK: Indeed, most people don’t know it, but the city of Glasgow owns one of the largest and most impressive civic collections in Europe, displayed in 13 museums across the Glasgow. Glasgow has a real love affair with its museums and although only about 2 percent of the collection is on display, a new research centre will soon provide access to the whole lot – like a massive reference library or public archive.It’s really innovative and shows a real commitment to access.

FS: So you went back to your archaeological roots, so to speak?
WK: Yes – but computing projects and the issue of digital preservation caught up with me once more. We developed online access to collections as well as trying to ‘make sense’ of the ever growing pool of native digital items which the collection contained. But the Executive job at the DPC became available and considering the history I had with DPC it was very attractive. I decided to apply, and now have an office in Glasgow University where I was already an honorary lecturer. So although DPC is based in York, my office is in ‘HATII’ – the Humanities Advanced Technology and Information Institute in Glasgow – which has an impressive track record in research and development with digital preservation. It’s a good place to work.

FS: I have spoken to both of your scholarship winners, who seemed very pleased to be given the opportunity to be funded to attend the most recent DPTP course. What other activities is DPC involved in to raise awareness of the importance of the preservation of digital material?
WK: You already mentioned part of our mission statement there. We are an agenda setting and enabling body with the ultimate goal to make our digital memory accessible in the future.

FS: I gathered from the competition, which you ran for the scholarships, that you are member’s only club. Is that right?
WK: Not really, we are a not-for-profit membership organisation, but by no means exclusive. Our members benefit through early bird rates at events, special discounts and preferred access to reports and such. Ultimately we share our reports and training with anyone who needs it. Members are able to share their work through the DPC and also able to set the agenda – to point us at issues they need resolved. Digital preservation is a topic for everyone from large commercial organisations, small charities down to each and every one of us. So it makes sense economically and intellectually if we work together.

FS: Interesting you should mention that. I realised there were a wide spread of backgrounds at the last DPTP.
WK: And I think that is one of the many benefits for people attending the course. Every time I present there is this sense of mutual problem solving, regardless which organisation the participants are from or what their background is. The chance for DPTP students to establish peer contacts and network with people trying to solve similar problems is something I like to see happening and it is also something we at DPC try to achieve.

FS: Can you shed some more light on the DPC’s involvement with the DPTP course?
WK: We helped ULCC develop the course with Cornell University, ADS, JISC and others, based on the general need within the community for training in digital preservation. I find it very satisfying to see some of the first students (in 2005) who went from looking for solutions to resolving their digital preservation issues to becoming significant leaders in the field – developing and applying solutions and sharing their experience with the community.

FS:Looking into the future, do you think there will be further scholarships by DPC?
WK: There’s no question about it, yes. There is a persistent need for digital preservation training and there are also growing expectations of what digital assets can do for organisations and individuals.

FS: Do you think the course itself has to change to stay relevant?
WK: Of course it does and it certainly has over the past 3 years. The field has developed quickly and I’m confident Patricia and the guys from ULCC will ensure the latest changes are reflected in the course syllabus. I think maybe there will be or could be a variation or targeting of DPTP, like ‘DPTP for Museums’, ‘DPTP for Publishers’ and so on. Although basic concepts and principles of digital preservation remain the same there is a trend of more specialised requirements which could be addressed by a more customised DPTP offering.

FS: William, thank you for your time and I hope to see you at the next DPTP course.
WK: Thanks for having me.

I took a smashing poster describing our work on Linnean Online and the SNEEP extensions for EPrints, and also spoke about these projects to the EPrints User Group sessions and had to endure the now inevitable Minute Madness. I was pleased to spot the SNEEP Comments plugin in use when Jessie Hey demonstrated EdShare, another of Southampton’s learning resource repository projects. It was also great to meet up again with Patrick McSweeney who has been tweaking SNEEP at Southampton, and discuss ways of keeping ongoing work on the plugins in sync. Regular readers may remember Patrick from OR08, and he cut an even more unforgettable figure this time.

The talk of the event seemed to be the relentless buzz around the unification of DSpace/Fedora Commons, engendering the new creation that is DuraSpace (and DuraCloud). This offers a lot of exciting possibilities that we’ll need to keep track of, though it won’t be the first repositories event that has offered us a surfeit of jam tomorrow… For now, for the curious, here’s the Duraspace FAQ.

By contrast, it’s slightly disappointing that, over the water, the EPrints user group seemed a tad under-subscribed. Features available in EPrints 3.1.x, and those imminent for 3.2, from cloud storage controllers and desktop folder visualisations to preservation support, promise quick wins for anyone wanting to push the repository model further: Les and the EPrints team waste no time in responding to the latest demands of the zeitgeist. All the same, informal discussions with users and non-users of EPrints suggested substantial resistance to its Perl-based core. Yet EPrints continues to push more configurability away from its Perl source: in the kind of repository-driven future oft foretold – from WordPress-type exensibility to modular service-oriented solutions – the underlying code base ought to become increasingly irrelevant as long as the package does what it says on the tin.

As usual it was great to meet some old friends, and lots of people for the first time. Memorably serendipitous (re-)discoveries included:

• Bibapp – “a Campus Research Gateway and Expert Finder”. There have been many attempts to integrate personalised, portfolio pages with repositories, and this looks like an effort worth investigating further, particularly as it claims to be repository neutral (and a good excuse to try out Ruby for real?).
• ParallelArchive – another variant on the repository model: “a personal scholarly workspace, a collaborative research environment, and a digital repository”. Run by Open Society Archives (OSA) at Central European University in Budapest – of particular interest to students of cold war and related issues
• E-Lis – still a superb multilingual collection of LIS resources, and undoubtedly the acid test of all EPrints internationalisation efforts
• MIT Open CourseWare – the mother of all OERs?
• The great Peter Sefton – great to meet him at last, at 6′ 7″, someone I can truly look up to. For a much more thorough account of the conference, see Pete’s Blog

I didn’t manage anything in the way of sightseeing, though the Aquarium seemed to be top of most locals’ list of recommendations, and we went there. Perhaps I should have made more of an effort to see the Civil War museum. For the visual record of OR09, content and context, you might like to see Jim Downing’s photos from the event, and the official photo OR09 set on Flickr.

The magic bullet holes the French made in Alicante

Well, as many of us know it doesn’t exist! This was part of my opening speech for the XV Jornadas de la Conferencia de Archiveros de las Universidades Españolas. This is the annual meeting of all Spanish university archivists. I spoke about “El perfil del archivero en el entorno digital”, or “the profile of the archivist in the digital world”. Most universities in Spain have an archivist, who also performs the role of records manager. Records manager as a profession doesn’t exist in Spain. Repositories existed in almost all universities but were on the whole in the libraries and not within the archives. There is a big divide between librarians and archivists in Spain also so not a great deal of exchange goes on between these sectors. Many questions concerned costs as well as approaches to preservation. An excellent book has been written in Spanish about digital preservation by Jordi Serra of the University of Barcelona, “Gestión de los documentos digitales: estrategias para su conservación” =” Electronic records management: strategies for long term preservation”.

A lot of discussion revolved around the struggle to convince the superiors in the organization of the importance of digial preservation; a lot of discussion about how access drives so much of what is going on, also the big problem of engaging our techy friends in the area of digital preservation and making them aware of the issue. The biggest universities such as Complutense and UNED (Universidad Nacional de Educación a Distancia.) were of course represented. UNED is the UK equivalent of the Open University with braches all over the world and is based in Madrid. A lot of discussion also took place about the involvement of non-anglo saxon countries in international fora such as ISO panels and ICA. While all these endeavours are important a certain amount of frustration is evident in relation to decision making and power!

Much talk and walking through the historical centre of Alicante, and in the interests of professional scholarship and in keeping with traditional Spanish culture I stayed awake until 3am catching the plane home the next day to rainy London.

Japanese Zen garden at Brunei gallery, School of Oriental and African Studies.

So another DPTP over! As presenters we felt it went really well. We again had a great group of people. The level of knowledge was very high and even so it seems the course really does help consolidate many levels of knowledge about digital preservation. For many it was OAIS and the class project seems to have helped put the theory into practice. One quote from the feedback:

‘Things really fell into place for me during this exercise and models started to make proper sense. Moved things from theory to practice.’

Overall the level of satisfaction with what we are providing is high.

‘Overall an excellent course. Bringing together so many disparate ideas and concepts and making sense of the muddle! Just hope we can move forward using the models. Excellent group too, good interaction and discussion – I got as much out of this element as from the taught content. Thank you so much all!’

We are now looking to developing links with the DCC as well as moving on to another stage of the DPTP. We will keep providing these 3 day courses with readjustments and updates but we are also looking at developing the modules into e-learning objects. Now all we need is funding!

Chiptunes are typically created using early personal computers or videogames and/or their soundchips. In that respect, they depend on technology preservation – the museum approach to digital preservation. Chiptune composers either use the systems as designed, programming them directly to create their music, or alter them in some way using techniques collectively known as ‘circuit-bending’, which makes the machines capable of producing sounds that they could not have originally produced. Some aspects of the chiptune scene utilise more modern synthetic techniques to recreate the sounds produced by these early chips – these are, in a loose sense, emulating the original systems, although not in a way that would allow you to use original software to create your sounds. But some adherents of the chiptune genre are going further, using the sounds of the systems themselves in their compositions.

The article which set my train of thought going covered Matthew Applegate’s (aka Pixelh8) concert in late March 2009 at the National Museum of Computing, which amongst other things used the electro-mechanical sounds produced by the Colossus and by hand-cranked adding machines. Now in this respect, as in many others, digital isn’t different to the analogue world. One could argue that found sounds have a long tradition in Western music, from Tchaikovsky’s use of cannon in the 1812 overture through to Leroy Anderson’s typewriter symphony and the work of the Boston Typewriter Orchestra, who eschew traditional instruments entirely and concentrate solely on percussive typewriter music, interleaved with the odd end-of-line bell.

But pixelh8’s work reminded me of some earlier, and unlikely, uses of computers to create music which I fear is already lost – although discerning listeners might well argue that we are all much the better for it being lost. One depended on the noise-making abilities of computer peripherals such as line and chain printers, the fiercest of which were capable of noise levels that drowned out conversation their vicinity. Devious users could create files which generated recognisable rhythms when printed on a specific printer. I’ve seen reference both to the “William Tell Overture” and “She’ll be Coming Round the Mountain” being produced by this method, which demanded detailed knowledge of the printer’s workings (but no programming knowledge – the effect was produced merely by printing a carefully-crafted file.) It seems that at least one recording of this exists, from the mid-1960s. Good ideas like this don’t die, or are reimagined by later generations, and 1999 brought us the Symphony for dot matrix printers.

These examples, in common with some of pixelh8’s work, are dependent on the relevant hardware still existing and being in working order in order to allow us to continue to perform them, or to create new compositions. That’s certainly a challenge for things like line printers, and I imagine the same is true of mechanical typewriters. These, however, are relatively simple mechanical devices and, so long as someone is still interested in maintaining them, they will be repairable for many generations to come, even if they are no longer being manufactured.

But other forms of of computer music aren’t so easily re-created. At about the same time as people were creating line printer music, others realised that the computer’s own electronics generated sound in an unplanned-for way. Almost all computers of the 1960s generated substantial amounts of radio frequency interference of a type that’s now illegal. The clock speeds of those computers meant that the interference fell mainly in the medium wave band, between 500 Khz and just over 1Mhz, exactly the right spot for any consumer AM radio to pick it up. (Present-day systems, if not shielded, would be generating interference for microwaves and mobile phones, not AM radios.) Enterprising programmers with time on their hands and dedicated systems to play with wrote programs which generated RF interference which approximated a tune, since the exact nature of the interference depended on the instructions being processed, the data being fetched, and probably the addresses from which it was being fetched, as well as the properties of the CPU circuitry.

I encountered one version of this, using one of DEC’s PDP10 processors (probably a KI10, although I can’t be sure) and the DECUS program library also contains a similar program for the PDP8. I’ve come across one reference to a recording being made, this time of the same technique on an IBM 1401, but the available example uses samples from a recording to make tunes rather than the original recording itself being musical.

All these examples illustrate, to different degrees, the digital preservation problem that pixelh8’s work brought to mind. The RFI music was exquisitely sensitive to properties of the original electronics. It wasn’t enough that the computer ran the same instruction set. DEC’s PDP10 series had four generations of processor – the KA10, KI10, KL10 and KS10 – each of which could have run the music program without error. But since each was entirely different at the circuit level, only one of those machines would produce the right sort of radio interference to generate the desired effect. Emulators of these systems now exist, and a number of manufacturers have produced hardware-level recreations such as the Toad 1, Foonly and Systems Concepts machines. But neither the software nor the hardware emulators will have the same radio interference properties, so again, although they will run the original program without errors, it won’t produce the desired effects.

In fact, it would be extremely difficult to produce an electronic equivalent of those systems again, and as technology moves on such problems will only get more difficult. The early systems, which used primarily discrete components (transistors, capacitors, etc) could be built again by sufficiently dedicated enthusiasts. But chip fabrication is another thing entirely. The music that those programs and systems produced is essentially unpreservable except by recording it at the time. The significant properties that emerge from the combination of software and hardware are extremely difficult to characterise, and no environment that allows the software to run will emulate the one property of the original environment that really matters.

Some stuff really, really can’t be digitally preserved. It’s good to remember that sometimes.

(And if you are aware of any other recordings of computer ‘music’ of this type, I and others would very much like to hear of them.)

[Updated to remove an embedded YouTube video which causes problems with the Iphone. Instead, you'll have to follow the link to see and hear some hardware-based chiptune hacking.]