Inspired by Rob Sanderson’s lightening talk at dev8D on Memento I decided to go for the bounty offered for writing a memento client. My tack was to enable a mediawiki instance to handle the Accept-Date protocol using an existing plugin. Then to write a little PICT tool that supplied a user interface by which users could specify a date and browse their PICT enabled mediawiki “from the past”… spooky!

Thanks to getting nowhere near the deadline (those involved with this project will be grinning at that), I got nowhere near the prize, but I did finish a prototype plugin and threw it up on a mediawiki instance for another project: CLASM-demo. CLASM is the name of project not a piece of conjurers’ onomatopoeia, so click on that link to see the PICT-memento client in action. A wiki with more pages would have been a better example, but at least it does have a lot of revisions.

“Would you want to put that in your repository?” asked Les. And this is a particularly extreme example where we can do some calculations to give us a fairly good answer. Scientific data centres and the researchers that use them have been considering this question for many years, and one way of looking at it is to see if the cost of recomputation exceeds the cost of storage over a particular time period. We’re assuming here that the initial question – is this worth keeping at all – has been answered at least vaguely positively.

Let’s look first at the cost of recomputation. Fabrice says the equipment used for this task cost no more than €2000. If we assume that it has a life of 3 years, that gives us a cost per day of €1.83. I’m avoiding the usual accounting practice of allowing for inflation, or lost interest on capital, in calculating the true depreciation value of the asset – there’s a number of different schemes and they all give similar results. I’ve just dividided the capital cost by the number of days of use we’ll get. But computers use electricity, and that costs money as well. Let’s assume this is a power-hungry beast that draws 400W and that power costs us 13.5¢ per kwH (which is what my domestic tarrif is if we assume a euro/sterling rate of €1.10 = £1 and 5% VAT.) That adds €1.30/day to the cost of running the system, for a total cost of €3.13/day.

Fabrice’s announcement says that it took 131 days of system time to calculate and verify his results, which gives a computational cost of €410.03 – which I’ll round to €410 since I’ve only been using 3 significant figures so far in the computations, and because there’s a lot of hand-waving involved in lots of these figures. So, we know how much it would take to recompute this result given the software, machine and instructions. (And the computational cost is likely to decline over time in the short term.)

The answer needs a Terabyte of storage. What will it cost to keep that in a repository? That’s a slightly more difficult question to answer, but we can give a number of figures that provide upper and lower bounds. SDSC quote $390/Tbyte/year for archival tape storage (dual copies), excluding setup costs and assuming no retrieval. Moore et al quote $500/year as a raw figure, obtained by dividing total system costs by usable storage within it. At current rates of $1 = €0.67, that gives us a cost of €261/year or €335/year. SDSC are likely to be at the cheap end of the scale. ULCC’s costs, given our lower total volumes, would be closer to €1500/year for a similar service (dual archival tape copies on separate sites) although that does include retrieval costs. Amazon’s AWS would be about €100/year for a single copy. You would want two copies, so it’s twice that, and the cost of transferring the data in would be about 25% more than the storage cost. Since I haven’t factored in ingest costs for any of the other models, I’ll ignore it for AWS as well. (And yes, AWS isn’t a repository, and there’s no metadata, and… This is a back-of-the-envelope calculation. It’s a small envelope.)

Which means, at a very rough level and ignoring many pertinent factors, that after about two years of storage in the repository, we would have been better off recalculating the data rather than storing it. There’s a lot of assumptions hidden there, however. For one, we’re assuming that this data will rarely, if ever, be required. If many people want it, the recalculation cost rapidly becomes prohibitive (and so does the 131 days they have to wait for their request to be satisfied!)

One of the other problems is more subtle. I said that, in the short term, recalculation costs would be likely to fall as computational power becomes cheaper. The energy costs involved will rise, of course, but there’s still a significant downward trend. But after a sufficient period of time, it becomes non-trivial to reconstruct the software and the environment it needs in order to allow the computation to happen. Imagine trying to recalculate something now where the original software is a PL/I program designed to run under OS/360. It’s not impossible by any means, but the cost involved and expertise required is non-trivial. At least with our example we won’t have any doubts about whether the right answer has been produced – the computation of π produces an exact, if never-ending, answer. Most scientific software doesn’t do this and the exact answers produced can depend on the compiler, the floating-point hardware, mathematical libraries and the operating system. Over time, it becomes harder and harder to recreate these faithfully, and we often don’t have any means of checking whether or not we have succeeded. (Keeping the original outputs would help in this, of course, but that’s exactly what we’re trying to avoid.) That’s part of the problem that Brian Matthews and his colleagues examine in the SigSoft project and there’s still a great deal of work to be done there.

So have we answered Les’s question ? My feeling is that in this case we have – there’s a fair amount of evidence that suggests that keeping this particular data set isn’t cost-effective. But in general, the question is far harder to answer. Yet we must strive harder for more general answers as the cost of not doing so is not trivial. Even if money did grow on trees, it still wouldn’t be free and at present we need to be very careful how we use it.

My naïve thinking in the area has been characterised by the assumption that the process is rather static or linear, and that the problem we’re facing is broadly the same every time; migrate data from a format that’s about to become obsolete or unsupported, onto another format that’s stable, supported, and open. MS Word document to PDF or PDF/A…now that, I can understand!

In fact, I learned at least two ways of thinking about formats that hadn’t occurred to me before. One simple one is costs; some formats can cost more to preserve than others. This can be calculated in terms of storage costs, multiplied over time, and the costs associated with migrations to new versions of that format. For example, we’ve tended to pin our faith on the TIFF format for images for many reasons, but there’s a high storage price to be paid for all that wonderful losslessness. This may be one reason why the DP world is looking with more favour on the JPEG2000 format, which is ‘virtually’ lossless and smaller in size.

Secondly, the problems of preserving digital data which doesn’t actually have a specified stable preservation format. Chris Puttick of Oxford Archaeology gave a vivid description of the problems he’s facing with CAD and GIS files, where the data can’t easily be tied to a single format in the first place (nor can a stable format for migration be identified). As the NLA put it on their PADI page, “At present there is little dealing specifically or comprehensively with the preservation of this particular type of data, although some aspects of database preservation are applicable to GIS. Some long term preservation issues include a lack of open source formats and metadata standards, large data volume and complex data objects.” Puttick suggests that his data doesn’t really perform at all unless it’s operated within a very specific environment of hardware and software. How do we preserve an environment? This appears to be quite a distinct preservation problem and much harder to solve than Word to PDF, to put it mildly.

William Kilbride suggested that such cases (and websites too, arguably, because they are time-based) are more like a stream of data – a handy image which conveys something about the dynamic of such information packages, and showing us that it’s much harder to nail them down into a single format. You can never step into the same river twice.

The JISC funded SNEEP project (Social Networking Extensions for EPrints) – part of the original JISC rapid innovation programme – aimed to provide a set of social networking tools for EPrints repositories. It ran for 6 months and ended in May 2008. Since the rather low key publication of the resultant EPrints plugin interest and uptake has been slowly but surely gathering momentum.

Today I am pleased to announce a couple of significant SNEEP related developments. Firstly , thanks to my colleague Ben Wheeler here at ULCC, SNEEP 0.3.2 released this week offers an automagic installer. This does away with the (slightly tortuous) manual install procedure that we suspect discouraged all but the hardier EPrints hac… I mean administrators.

You can download SNEEP 0.3.2 and/or read Ben’s post to the EP-tech mailling list. The download page is also a good place to see SNEEP in action.

PICT

I am also pleased to announce a new project (funded as part of the 2009 JISC rapid innovation programme) that aims to build on the SNEEP work to provide SNEEP-ish services to a broader range of web resources. The goal of the PICT project (Platform Independent Community Toolbox) is a lightweight javascript tool that can be deployed across an number of web resources (not just a repository) to encompass the web-based real estate of a given research community and provide that community with collaborative tools available at the on-line research coalface.

Effectively PICT will allow resource owners to offer

• tags
• comments
• notes
• other goodies

from their web page. The data gathered by these tools will be managed by a PICT server (probably run by a community-minded resource owner) and be available for cross referencing with other resources in a PICT community.

If all that is a bit difficult to picture, rest assured that demos will appear throughout the course of the project that should help to clear the murk.

Calls in the literature have advocated that blogs, as potentially valuable additions to the human record, are worthy of stewardship and long-term preservation. It’s hard to argue with this, in the light of recent widely publicised use of blogging by President Obama, the Number Ten Downing Street blog, the blogging and tweeting activities of certain celebrities, and so on. But of course anyone can have a blog (and it sometimes seems that almost everyone does). Blogs are phenomenally useful, and have a wide variety of applications: individual or corporate announcements and publications; time-stamped records of activities; public or private figures writing as themselves or pseudonymously; public and private journal and discussion spaces for students and a wide range of communities. A blog may well be just for Christmas (or any kind of project) no less than a traditional diary or journal. It is in essence, after all, just a sequential log (“blog” is merely a geeky contraction of “web log”). Blogging is therefore not so fundamentally alien: Captain Cook kept a log (and so will Captain Kirk). Rather than make it sound very technical and complicated, let’s start by taking some comfort in what’s familiar.

Should all blogs be preserved? Which blogs should be preserved? The DPE Briefing Paper highlights (as all good preservation advice must) the issue of selection. How do we decide which blogs to preserve? To start with, the remit of any archival body (national, municipal, corporate, specialist, etc.) is itself self-selecting. It’s unlikely any archivist at NARA, TNA or any other archive would ever have to begin their deliberations with “all blogs”. Of matters of authenticity and attribution, the nature of the archive undertaking the preservation will, equally, dictate whether knowing the true identity is essential or not. Common sense should prevail. Clearly for records of official activities, such as the Downing Street and Obama blogs, reliable provenance and authenticity are critical. On the other hand, satirical blogs by “Gordon Brown” or “Vladimir Putin“  are clearly not records of official activities, but they still may merit preservation as publications and/or as interesting artefacts: as with many fictional or satirical diaries or letters, the true identity of the author may or may not be material (Beachcomber, Henry Root, Primary Colors).

Is preservation of content, regardless of context, sufficient? This is one of the most significant questions raised by the briefing paper. Has the Web changed our expectations of context? Once a descriptive catalogue or historical account seemed sufficient to establish context. Digital media now tempt us into thinking that, for any object, we should automatically gather (a copy of)  everything linked-to, and perhaps everything that links to, and so on. We can leave a computer to follow the endlessly forking paths, and create a virtual machine environment that will always render the gather materials faithfully – all without any manual intervention. Will that provide context, or make the preserved content understandable? Can we do without expert researchers and information professionals to scope, select and provide commentary where necessary?

The inability to capture and preserve the design and features of the blog contradicts defining attributes of a blog. In fact it’s not impossible to capture and preserve many if not all of the design features of a blog. The JISC-PoWR blog has been harvested by UKWAC with most of its features intact (experiments at ULCC have suggested that, with a little care, WordPress blogs are particularly well-suited to harvesting). But in general, let’s accept (as the Blogbackuponline people do) the overriding significance of the textual content over and above the bells and whistles of the web presentation. If there is one particularly interesting feature of blogs, that does set them apart from conventional web content, it is that they are generally intended for delivery and consumption in two distinct formats: by HTML web page, and by RSS/Atom XML newsfeed. The newsfeed version is bare-bones hypertext stripped of all the styling, bells and whistles of the web version, and will be read in many contexts, from online newsreaders and aggregators, like Netvibes or Bloglines, to desktop clients, such as Mozilla Thunderbird, as well as on mobile phones, and by automatic syndication to other sites.

Blogs, including posts and comments, may be intentionally edited or deleted. Looked at in isolation, editability is a significant property of blogs in everyday use. But, as Chris Rusbridge observed , “editability, while vital for some kinds of re-use, is not essential for conveying the information essence of the objects”. Any preserved object we have only in the state it was when taken into archival custody (though we may also capture and preserve records of how it changed). Preservation schedules for blogs will need to decide how important this really is: when is it and when is it not important to preserve an edit that no one ever saw? (A transactional approach – only capturing page impressions users request – is one way to address this without excessive redundancy.) However, if anything, this volatiliy of blog content is a more serious issue for those citing an active blog (who may find their quotations no longer match): a copy in an archive ought to be fixed, and reliably citable. If anything, this would suggest that creating stable archives of valuable blogs for the record is essential.

James Currall rightly warned us against making an undue fetish of digital media, at the expense of the information (and entertainment) which they convey; arguably as dangerous is the fetishisation of specific manifestations of digital media. How different, really, are blogs from other publications or applications built on the Web platform? Blogs are essentially web pages and don’t exist in a void. The “problem” of hypertext links to possibly unstable objects in remote locations is the bête-noire of all efforts to tame the Web. “Not in archive” is, and will ever be, a familiar message to regular users of archive.org. But we can perhaps take heart that many archives of correspondence tend to be one-sided too.

Sometimes thinking about these things in the abstract is no substitute for actually doing them for real. Perhaps the best advice for anyone tasked with preserving blogs is, first of all, to try keeping one yourself for a few days or weeks. Then it only takes a few minutes to try out BlogBackupOnline; or check how your favourite blog is faring in the Internet Archive , the European Web Archive, or the UK Web Archive; or even download HTTrack or Wget and try harvesting something yourself.

Then it’s time to get Twittering…

In fact, as I was relieved to note, this group of standards isn’t entirely new to the digital preservation community, and the authors are also aware of general DP standards such as OAIS. They derive from a group of standards known as STEP (Standard for Exchange of Product Model Data), codified in ISO 10303. STEP came up in a presentation and discussions at PresDB’07, although it’s been in development since the turn of the century at least. But STEP is a huge family of standards, and this particular work appears to have emerged from specific work going on in the aerospace industry, which is known to have had concerns about long-term survival of CAD data for some time.

LOTAR is one of the results of this concern, and a presentation by Pierre Duchier of AIRBUS at an Aristote conference in April this year gives a clear picture of the concerns of industry and the approach they are taking. LOTAR and related work were also covered at a invitation-only event at Bath in 2007, Atlantic Workshop on Long-Term Knowledge Retrieval whose attendees included many names familiar from the DCC and the repository community.

Overall, this is reassuring. Here is a set of digital preservation standards being developed and driven by concerns in industry, but where the work is taking place in dialogue with with the academic research and development community. CAD data in particular has long been a concern at the Archaeology Data Service, which has significant holdings of Autocad files, and it will be interesting to see to what extent LOTAR has relevance for such activities. In the meantime, I can see I’ve got a lot of reading to do to catch up, but it would be interesting to hear from others who have more insight into the possible use of these standards outside the aerospace industry.

Incidentally, BS EN 9300 is not to be confused with BS EN ISO 9300, an entirely different standard concerned with “Measurement of gas flow by means of critical flow Venturi nozzles“.

I just spotted a posting from a fellow Antipodean, made to the Australian Archivists (aus-archivists) listserv, which has certainly raised some interesting questions surrounding web 2.0 technologies and their impact on the Archive sector…. Perhaps a debate well worth monitoring, and further exploring here, within the realm of web 2.0 itself?

See Australian Archivists listserv posting below:

Archival institutions are increasingly using social networking sites, tagging (folksonomies), blogs, wikis, and Flickr to promote their collections. Does anyone know any studies evaluating this phenomenon in the archival setting? I don’t mean the this-is-how-we-did-it, isn’t-it-exciting or look-how-many-hits-we’re-getting articles. I mean thoughtful consideration of the value of these tools and the effects they are having on our work. I’d also love some discussion on the list.

Why we have decided to use these tools? What benefits have they brought? What kind of new audiences are they attracting? How long do these audiences stick around? Is the resource taken to sustain these ‘relationships’ worth it? Do these audiences engage with us beyond the social network stuff? Do they use our databases? come in and use our collections? order quality copies? Does it matter if they don’t? How is our adoption of these tools affecting what material we choose to process and promote? With user-generated content and tagging are our formal documentation skills, cataloguing standards, thesauri, etc passé? Does mashing trivialise our research collections? Any observations or leads to articles welcome. [ Helen Yoxall, Archives Manager, Registration and Collection Management, Powerhouse Museum, PO Box K346, Haymarket NSW 1238, Australia, URL: www.powerhousemuseum.com/archives/index.asp]“

After a quick (and rudimentary) search, I stumbled across a few sources that may be of interest:

• UKOLN have a presentation “Web 2.0: Implications For The Cultural Heritage Sector”
• The Archives & Museum Informatics Museums and the Web website, includes a paper “Towards Community Contribution: Empowering Community Voices On-Line” (Angèle Alain, Library and Archives Canada, Canada; Michelle Foggett, The National Archives of England and Wales, UK). It refers to Web 2.0 and community involvement in museums, libraries and archives e.g. the Moving Here project which has sought to “break down barriers to the direct involvement of minority ethnic groups in sharing their history on-line” and is among other projects keen to “embrace social networking in future to give users a higher profile voice to enable their knowledge to be passed down to the next generation”. (However, “specialised and appropriate training” was identified as crucial to tackling the barriers,[such as the 'digital divide' itself]). On the related MuseumsComputerGroup website is the article abstract “Museums and Web 2.0: Connections + Community” (by Jennifer Trant, Archives and Museum Informatics), noting both the possibilities, and challenges, surrounding the adoption of web 2.0.
• Also of interest are the web 2.0 blogs: Library 2.0 network, Library 2.0 gang, Museum 2.0; and while I couldn’t spot an Archives 2.0-specific blog anywhere, there was an interesting posting on the Archives Hub blog and on ArchivesNext (which, as one of its aims, invites bloggers to explore “Web 2.0 applications and discussing their applicability to archival institutions”).
• Our own dablog has highlighted one dimension of the web 2.0 impact (inspired by UCL’s Dr Andrew Flinn) i.e. in relation to urgent calls to preserve the heritage outputs of web 2.0, due to “the transient history of the increasing number of minority/dissenting voices, whose heritage is only documented via websites, blogs, wikis and social software”.
• In true folksonomic Web 2.0 style, we can also see what’s been tagged as Archives2.0 in del.icio.us.

Know of any other sources that discuss the impact of Web 2.0 on the Archives Sector? Or would you like to share your opinions in response to the questions posed by Helen Yoxall (above)?

The proprietary nature of RAW formats (an emerging OpenRAW standard notwithstanding) and the relative newness on the block of JPEG 2000 would both tend to bolster the longstanding preference for TIFF, but as Chris’s posts make clear, each preservation project should nevertheless weigh the options based on its own requirements and resources.

If in doubt, the “keep everything” approach is attractive, as ever, but – in spite of the old mantras about ever-cheaper filestore – the implications for storage space and management are potentially very costly once one enters the world of Terabytes and Petabytes. In one example, Sean Martin of BL concludes that

probably only a small amount of additional value is created for the additional expense approaching £200K. This leads to the question “if we had £200K on what would we spend it?” and probably the answer is “not in this way”.

The posts contain many interesting examples of costings, based on the experience of BL, SDSC and others. I won’t even attempt to summarise (Chris’s summary of) them here, but I hope that this post will still be available for me to consult next time I have to dabble in the murky science of DP costings.

In some instances, the case for preserving the RAW image may nevertheless be compelling. One can imagine that the risk of missing a new planet or virus (or identifying a non-existent one), or other potential infelicities in scientific and medical imaging, is not worth contemplating. By contrast, it’s hard to see what value might be added to a collection like the Linnean Society’s by keeping raw camera data alongside the TIFFs, and more than doubling the amount of storage capacity required.

It’s for making decisions like this that the intellectual exercise of identifying significant properties and the needs of all stakeholders – creators, curators and users – seems particularly essential.

Elements of the standard would benefit from knowledge of standards such as OAIS, which try to address long-term accessibility in a much more rigorous fashion. Comments close next Monday, June 30th, so if you have not yet commented but are concerned by any of this, do so now.

Authenticity is a key requirement in any digital preservation application, and it’s not a particularly novel one either as this post by Maureen Pennock makes clear. One of the six mandatory responsibilities which OAIS places on an archive deal with authenticity and traceability, and this was reflected virtually unchanged in the TRAC methodology, in which one of the audit checkpoints was that the repository

… enables the dissemination of authentic copies of the original or objects traceable to originals.

That choice of words is important: most repositories aren’t dealing with use cases that involve legal evidence on a daily basis, and for most users it isn’t necessary to provide dissemination copies of material that can be taken straight into a courtroom. Sometimes, doing so places extra costs on the dissemination process that aren’t needed. Other repositories may find it easier to make all copies ‘authentic’, or ‘capable of being traceable to originals.’ But we simply wanted to ensure that repositories were capable of doing this when it was required. If they could not, it called into question their own confidence in the integrity of their holdings.

BS 10008 deals with a subset of the problem, one suitable for document-oriented problems, and it isn’t at all specific about the mechanics of how it is achieved. The standard is going to be useful and it’s likely to have more weight as a standard (to which it’s easier to force compliance) than as a code of practice. But there’s areas that are worthy of comment from the DP community. Take, for instance, this definition of migration:

transfer of electronic information from one storage media to another

That’s certainly not the usual use of the term – I would use either ‘refreshing’ to refer to this action, or ‘media migration’ if I wanted to make it clear that no change in format or encoding was involved but still wanted to talk of ‘migration.’

The document is open for public comment until June 30th, 2008. You can view the draft and make your comments using BSI’s online review website (Incidentally, this is the first time I’ve used the online review process, and it works a treat. I last reviewed a standard about 10 years ago, when it was still a very paper-based process, although one could at least submit comments by email then. This is a great improvement. And while you’re there, you can take the opportunity to review such upcoming standards as BS 8507-1 for “Close Protection Services” – otherwise known as bodyguards, methinks)