If you haven’t seen it, can I recommend Kristen Snawder’s recent post on the Library of Congress Digital Preservation blog, Digitization is different than digital preservation. Kristen reiterates familiar points about the long-term commitment necessary for serious digital preservation, contrasted with the quick hit of a scanning project. “In the hurry to meet user expectations, institutions may scan large quantities of materials without having a solid plan for preserving the digital images into the future.”

However another recent find on the Web compels me to make an additional point, namely that we might do equally well to differentiate between scanning and digitisation. Anyone can set to work with a scanner and create a bunch of digital images – but that barely scratches the surface of what I think we should be expecting of a digitisation project in 2011.

First and foremost, we need metadata: the more the merrier, but something at least. Even if we expect to come back later and polish it up (once the images can be browsed and examined on screen). In the absence of any established metadata profiles for a project, at least try to cover as many Dublin Core elements as possible – title, creator, date, subject/keywords… Images, in particular, may prove tricky or time-consuming to find again, especially once there are thousands of them on a disk. We should probably keep the metadata in a database, and perhaps additionally store metadata with the objects. This can be as XML or plain text files stored alongside the digital images, or embedded in the files we create (many common file formats – TIFF, JPEG, MPEG, PDF – support metadata embedding, and there are many free tools available to help).

There is yet more, though, that we should be doing, particularly when we are scanning text-based objects (articles, books, magazines, reports, etc). Most importantly, we really should try and extract the text from the image if possible. ^[1]

My recent web find was the teaching blog of Dr Toine Bogers at the Royal School of Library and Information Science (RSLIS) in Copenhagen, Denmark. One fascinating post describes a Lab Session exercise, From OCR To NER, a set of comparatively simple command-line processes to get the most out of a scanned-text project.

Toine’s post walks us through the process. Once the article is scanned, we should apply some OCR. The exercise goes further and also describes the use of tools to clean up and spell-check the resulting OCR’d text. This will, at the very least, result in a separate text file, hopefully containing a fairly accurate version of the article text. Finally, the cleaned-up text can be submited to a Named Entity Recognition service. Toine’s exercise uses NER tools at University of Illinois. (We’ve been using similar functionality provided by OpenCalais and GATE for our AIM25 Open Metadata project.)

Why do all this? The most important, instant, result of this is that we can now easily index our article for full-text searching – in a local repository system, such as EPrints or DSpace provide – and of course by Google. None of this is possible if we leave the scanned image as just that – an image.

Another  side-effect of any successful OCR outcome, is that the text is now free to be re-flowed. This means that we might consider sharing it with users in a variety of forms enhancing usability and accessibility.

It’s important not to confuse preservation formats with formats for access and dissemination. You probably will have your scanned image masters in TIFF, RAW, JPEG2000, PostScript, SVG. None of these are likely to be of much use to your users over the Web. Not only are the formats not widely supported by Web browsers, but most users probably don’t need or want your master image. If it’s a high-resolution scan of a 100 page book, they might be looking at 100Mb download, or worse – slow to load, and probably slow to render and navigate.

Time taken thinking what formats will give users the best experience is time well spent. What platforms might they want to use now and in the foreseeable future? It’s less than 18 months since Kindle3 made e-book readers affordable, and the Ipad made them sexy. E-books look and function very impressively on both platforms (albeit in different ways): for an overview of some of the benefits of the EPUB format, see Martin Fenner’s post Beyond The PDF… is EPUB. PDF outputs may yet have their uses, if users can at least search for text within them. The point is that only with properly digitised text, do these kinds of accessibility options become possible.

Even image collections can also be disseminated as E-books – nice offline items some users might care to flick through on their tablet computers, possibly even smartphones. I’ve demonstrated how we can create OJS XML from EPrints XML on-the-fly with XSLT: since EPUB and Mobi/Kindle are XML-based formats, we should be able to do something similar to create e-books using repository APIs. Also, by using appropriately sized images in dissemination formats (Ipad screen is 1024x768px; Iphone4 is 960x640px) we can not only ship our users a sensibly-sized download, we can protect any capital we may have in the master images, without having to resort to ugly tricks like watermarking. (Giving users full-size, high-res images with embedded watermarks seems to me the worst of all worlds.)

Therefore I’d suggest that, in order to get the best out of a digitisation project, consider what would you like to see at the end of the project – and, more importantly, what would give your users the best experience, or even win you new users? Ask around, do some tests, with users if possible, and get an idea how they want to use the materials and how they will get the best out of them. Maybe there are comparable projects and systems that you admire, with features you’d like to be available for your collection. What about in five or ten years’ time: will your current project outputs help or hinder longer term accessibility goals?

This kind of vision of is essential. Without some conception of the end result, how the materials will be used and managed most effectively, all the scanning in the world isn’t going to amount to a successful digitisation project.

At an atomic level, a “blog” comprises “blog posts”, which are continually added to the blog corpus: that is the dynamic essence of a blog, and distinguishes it from old-fashioned, largely static Websites and hypertexts in which little content changed between major update iterations, which process was probably more akin to “publishing a new edition” in the world of non-digital publications.

The blog also displays, as part of its frame, other graphical and functional elements (sidebars, widgets, “blogrolls”, etc) which may themselves contain dynamically updated, constantly changing information. These can be added, removed, amended and rearranged at will by the blog author/editor. Blog posts that were “published” in the context of one set of framing elements, will persist through subsequent versions of that framework.

Similarly with design (layout, colours, mastheads, etc), though the persistence tends to be longer, the informal nature of blogs means that these may be easily changed by the blog editor/author, and are thus more volatile than a typical “corporate” website. Again, blog posts may persist, unchanged in themselves, through many iterations of the blog site design and layout.

A simple view of blog elements and their temporal relationship

This very simplified visualisations suggests where we might start conceptualising key elements of a blog. It indicates that they iterate over time, but in the cases of Design, Posts and Widgets (as we’ll call them for brevity), according to independent schedules. While Posts and Comments persist in the online view of a blog, designs and widget arrangements are overwritten.

With my earlier ArchivePress project we deliberately overlooked preservation of the blog’s framing elements, and (given the much smaller scope of that project) established an acceptable rationale for doing so. The challenge for BlogForever is to find a solution to  precisely these issues. Unless we were simply to adopt the snapshot approach of Heritrix-based web archiving initiatives (e.g. Wayback/archive.org, UK Web Archive), we need to ensure the BlogForever repository supports a degree of granularity that can capture, describe and preserve atomic blog objects in a way that reflects the particular interdependencies, in order to understand and preserve them authentically, and permit the many possible authentic and valid “time slice” views and analyses that users of the archive will need.

(I appreciate, by the way that these objects themselves are compound objects, so not strictly “atomic”: but the same is also true of atoms, as our CERN colleagues can attest!)

My naïve thinking in the area has been characterised by the assumption that the process is rather static or linear, and that the problem we’re facing is broadly the same every time; migrate data from a format that’s about to become obsolete or unsupported, onto another format that’s stable, supported, and open. MS Word document to PDF or PDF/A…now that, I can understand!

In fact, I learned at least two ways of thinking about formats that hadn’t occurred to me before. One simple one is costs; some formats can cost more to preserve than others. This can be calculated in terms of storage costs, multiplied over time, and the costs associated with migrations to new versions of that format. For example, we’ve tended to pin our faith on the TIFF format for images for many reasons, but there’s a high storage price to be paid for all that wonderful losslessness. This may be one reason why the DP world is looking with more favour on the JPEG2000 format, which is ‘virtually’ lossless and smaller in size.

Secondly, the problems of preserving digital data which doesn’t actually have a specified stable preservation format. Chris Puttick of Oxford Archaeology gave a vivid description of the problems he’s facing with CAD and GIS files, where the data can’t easily be tied to a single format in the first place (nor can a stable format for migration be identified). As the NLA put it on their PADI page, “At present there is little dealing specifically or comprehensively with the preservation of this particular type of data, although some aspects of database preservation are applicable to GIS. Some long term preservation issues include a lack of open source formats and metadata standards, large data volume and complex data objects.” Puttick suggests that his data doesn’t really perform at all unless it’s operated within a very specific environment of hardware and software. How do we preserve an environment? This appears to be quite a distinct preservation problem and much harder to solve than Word to PDF, to put it mildly.

William Kilbride suggested that such cases (and websites too, arguably, because they are time-based) are more like a stream of data – a handy image which conveys something about the dynamic of such information packages, and showing us that it’s much harder to nail them down into a single format. You can never step into the same river twice.

Chiptunes are typically created using early personal computers or videogames and/or their soundchips. In that respect, they depend on technology preservation – the museum approach to digital preservation. Chiptune composers either use the systems as designed, programming them directly to create their music, or alter them in some way using techniques collectively known as ‘circuit-bending’, which makes the machines capable of producing sounds that they could not have originally produced. Some aspects of the chiptune scene utilise more modern synthetic techniques to recreate the sounds produced by these early chips – these are, in a loose sense, emulating the original systems, although not in a way that would allow you to use original software to create your sounds. But some adherents of the chiptune genre are going further, using the sounds of the systems themselves in their compositions.

The article which set my train of thought going covered Matthew Applegate’s (aka Pixelh8) concert in late March 2009 at the National Museum of Computing, which amongst other things used the electro-mechanical sounds produced by the Colossus and by hand-cranked adding machines. Now in this respect, as in many others, digital isn’t different to the analogue world. One could argue that found sounds have a long tradition in Western music, from Tchaikovsky’s use of cannon in the 1812 overture through to Leroy Anderson’s typewriter symphony and the work of the Boston Typewriter Orchestra, who eschew traditional instruments entirely and concentrate solely on percussive typewriter music, interleaved with the odd end-of-line bell.

But pixelh8′s work reminded me of some earlier, and unlikely, uses of computers to create music which I fear is already lost – although discerning listeners might well argue that we are all much the better for it being lost. One depended on the noise-making abilities of computer peripherals such as line and chain printers, the fiercest of which were capable of noise levels that drowned out conversation their vicinity. Devious users could create files which generated recognisable rhythms when printed on a specific printer. I’ve seen reference both to the “William Tell Overture” and “She’ll be Coming Round the Mountain” being produced by this method, which demanded detailed knowledge of the printer’s workings (but no programming knowledge – the effect was produced merely by printing a carefully-crafted file.) It seems that at least one recording of this exists, from the mid-1960s. Good ideas like this don’t die, or are reimagined by later generations, and 1999 brought us the Symphony for dot matrix printers.

These examples, in common with some of pixelh8′s work, are dependent on the relevant hardware still existing and being in working order in order to allow us to continue to perform them, or to create new compositions. That’s certainly a challenge for things like line printers, and I imagine the same is true of mechanical typewriters. These, however, are relatively simple mechanical devices and, so long as someone is still interested in maintaining them, they will be repairable for many generations to come, even if they are no longer being manufactured.

But other forms of of computer music aren’t so easily re-created. At about the same time as people were creating line printer music, others realised that the computer’s own electronics generated sound in an unplanned-for way. Almost all computers of the 1960s generated substantial amounts of radio frequency interference of a type that’s now illegal. The clock speeds of those computers meant that the interference fell mainly in the medium wave band, between 500 Khz and just over 1Mhz, exactly the right spot for any consumer AM radio to pick it up. (Present-day systems, if not shielded, would be generating interference for microwaves and mobile phones, not AM radios.) Enterprising programmers with time on their hands and dedicated systems to play with wrote programs which generated RF interference which approximated a tune, since the exact nature of the interference depended on the instructions being processed, the data being fetched, and probably the addresses from which it was being fetched, as well as the properties of the CPU circuitry.

I encountered one version of this, using one of DEC’s PDP10 processors (probably a KI10, although I can’t be sure) and the DECUS program library also contains a similar program for the PDP8. I’ve come across one reference to a recording being made, this time of the same technique on an IBM 1401, but the available example uses samples from a recording to make tunes rather than the original recording itself being musical.

All these examples illustrate, to different degrees, the digital preservation problem that pixelh8′s work brought to mind. The RFI music was exquisitely sensitive to properties of the original electronics. It wasn’t enough that the computer ran the same instruction set. DEC’s PDP10 series had four generations of processor – the KA10, KI10, KL10 and KS10 – each of which could have run the music program without error. But since each was entirely different at the circuit level, only one of those machines would produce the right sort of radio interference to generate the desired effect. Emulators of these systems now exist, and a number of manufacturers have produced hardware-level recreations such as the Toad 1, Foonly and Systems Concepts machines. But neither the software nor the hardware emulators will have the same radio interference properties, so again, although they will run the original program without errors, it won’t produce the desired effects.

In fact, it would be extremely difficult to produce an electronic equivalent of those systems again, and as technology moves on such problems will only get more difficult. The early systems, which used primarily discrete components (transistors, capacitors, etc) could be built again by sufficiently dedicated enthusiasts. But chip fabrication is another thing entirely. The music that those programs and systems produced is essentially unpreservable except by recording it at the time. The significant properties that emerge from the combination of software and hardware are extremely difficult to characterise, and no environment that allows the software to run will emulate the one property of the original environment that really matters.

Some stuff really, really can’t be digitally preserved. It’s good to remember that sometimes.

(And if you are aware of any other recordings of computer ‘music’ of this type, I and others would very much like to hear of them.)

[Updated to remove an embedded YouTube video which causes problems with the Iphone. Instead, you'll have to follow the link to see and hear some hardware-based chiptune hacking.]

I began the day by chairing the final meeting of RPAG(the repositories and preservation advisory group.) We had a short meeting mainly to follow up on discussions we had been having on how the group had operated and how JISC might make use of advisory bodies in future. Those who expressed an opinion all felt it had been useful to them, but we all had concerns about how our time, and the JISC Executive’s time, might have been used more effectively. Future advisory groups may try to split responsibility for some areas into smaller working groups. All were agreed that the face-to-face meetings were invaluable, but we weren’t all agreed on which online technology would be best to use in between times. Enthusiasts for tools like ideascale were matched by those who found them unusable.

The meeting proper opened with some background and perspective from Rachel Bruce and Neil Grindley of JISC and myself. I tried – partly seriously, but without much expectation of accuracy – to give a one-line summary of what each project set out to do. But there were two things I meant to say which I failed to do. One was to look forward to the theme of day 2 (Value) and stress that repositories are not ends in themselves, but need to be thought of in terms of value, impact and benefits to someone. The second point I omitted was to remind us that , for innovation projects, failure in one sense can still mean success, as long as we understand the nature of the failure and are able to use it to improve and adapt future work. Not achieving what you set out to do is disappointing. Analysing the reasons for that and making sure others are aware of them can be of great value.

But it was the rest of the event that provided greatest interest. The discussion sessions on text mining, research data, teaching and learning repositories and more; presentations from projects from stakeholder, developer and other perspectives; posters and demos from many of the projects; and the fever of activities in the ideas room, which deployed technology ranging from post-it notes upwards to catalyse, capture and refine ideas from the attendees. These activities gave the event much more of a participatory feel – everyone became a contributor rather than being a consumer.

I learned a few things over the course of a day or two, most of them unexpected. David Flanders (via Chris Rusbridge) passed on the neat idea of feeding funding proposals through Wordle before marking them. That’s what ULCC’s AIDA project looked like. Perhaps you ought to try the same with your proposals prior to submitting them?

I learned that talking unprepared and unscripted to a video camera doesn’t produce great results unless you’ve had practice or training – neither of which I’ve had. I knew that in an abstract sense and now have the unfortunate experience to back it up. But Andy McGregor and Dave Flanders did capture some other people talking far more sense than I did and far more clearly, and you can see the results on the dev8D youtube channel.

Andrew Prescott’s overview of the Welsh Repository Network provided us with the surprising finding that smaller institutions are more, not less, likely to want to run their own repository rather than contract it out to someone else.

And via a serendipitous typo, we all contemplated whether working in a repositoire might not be an altogether more rarified and sophisticated career option than working with a repository.

DPTP is of value to anyone with responsibility for digital preservation in an institutional context – its aim is to equip you with the knowledge to effect change in the organisation to allow the right things to happen. (If your primary responsibility is scientific data curation, you may find the DCC’s DC 101 course more applicable.)

Applications need to be in by May 5th – it’s not an onerous process, so don’t delay.

Since it is blogging applications that provide this stucture and behaviour (usually from simple database tables of Posts, Comments, Users, etc), perhaps we should consider making blogging software behave more like an archive. How difficult would that be? Do we need to hire a developer?

One interesting thing about Wordpress is the number of uses its simple blog model has been put to. Under-the-hood it is based on a remarkably simple data base schema of about 10 tables and a suite of PHP scripts, functions and libraries that provide the interface to that data. Its huge user-base has contributed a wide variety of themes and additional functions. It can be turned into a Twitter-like microblog (P2 and Prologue) or a fully-fledged social network (Wordpress MU, Buddypress).

Another possibility exploited by a 3rd-party plugin is that of using Wordpress as an aggregating blog, collecting posts automatically via RSS from other blogs: this seems like a promising basis for starting to develop an archive of blogs, in a blog.

The plugin in question is called FeedWordPress. It uses the Links feature of Wordpress as the basis of a list of feeds which it checks regularly, importing new content when it finds it, as Posts within Wordpress.

I installed FeedWordPress a while ago on ULCC’s DA Blog, and set it up to import all of the ULCC-contributed posts to JISC-PoWR, i.e. those by Ed Pinsent, Kevin Ashley and myself. I did this because I felt that these contributions warrant being part of ULCC’s insitutional record of its activities, and that DA Blog was the best to place to address this, as things stand.

JISC-PoWR also runs on Wordpress, therefore I knew that, thanks to Wordpress’s REST-like interface and Cool URIs, it is easy not only to select an individual author’s posts (/author/kevinashley) but also the RSS feed thereof (/author/kevinashley/feed). This, for each of the three author accounts, was all I needed to start setting up FeedWordPress in DA Blog to take an automatic copy each time any of us contributed to JISC-PoWR.The “author” on the original post has been mapped to an author in DA Blog, so posts are automatically (and correctly) attributed. The import also preserves, in custom fields, a considerable amount of contextual information about the posts in their original location.

In many cases, I’ve kept the imported post private in DA Blog. “Introductory” posts for the JISC-PoWR project blog, for example: as editor of DA Blog, I didn’t feel we needed to trouble our readers there with them; nevertheless they are stored in the blog database, as part of “the record” of our activities.

This is, admittedly, a very small-scale test of this approach, but the kind of system I’ve described is unquestionably a rudimentary blog archive, that can be set up relatively easily using WordPress and FeedWordPress – no coding necessary. Content is then searchable, sortable, exportable (SQL, RSS, etc). (Note, by the way, what happens when you use the Search box on the JISC-PoWR blog copy in UKWAC: this won’t happen with this approach!)

For organisations with many staff blogging on diverse public platforms this would be one approach to ensuring that these activities are recorded and preserved. UKOLN, for example, manages its own blog farm, while Brian and Marieke have blogs at Wordpress.com (as well as contributing to this one), and Paul Walk appears to manage his own blog and web space. This kind of arrangement is not uncommon, nor the problem of how an institution get a grasp on material in all these different locations (it’s been at the heart of many JISC-PoWR workshop discussions). A single, central, self-hosted, aggregating blog, automatically harvesting the news feeds of all these blogs, might be a low-cost, quick-start approach to securing data in The Cloud, and safeguarding the corporate memory.

There are more issues to address. What of comments or embedded images? Can it handle Twitter tweets as well as blog posts? Does it scale? What of look-and-feel, individual themes, etc? Now we start needing some more robust tests and decisions, maybe even a developer or two to build a dedicated suite of “ArchivePress” plugins. But thanks to the power and Open-ness of  Wordpress, and the endless creativity of its many users, we have a promising and viable short-term solution, and a compelling place to start further exploration.

In the Handbook we’ve tried to cover a huge, and sometimes controversial, area in as accessible a way as possible. The workshops, attended by both web-management and records-management professionals from HE institutions, brought  a wide range of concerns and issues to light. It’s been quite a job fitting it all in.

Even as the project progressed, we became aware of new developments in thinking about how to approach the special issues of managing web resources, including everybody’s favourite new fast automatic Web 2.0 applications. We saw the publication of Steve Bailey’s Records Management 2.0 book, TNA’s Web Continuity project, and further web archiving developments at UKWAC. We’ve even heard it whispered in some quarters that approaches to preservation may need a more profound reassessment in the context of the Web and the Cloud. Many of these issues were recorded on the PoWR blog, and we tried to reflect as much of this in the Handbook as possible.

Another recent JISC publication, The Digital Preservation Policies Study by Charles Beagrie Ltd, published at the same time, is complementary in many ways, and reassured us that many of the conclusions we groped towards in the Handbook were not so wide of the mark! Like PoWR, the  Digital Preservation Policies Study identified the necessity of high-level policy engagement as the sine qua non of effective digital preservation.

Digital preservation solutions are undoubtedly partly technical, and the tools being created will enhance digital longevity, but these solutions are also equally dependent on organisational issues. It is important to remember that digital preservation relies on the interaction between the digital preservation environment and wider organisational objectives and procedural issues. These could be financial and staffing issues, collection management, legal obligations, auditing requirements, and other strategies and policies. In this respect, recognition by organisational divisions that digital data is important and key to the successful running of an organisation is crucial.

The Digital Preservation Policies Study, p.11

Among the other recommendations the Study shares with PoWR include:

• Analysis of existing policies and strategies, and how our work can support them even if said polices don’t explicitly refer to preservation or digital assets
• Taking a phased approach – nothing happens all at once. (PoWR recommends pilot projects and working with supportive departments.)
• Careful scoping of preservation requirements. (With regard to web resources, PoWR suggests not everything, not every version, and not forever.)
• Identifying if and where existing systems will do the job
• Consideration of lifecycle, publication, and retention schedules.

The Charles Beagrie survey is a very concise and accessible contribution to the field, and we hope the PoWR Handbook, with its specific focus on established and emerging Web issues, and attention to the detailed and everyday concerns of our many contributors and correspondents, will be similarly useful. We also hope that the work of PoWR will continue in some form, on the blog and perhaps in the form of new projects and workshops, to fill in the gaps we left, and deal with the constantly emerging Web developments. Anyone for PoWR 2.0?

The JISC Digitisation Programme has made a wide variety of valuable resources digitally accessible, including:

• British Newspapers (1620-1900)
• Newsfilm Online
• First World War Poetry
• Newspaper Cartoons
• Welsh Periodicals
• Pre Raphaelite drawings
• East London Theatre Archive

More information about these, and other projects, is available on the JISC Digitisation web page.

The project will review the preservation plans and processes of the sixteen projects funded under Phase 2 of the JISC Digitisation Programme, and identify any medium or long-term access risks to the digitised content. It will also produce recommendations – for individual projects and for JISC as a whole – for processes and strategies to mitigate the risks, and case studies which would be helpful to the broader community.

This is an exciting opportunity for us to apply and extend the experience we have gained working on a range of projects in the field, including the European Visual Archive Market-validation Project (EVAMP) and risk assessments for the recently launched Newsfilm Online project. We will shortly be creating an online home to for the project collaboration and development, and will use DA Blog and the DigiPresSurvey Blog (on JISCInvolve) to keep you updated.

• ‘It was all good. Thank you for the extra bits of reading in the pack too.’
• ‘This course came highly recommended and didn’t disappoint!’
• ‘Even the things that don’t directly apply to me now were useful to know for the future, and to understand the needs of our collaborators.’

We are now studying the feedback with a view to further improving the course and looking at developing some social networking opportunities as well as looking at e-learning. The next DPTP will be in February 2009. Once dates are confirmed we will post them here.

I’m here because I was invited to speak this morning about Ingest, the previous stage in the lifecycle. That was followed by a stimulating set of exercises developed and led by Suzanne Embury which got attendees to think about data quality and the varied measures we might use for it. Suzanne’s exercise was based on work by Wang and Strong (“Beyond accuracy: what data quality means for data consumers”) which starts from the premise that we can never directly measure data quality – rather, we must construct measures which are proxies for quality, or measure only one facet of it. This was new work to me, but it was reassuring to see that it fitted very well with my own experience and with what I know of the ways in which others have approached quality in data. Their original paper took 176 measures of data quality and condensed them into 15 generic aspects of quality. Not all are applicable to all settings, and part of the exercise today got our students considering which were appropriate to them and how they might measure them. This struck me as being particularly useful, as it provides a formal means for deciding which aspects of quality are important to your consumers. Effort can then be focussed on dealing with those aspects well, and not wasting effort on areas that are of lesser relevance, or for which measures are difficult to establish.

The DCC 101 looks to be a promising addition to the canon of training and education in digital curation. We’re looking forward to working with Joy Davidson and her colleagues at the DCC to explore how our training provision and theirs can complement, rather than compete with, each other. There are plans to discuss these issues at a workshop following the IDCC this year, and I’m interested in opinions from a wider audience that would help inform our discussions. If you are a potential attendee at one or more of these training courses, would it help you to see them more integrated ? Do you find it easy to identify the training and education that best suits you at present, and what could improve that situation ? If you are responsible for staff that may need training now and in the future, what type of training provision best suits you and your organisation and what constraints – budgets, time, place – should we all be aware of ? Responses as comments to this post, or by email, would help us all greatly to plan future training that will meet the community’s needs.

Chris Rusbridge has already done an excellent job of some near-real-time reporting on the sessions, and I’m not going to try to replicate that level of detail in this post. As a first-time attendee at iPres, I was impressed by the professional mix attending, which took in hard-core computer science, digital preservation and curation folk, repository managers and those from the traditional custodial professions. In that respect it was very reminiscent of the early DLM-Forums, which were eye-opening for me when I attended the first one in 1996. But it was also interesting to observe that, just as DLM was dominated by archivists and records managers, iPres is a very library-oriented event. For example, those who expressed a desire for a Europe-wide event bringing together all those with an interest in digital preservation didn’t seem to be aware that the DLM-Forums existed.

One positive observation (of many) is that there is more reassuring news on the oft-vexed issue of IPR barriers to digital preservation. At the close of day 1, we heard a summary of the findings of the international survey on the impact of copyright law on digital preservation. That indicated that the UK had one of the strictest set of constraints of all the countries looked at – in terms of who is permitted to carry out certain acts in the name of preservation and what those acts are. Other countries have more relaxed exemptions and that doesn’t appear to be causing the major rightsholdfers any significant financial loss. That should give us hope for some change in the law in the UK at least. And Horst Foster, making the keynote speech opening day 2, appeared to echo this at the European level, implying that the case for change had been made and accepted, although he was notably cautious about making any promises as to when this change might come about.

That’s an improvement on the situation in Europe a few years ago, though, when I was at one of a number of expert panels helping the European Commission to frame the forthcoming research questions and challenges in the digital preservation arena. At that time we were all warned off mentioning the (C) word at all – it seemed to have a somewhat toxic flavour. It’s really heartening to see that things have changed.

One should add a note of caution, however. After Adrienne Muir had commented favourably on how Australian law allows institutions such as the national library and archives to bypass DRM systems in order to preserve material, Colin Webb injected a note of caution. It is apparently still illegal to manufacture or to import a device to Australia which allows a DRM system to be bypassed. But if the national library happens to find such a device on its premises, it can use it without fear of breaking the law. Still some way to go, then, before the law is ‘joined up and working’ – the strapline of iPres2008.

In fact, as I was relieved to note, this group of standards isn’t entirely new to the digital preservation community, and the authors are also aware of general DP standards such as OAIS. They derive from a group of standards known as STEP (Standard for Exchange of Product Model Data), codified in ISO 10303. STEP came up in a presentation and discussions at PresDB’07, although it’s been in development since the turn of the century at least. But STEP is a huge family of standards, and this particular work appears to have emerged from specific work going on in the aerospace industry, which is known to have had concerns about long-term survival of CAD data for some time.

LOTAR is one of the results of this concern, and a presentation by Pierre Duchier of AIRBUS at an Aristote conference in April this year gives a clear picture of the concerns of industry and the approach they are taking. LOTAR and related work were also covered at a invitation-only event at Bath in 2007, Atlantic Workshop on Long-Term Knowledge Retrieval whose attendees included many names familiar from the DCC and the repository community.

Overall, this is reassuring. Here is a set of digital preservation standards being developed and driven by concerns in industry, but where the work is taking place in dialogue with with the academic research and development community. CAD data in particular has long been a concern at the Archaeology Data Service, which has significant holdings of Autocad files, and it will be interesting to see to what extent LOTAR has relevance for such activities. In the meantime, I can see I’ve got a lot of reading to do to catch up, but it would be interesting to hear from others who have more insight into the possible use of these standards outside the aerospace industry.

Incidentally, BS EN 9300 is not to be confused with BS EN ISO 9300, an entirely different standard concerned with “Measurement of gas flow by means of critical flow Venturi nozzles“.

To warm up my brain before the afternoon’s PoWR workshop on preserving web resources, I sampled James Currall’s enjoyable discussion of web archiving, and also a talk on Institutional Repositories by Stephanie Taylor of UKOLN and RRT.

In The Tangled Web is but a Fleeting Dream … but then again…, James Currall covered the essentials of web archiving in a clear and engaging way, drawing comparisons between the survival of WWI soldiers’ diaries, and the blogs of present-day servicemen in Iraq. Another example given was the trials and tribulations of the website for the Lockerbie Trial Briefing Unit: outsourced, host ceased trading, domain name lapsed, original website content dependent on outdated Coldfusion and Access environment. Thankfully a remote-harvested, static HTML image of the site does survive.

Why have there not been more conspicuous successes in web archiving in the past two decades? Partly because it can be difficult to decide exactly whether and when a website should be treated as an authentic (and authenticable) record, a publication channel, or a publication itself – among other things. Partly because there are always waves of innovation continually washing up more interesting things to do. Partly because archiving is a policy issue, yet is generally addressed as purely a techn(olog)ical problem (where it is addressed at all).

It was reassuring, as ever, to hear it said again that there is not one single tool that addresses all possible web preservation issues (behaviour, dynamic content, scripts, versioning, emerging standards, etc.); that depending on the Internet Archive is at best a partial and risky solution; and that “whatever you do is likely to be imperfect”.

James put the chamber’s electronic voting systems to entertaining and informative use with a number of snap votes: would, for example, present-day soldiers’ blogs still be available in 90 years’ time? Of course I could confidently vote ‘yes’, knowing that the publication of the JISC PoWR handbook is barely months away! Other ad hoc vox pops revealed that the audience was about as familiar with OAIS as with the Book of Ezra.

James is director of the Espida Project, and this was a timely reminder that we must consider the relevance of that project’s work on assessing and controlling costs to the guidance we’re assembling for the forthcoming PoWR handbook. James and I share what must be a fairly unusual distinction of citing Thomas Carlyle in support of our cause in recent presentations. Unlike James, however, I don’t bear such an uncanny likeness to the great man.

Stephanie Taylor’s talk, Institutional Repositories: Asset or Obstacle?, gave us a brief history of the Insitutional Repository, and a well-paced explanation of the value and purpose of IRs – rather more substantial, thankfully, than my Bluffer’s Guide To IRs last year. I was initially intrigued that her presentation might be leading to the conclusions that IRs are “obstacles”, but I quickly realised the alarmist title was rhetorical in nature. The talk was, among other things, an engaging appeal to the better nature of web managers and other techies to appreciate the value of, and issues faced by librarians.

In fact, the emergence of repositories, and the recognition of their place among institutional information systems, is a watershed in the evolution of electronic resources. Leaving it to researchers and teachers to manage what they create – variously on websites, blogs, Google Docs, thumb drive, or what you will – is no more sensible now than it ever was, if we want to ensure that they are consistently managed and accessible, let alone think about their preservation over the longer term.

There are many ways to approach it, none the only right and proper way. You may add extra value to it, through your choice of software or implementation (in-house or outsourced), or by using Web-Two-Oh-ish features and services. Institutional Repositories are only “one of many small conversations going on in different ways in different mediums”, but the need for an institution to manage valuable academic outputs in an orderly way is unarguable.

However one tweet that flew over the Twitosphere during the afternoon suggests that there may be more to do in assessing the pros and cons of different approaches – particularly the merits or otherwise of Google’s omnipresent panaceas. Mike Ellis commented: “I’ve never been clear why it is that institutions would trust a repository vendor more than someone like Google….?”

I won’t write up the JISC-PoWR workshop – the results of that will be on the JISC-PoWR blog - except to say that Marieke succinctly summarised the many web preservation issues we’ve accumulated, while Brian found effective waysto draw out issues and concerns around the growing range of Web 2.0 applications finding favour among staff and students at our institutions; and we were pleased that both Stephanie and James were able to contribute to the discussion.

And then it was off to the Aberdeen City Art Gallery for a glass of wine or three and some inspirational art….

But what’s often given less serious consideration, in the excitement of using a third-party provider of wikis, blogs, Ning, etc., to get your collaborative hypertext project off the ground so quickly and easily – and without having to go cap or cheque in-hand to whoever guards your web space – is this key preservation issue: what happens when you want to get your painstakingly intricate web of hyperlinked pages out?

There are many good reasons why you might want to do this: you might want to migrate to another wiki system or CMS, as the shape and nature of your content evolves; or put it on a permanent, persistent footing by moving it into your own domain; you might simply want to back it up or take a snapshot; or you might want to pull out information for publication in a different form. When you had one or two pages, it might have seemed trivial; but what if you now have hundreds?

Unfortunately, just as exporting the information is often a secondary consideration for wiki content creators, so it also is for the wiki farm systems. The Wetpaint Wiki discussion boards indicate that an export feature was a long time in coming (and its absence quite a blocker to adoption by a number of serious would-be users). And what was eventually provided leaves a lot to be desired.

Wetpaint’s backup option “lets” you download your wiki content as a set of HTML files. Well, not really HTML files: text files with some embedded HTML-like markup. (Which version? Not declared.) Don’t expect to open these files locally in your browser and carry on surfing your wiki hypertext (even links between wiki pages need fixing). The export doesn’t include comment threads or old versions. Restoring it to your online wiki is not possible. But, for what it’s worth, you have at least salvaged some sort of raw content, that might be transformed into something like the wiki it came from, if hit with a bit of Perl script or similar.

I checked out Wikidot – another impressively-specced, free “wiki farm”. Wikidot’s backup option will deliver you a zip file containing each wiki page as a separate text file, containing your wiki markup as entered, as well as all uploaded file attachments. However, according to Wikidot support:

you can not restore from it automatically, it does not include all page revisions, only current (latest), it does not include forum discussion or page comments.

To reconstruct your wiki locally, you’ll, again, need some scripting, including using the Wikidot code libraries to reconvert its non-standard wiki-markup into standard HTML.

A third approach can be seen with a self-hosted copy of Mediawiki. Here you can select one or more pages by name, and have them exported as an XML file, which also contains revisions and assorted other metadata. Within the XML framework, the page text is stored as original wiki markup, raising the same conversion issues as with Wikidot. However, the XML file can be imported fairly easily into a different or blank instance of Mediawiki, recreating both hypertext and functionality more or less instantly.

In contrast to all these approaches, if you set a spidering engine like HTTrack or Wget to work “remotely harvesting” the site, you would get a working local copy of your wiki looking pretty much as it does on the web. This might be an attractive option if you simply want to preserve a record of what you created, a snapshot of how it looked on a certain date; or just in case a day should come when Wetpaint.com Inc., and the rest, no longer exist.

However, this will only result in something like a preservation copy – not a backup that can be easily restored to the wiki, and further edited – in the event, say, the wiki is hacked/cracked, or otherwise disfigured. For security alone, it may be sufficient to depend on regular backups of the underlying database, files and scripts: but you still ought to reassure yourself exactly what backup regime your host is operating, and whether they can restore them in a timely fashion. (Notwithstanding the versioning features of most wikis, rolling back a raft of abusive changes across a whole site is not usually a quick, easy or particularly enjoyable task.)

All this suggests some basic questions that one needs to ask when setting up a wiki for a project:

• How long do we need it for?
• Will it need preserving at intervals, or at a completion date?
• Is it more important to preserve its text content, or its complete look?
• Should we back it up? If so, what should we back up?
• Does the wiki provide backup features? If so, what does it back up (e.g. attachments, discussions, revisions)?
• Once “backed up”, how easily can it be restored?
• Will the links still work in our preservation or backup copy?
• If the backup includes raw wiki markup, do you have the capabilities to re-render this as HTML?

And questions like these are no less relevant when considering your uses of blogs and other social software: I hope we’ll be able to look at them more closely in another post.

The proprietary nature of RAW formats (an emerging OpenRAW standard notwithstanding) and the relative newness on the block of JPEG 2000 would both tend to bolster the longstanding preference for TIFF, but as Chris’s posts make clear, each preservation project should nevertheless weigh the options based on its own requirements and resources.

If in doubt, the “keep everything” approach is attractive, as ever, but – in spite of the old mantras about ever-cheaper filestore – the implications for storage space and management are potentially very costly once one enters the world of Terabytes and Petabytes. In one example, Sean Martin of BL concludes that

probably only a small amount of additional value is created for the additional expense approaching £200K. This leads to the question “if we had £200K on what would we spend it?” and probably the answer is “not in this way”.

The posts contain many interesting examples of costings, based on the experience of BL, SDSC and others. I won’t even attempt to summarise (Chris’s summary of) them here, but I hope that this post will still be available for me to consult next time I have to dabble in the murky science of DP costings.

In some instances, the case for preserving the RAW image may nevertheless be compelling. One can imagine that the risk of missing a new planet or virus (or identifying a non-existent one), or other potential infelicities in scientific and medical imaging, is not worth contemplating. By contrast, it’s hard to see what value might be added to a collection like the Linnean Society’s by keeping raw camera data alongside the TIFFs, and more than doubling the amount of storage capacity required.

It’s for making decisions like this that the intellectual exercise of identifying significant properties and the needs of all stakeholders – creators, curators and users – seems particularly essential.

• Government departments are increasingly citing URLs in answer to Parliamentary Questions
• 60% of links in Hansard to UK government websites for the period 1997 to 2006 are now broken
• Departments vary considerably: for one, every link works; for another every link is broken. (TNA’s own website is not immune!)

• All links work in perpetuity
• No cited information is lost through deletion
• Information is preserved long-term, even if the Web is no longer the dominant publishing medium it is today

• Guidance on creation and use of XML Sitemaps
• A website component (for MS IIS and Apache) that will redirect users to the Web archive if a link is no longer active but is in the archive
• Guidance to government webmasters on best practice for website design and maintenance for archiving purposes

As Richard noted in Part I, digital preservation is a “series of managed activities necessary to ensure continued access to digital materials for as long as necessary.” But what sort of digital materials might be in scope for the PoWR project?

We think it extremely likely that institutional web resources are going to include digital materials such as “records created during the day-to-day business of an organisation” and “born-digital materials created for a specific purpose”.

What we want is to “maintain access to these digital materials beyond the limits of media failure or technological change”. This leads us to consider the longevity of certain file formats, the changes undergone by proprietary software, technological obsolescence, and the migration or emulation strategies we’ll use to overcome these problems.

By migration we mean “a means of overcoming technological obsolescence by transferring digital resources from one hardware/software generation to the next.” In contrast, emulation is “a means of overcoming technological obsolescence of hardware and software by developing techniques for imitating obsolete systems on future generations of computers.”

Note also that when we talk about preserving anything, “for as long as necessary” doesn’t always mean “forever”. For the purposes of the PoWR project, it may be worth us considering medium-term preservation for example, which allows “continued access to digital materials beyond changes in technology for a defined period of time, but not indefinitely.”

We also hope to consider the idea of life-cycle management. According to DPC, “The major implications for life-cycle management of digital resources is the need actively to manage the resource at each stage of its life-cycle and to recognise the inter-dependencies between each stage and commence preservation activities as early as practicable.”

From these definitions alone, it should be apparent that success in the preservation of web resources will potentially involve the participation and co-operation of a wide range of experts: information managers, asset managers, webmasters, IT specialists, system administrators, records managers, and archivists.

(All the quotations and definitions above are taken from the DPC’s online handbook.)

I don’t think I could begin to do justice in a few words to the wide-ranging debate at the JISC/BL/DPC Workshop on Significant Properties at the British Library on Monday: I’d rather leave it to others to analyse the significant outcomes in more detail, or to further discussions like the one started by Chris Rusbridge (our cucumber-cool chairman on the day) on the DCC Blog. Suffice it to say there was a sack of food for thought in all the presentations, and lots of opportunities to wonder “now why didn’t I think of that?”

I think the most enjoyable presentation was that of Cal Lee from UNC Chapel Hill, who managed to lift us with a few laughs about his work – with Microsoft Word file format specifications – just as we were beginning to flag at the end of a long day of considering many things from many angles. The Moving Images study also looks like it may have useful information for our deliberations on standards and formats for PRIMO. If I wasn’t so taken with the presentation on E-learning objects, that was chiefly because I gave it – a mind-bending task to give our account of not only Learning Objects but also Significant Properties in 15 mins. But the slides look nice on the DPC website. (All credit to Ed for the excellent diagrams.)

Neil Grindley and the rest of the JISC/DPC/BL organisers did well to run it so smoothly and facilitate the discussion, and it was a rare opportunity to catch Andrew Wilson on a flying visit to this hemisphere and hear his first hand account of NAA’s approaches and models. I can’t remember if I’ve been to the British Library Conference Centre before or not, but it is certainly an impressive space, and with super-slick AV facilities.

I say ‘silently’ because the information about the effect that this update will have is extremely well hidden. Many individuals, and many large organisations, will do as MS recommends and allow their systems to download and install updates like this automatically. They’ll be caught unawares. Others may be more like me and read through the release information to see what the update is going to do beforehand. They would be caught unawares. The information linked to at the front of this post tells you that this is a major security update for Office. It says nothing about losing access to old files. It links to another page which goes into greater detail about what the update contains. That doesn’t mention the issue either, except that under the “known issues” section, it says:

After you install Office 2003 SP3, you may receive an error message when you try to open or to save a file. For more information about this issue, click the following article number to view the article in the Microsoft Knowledge Base:

And even that article doesn’t really list things in detail, although it does mention that older office formats are amongst those affected. The recommended solutions are not ideal either.

Microsoft has belatedly realised that it was a bad idea and released a patch to circumvent the problem. But you need to explicitly request that and many people won’t realise they need it for some time.

There’s been a good deal of discussion on the e-recs list about this, including this remark from Marc Fresko:

Personally, I think this is one of the more “visible” wake-up calls that
should contribute to everyone realising that long-term access to
electronic records, even over less than ten years, cannot be ignored.
Active management is needed. Anything that helps to get this message
across is welcome, so long as it is (as is the case here) practically
harmless.

I agree with Marc that it’s a useful wake-up call. Consider that this could cause problems even to those organisations that had taken explicit steps to ensure continued access:

• You checked when moving to Office 2003 that you could still access your older documents using it.
• You checked the documentation with each update to ensure that it wasn’t going to invalidate your earlier checks

With those precautions, you would still be caught out by a silent update that you cannot uninstall. Only if yours is the sort of organisation that explicitly tests all updates in a safe environment, and specifically includes a check for access to legacy material as part of those tests, would you be alerted to the problem in time.

Microsoft aren’t the only software providers to build in automated update features to their products – the practice is common with both propretary and open-source products. Incidents like this will occur again. It reinforces the belief that one needs to have good control over the application(s) needed to access preserved content as well as the formats used to preserve it, whether one is looking at 5-year retention or 500-year retention.

I’ve been a judge for each of the awards and chaired the judging panel on this occasion; it’s been a fascinating and enjoyable experience and certainly makes one think again about what makes a project valuable to others. But we’re concerned that the awards might not be having the impact that they could be, and that we need to do something to change the type of entry we receive – as well as thinking about how we publicise the event and our association with it. The DPC has established a working group to look at a number of these issues, and I’m interested in views from the wider world – particularly from those outside the DPC itself – about the awards, their relevance to you and your work, and what might encourage you to submit an entry.

Have you considered submitting an entry ? What held you back ? What do you think of the entries that have been shortlisted in the past ? One contributor has suggested that the final judging process (in which each entrant does a 25 minute presentation, followed by about 25 minutes of questions from the judges) should be held in public, instead of in private as it is at present. What do you think of that idea ?

As well as using the comment facility on this blog to share your thoughts, you’re welcome to email me (K.Ashley@ulcc.ac.uk) if you would rather not share your thoughts with the rest of the world.