However another recent find on the Web compels me to make an additional point, namely that we might do equally well to differentiate between scanning and digitisation. Anyone can set to work with a scanner and create a bunch of digital images – but that barely scratches the surface of what I think we should be expecting of a digitisation project in 2011.

First and foremost, we need metadata: the more the merrier, but something at least. Even if we expect to come back later and polish it up (once the images can be browsed and examined on screen). In the absence of any established metadata profiles for a project, at least try to cover as many Dublin Core elements as possible – title, creator, date, subject/keywords… Images, in particular, may prove tricky or time-consuming to find again, especially once there are thousands of them on a disk. We should probably keep the metadata in a database, and perhaps additionally store metadata with the objects. This can be as XML or plain text files stored alongside the digital images, or embedded in the files we create (many common file formats – TIFF, JPEG, MPEG, PDF – support metadata embedding, and there are many free tools available to help).

There is yet more, though, that we should be doing, particularly when we are scanning text-based objects (articles, books, magazines, reports, etc). Most importantly, we really should try and extract the text from the image if possible. ^[1]

My recent web find was the teaching blog of Dr Toine Bogers at the Royal School of Library and Information Science (RSLIS) in Copenhagen, Denmark. One fascinating post describes a Lab Session exercise, From OCR To NER, a set of comparatively simple command-line processes to get the most out of a scanned-text project.

Toine’s post walks us through the process. Once the article is scanned, we should apply some OCR. The exercise goes further and also describes the use of tools to clean up and spell-check the resulting OCR’d text. This will, at the very least, result in a separate text file, hopefully containing a fairly accurate version of the article text. Finally, the cleaned-up text can be submited to a Named Entity Recognition service. Toine’s exercise uses NER tools at University of Illinois. (We’ve been using similar functionality provided by OpenCalais and GATE for our AIM25 Open Metadata project.)

Why do all this? The most important, instant, result of this is that we can now easily index our article for full-text searching – in a local repository system, such as EPrints or DSpace provide – and of course by Google. None of this is possible if we leave the scanned image as just that – an image.

Another  side-effect of any successful OCR outcome, is that the text is now free to be re-flowed. This means that we might consider sharing it with users in a variety of forms enhancing usability and accessibility.

It’s important not to confuse preservation formats with formats for access and dissemination. You probably will have your scanned image masters in TIFF, RAW, JPEG2000, PostScript, SVG. None of these are likely to be of much use to your users over the Web. Not only are the formats not widely supported by Web browsers, but most users probably don’t need or want your master image. If it’s a high-resolution scan of a 100 page book, they might be looking at 100Mb download, or worse – slow to load, and probably slow to render and navigate.

Time taken thinking what formats will give users the best experience is time well spent. What platforms might they want to use now and in the foreseeable future? It’s less than 18 months since Kindle3 made e-book readers affordable, and the Ipad made them sexy. E-books look and function very impressively on both platforms (albeit in different ways): for an overview of some of the benefits of the EPUB format, see Martin Fenner’s post Beyond The PDF… is EPUB. PDF outputs may yet have their uses, if users can at least search for text within them. The point is that only with properly digitised text, do these kinds of accessibility options become possible.

Even image collections can also be disseminated as E-books – nice offline items some users might care to flick through on their tablet computers, possibly even smartphones. I’ve demonstrated how we can create OJS XML from EPrints XML on-the-fly with XSLT: since EPUB and Mobi/Kindle are XML-based formats, we should be able to do something similar to create e-books using repository APIs. Also, by using appropriately sized images in dissemination formats (Ipad screen is 1024x768px; Iphone4 is 960x640px) we can not only ship our users a sensibly-sized download, we can protect any capital we may have in the master images, without having to resort to ugly tricks like watermarking. (Giving users full-size, high-res images with embedded watermarks seems to me the worst of all worlds.)

Therefore I’d suggest that, in order to get the best out of a digitisation project, consider what would you like to see at the end of the project – and, more importantly, what would give your users the best experience, or even win you new users? Ask around, do some tests, with users if possible, and get an idea how they want to use the materials and how they will get the best out of them. Maybe there are comparable projects and systems that you admire, with features you’d like to be available for your collection. What about in five or ten years’ time: will your current project outputs help or hinder longer term accessibility goals?

This kind of vision of is essential. Without some conception of the end result, how the materials will be used and managed most effectively, all the scanning in the world isn’t going to amount to a successful digitisation project.

At an atomic level, a “blog” comprises “blog posts”, which are continually added to the blog corpus: that is the dynamic essence of a blog, and distinguishes it from old-fashioned, largely static Websites and hypertexts in which little content changed between major update iterations, which process was probably more akin to “publishing a new edition” in the world of non-digital publications.

The blog also displays, as part of its frame, other graphical and functional elements (sidebars, widgets, “blogrolls”, etc) which may themselves contain dynamically updated, constantly changing information. These can be added, removed, amended and rearranged at will by the blog author/editor. Blog posts that were “published” in the context of one set of framing elements, will persist through subsequent versions of that framework.

Similarly with design (layout, colours, mastheads, etc), though the persistence tends to be longer, the informal nature of blogs means that these may be easily changed by the blog editor/author, and are thus more volatile than a typical “corporate” website. Again, blog posts may persist, unchanged in themselves, through many iterations of the blog site design and layout.

A simple view of blog elements and their temporal relationship

This very simplified visualisations suggests where we might start conceptualising key elements of a blog. It indicates that they iterate over time, but in the cases of Design, Posts and Widgets (as we’ll call them for brevity), according to independent schedules. While Posts and Comments persist in the online view of a blog, designs and widget arrangements are overwritten.

With my earlier ArchivePress project we deliberately overlooked preservation of the blog’s framing elements, and (given the much smaller scope of that project) established an acceptable rationale for doing so. The challenge for BlogForever is to find a solution to  precisely these issues. Unless we were simply to adopt the snapshot approach of Heritrix-based web archiving initiatives (e.g. Wayback/archive.org, UK Web Archive), we need to ensure the BlogForever repository supports a degree of granularity that can capture, describe and preserve atomic blog objects in a way that reflects the particular interdependencies, in order to understand and preserve them authentically, and permit the many possible authentic and valid “time slice” views and analyses that users of the archive will need.

(I appreciate, by the way that these objects themselves are compound objects, so not strictly “atomic”: but the same is also true of atoms, as our CERN colleagues can attest!)

My naïve thinking in the area has been characterised by the assumption that the process is rather static or linear, and that the problem we’re facing is broadly the same every time; migrate data from a format that’s about to become obsolete or unsupported, onto another format that’s stable, supported, and open. MS Word document to PDF or PDF/A…now that, I can understand!

In fact, I learned at least two ways of thinking about formats that hadn’t occurred to me before. One simple one is costs; some formats can cost more to preserve than others. This can be calculated in terms of storage costs, multiplied over time, and the costs associated with migrations to new versions of that format. For example, we’ve tended to pin our faith on the TIFF format for images for many reasons, but there’s a high storage price to be paid for all that wonderful losslessness. This may be one reason why the DP world is looking with more favour on the JPEG2000 format, which is ‘virtually’ lossless and smaller in size.

Secondly, the problems of preserving digital data which doesn’t actually have a specified stable preservation format. Chris Puttick of Oxford Archaeology gave a vivid description of the problems he’s facing with CAD and GIS files, where the data can’t easily be tied to a single format in the first place (nor can a stable format for migration be identified). As the NLA put it on their PADI page, “At present there is little dealing specifically or comprehensively with the preservation of this particular type of data, although some aspects of database preservation are applicable to GIS. Some long term preservation issues include a lack of open source formats and metadata standards, large data volume and complex data objects.” Puttick suggests that his data doesn’t really perform at all unless it’s operated within a very specific environment of hardware and software. How do we preserve an environment? This appears to be quite a distinct preservation problem and much harder to solve than Word to PDF, to put it mildly.

William Kilbride suggested that such cases (and websites too, arguably, because they are time-based) are more like a stream of data – a handy image which conveys something about the dynamic of such information packages, and showing us that it’s much harder to nail them down into a single format. You can never step into the same river twice.

Chiptunes are typically created using early personal computers or videogames and/or their soundchips. In that respect, they depend on technology preservation – the museum approach to digital preservation. Chiptune composers either use the systems as designed, programming them directly to create their music, or alter them in some way using techniques collectively known as ‘circuit-bending’, which makes the machines capable of producing sounds that they could not have originally produced. Some aspects of the chiptune scene utilise more modern synthetic techniques to recreate the sounds produced by these early chips – these are, in a loose sense, emulating the original systems, although not in a way that would allow you to use original software to create your sounds. But some adherents of the chiptune genre are going further, using the sounds of the systems themselves in their compositions.

The article which set my train of thought going covered Matthew Applegate’s (aka Pixelh8) concert in late March 2009 at the National Museum of Computing, which amongst other things used the electro-mechanical sounds produced by the Colossus and by hand-cranked adding machines. Now in this respect, as in many others, digital isn’t different to the analogue world. One could argue that found sounds have a long tradition in Western music, from Tchaikovsky’s use of cannon in the 1812 overture through to Leroy Anderson’s typewriter symphony and the work of the Boston Typewriter Orchestra, who eschew traditional instruments entirely and concentrate solely on percussive typewriter music, interleaved with the odd end-of-line bell.

But pixelh8′s work reminded me of some earlier, and unlikely, uses of computers to create music which I fear is already lost – although discerning listeners might well argue that we are all much the better for it being lost. One depended on the noise-making abilities of computer peripherals such as line and chain printers, the fiercest of which were capable of noise levels that drowned out conversation their vicinity. Devious users could create files which generated recognisable rhythms when printed on a specific printer. I’ve seen reference both to the “William Tell Overture” and “She’ll be Coming Round the Mountain” being produced by this method, which demanded detailed knowledge of the printer’s workings (but no programming knowledge – the effect was produced merely by printing a carefully-crafted file.) It seems that at least one recording of this exists, from the mid-1960s. Good ideas like this don’t die, or are reimagined by later generations, and 1999 brought us the Symphony for dot matrix printers.

These examples, in common with some of pixelh8′s work, are dependent on the relevant hardware still existing and being in working order in order to allow us to continue to perform them, or to create new compositions. That’s certainly a challenge for things like line printers, and I imagine the same is true of mechanical typewriters. These, however, are relatively simple mechanical devices and, so long as someone is still interested in maintaining them, they will be repairable for many generations to come, even if they are no longer being manufactured.

But other forms of of computer music aren’t so easily re-created. At about the same time as people were creating line printer music, others realised that the computer’s own electronics generated sound in an unplanned-for way. Almost all computers of the 1960s generated substantial amounts of radio frequency interference of a type that’s now illegal. The clock speeds of those computers meant that the interference fell mainly in the medium wave band, between 500 Khz and just over 1Mhz, exactly the right spot for any consumer AM radio to pick it up. (Present-day systems, if not shielded, would be generating interference for microwaves and mobile phones, not AM radios.) Enterprising programmers with time on their hands and dedicated systems to play with wrote programs which generated RF interference which approximated a tune, since the exact nature of the interference depended on the instructions being processed, the data being fetched, and probably the addresses from which it was being fetched, as well as the properties of the CPU circuitry.

I encountered one version of this, using one of DEC’s PDP10 processors (probably a KI10, although I can’t be sure) and the DECUS program library also contains a similar program for the PDP8. I’ve come across one reference to a recording being made, this time of the same technique on an IBM 1401, but the available example uses samples from a recording to make tunes rather than the original recording itself being musical.

All these examples illustrate, to different degrees, the digital preservation problem that pixelh8′s work brought to mind. The RFI music was exquisitely sensitive to properties of the original electronics. It wasn’t enough that the computer ran the same instruction set. DEC’s PDP10 series had four generations of processor – the KA10, KI10, KL10 and KS10 – each of which could have run the music program without error. But since each was entirely different at the circuit level, only one of those machines would produce the right sort of radio interference to generate the desired effect. Emulators of these systems now exist, and a number of manufacturers have produced hardware-level recreations such as the Toad 1, Foonly and Systems Concepts machines. But neither the software nor the hardware emulators will have the same radio interference properties, so again, although they will run the original program without errors, it won’t produce the desired effects.

In fact, it would be extremely difficult to produce an electronic equivalent of those systems again, and as technology moves on such problems will only get more difficult. The early systems, which used primarily discrete components (transistors, capacitors, etc) could be built again by sufficiently dedicated enthusiasts. But chip fabrication is another thing entirely. The music that those programs and systems produced is essentially unpreservable except by recording it at the time. The significant properties that emerge from the combination of software and hardware are extremely difficult to characterise, and no environment that allows the software to run will emulate the one property of the original environment that really matters.

Some stuff really, really can’t be digitally preserved. It’s good to remember that sometimes.

(And if you are aware of any other recordings of computer ‘music’ of this type, I and others would very much like to hear of them.)

[Updated to remove an embedded YouTube video which causes problems with the Iphone. Instead, you'll have to follow the link to see and hear some hardware-based chiptune hacking.]

Some of us at ULCC, and over 100 other people from around the UK, spent a couple of days this week at the Aston Business School reviewing the outcomes of JISC’s repositories and preservation programme and looking forward to what comes next. It was a useful and stimulating couple of days – the best programme meeting I’ve attended so far. The few projects that weren’t represented at the meeting missed out in a lot of ways. If you’re involved in a JISC project, make sure you, your project manager, or both of you go to a programme meeting when you are invited. You’ll learn a lot, make some useful contacts, save some time, get some useful ideas and possibly lay the groundwork for future projects or collaborations.

I began the day by chairing the final meeting of RPAG(the repositories and preservation advisory group.) We had a short meeting mainly to follow up on discussions we had been having on how the group had operated and how JISC might make use of advisory bodies in future. Those who expressed an opinion all felt it had been useful to them, but we all had concerns about how our time, and the JISC Executive’s time, might have been used more effectively. Future advisory groups may try to split responsibility for some areas into smaller working groups. All were agreed that the face-to-face meetings were invaluable, but we weren’t all agreed on which online technology would be best to use in between times. Enthusiasts for tools like ideascale were matched by those who found them unusable.

The meeting proper opened with some background and perspective from Rachel Bruce and Neil Grindley of JISC and myself. I tried – partly seriously, but without much expectation of accuracy – to give a one-line summary of what each project set out to do. But there were two things I meant to say which I failed to do. One was to look forward to the theme of day 2 (Value) and stress that repositories are not ends in themselves, but need to be thought of in terms of value, impact and benefits to someone. The second point I omitted was to remind us that , for innovation projects, failure in one sense can still mean success, as long as we understand the nature of the failure and are able to use it to improve and adapt future work. Not achieving what you set out to do is disappointing. Analysing the reasons for that and making sure others are aware of them can be of great value.

But it was the rest of the event that provided greatest interest. The discussion sessions on text mining, research data, teaching and learning repositories and more; presentations from projects from stakeholder, developer and other perspectives; posters and demos from many of the projects; and the fever of activities in the ideas room, which deployed technology ranging from post-it notes upwards to catalyse, capture and refine ideas from the attendees. These activities gave the event much more of a participatory feel – everyone became a contributor rather than being a consumer.

I learned a few things over the course of a day or two, most of them unexpected. David Flanders (via Chris Rusbridge) passed on the neat idea of feeding funding proposals through Wordle before marking them. That’s what ULCC’s AIDA project looked like. Perhaps you ought to try the same with your proposals prior to submitting them?

I learned that talking unprepared and unscripted to a video camera doesn’t produce great results unless you’ve had practice or training – neither of which I’ve had. I knew that in an abstract sense and now have the unfortunate experience to back it up. But Andy McGregor and Dave Flanders did capture some other people talking far more sense than I did and far more clearly, and you can see the results on the dev8D youtube channel.

Andrew Prescott’s overview of the Welsh Repository Network provided us with the surprising finding that smaller institutions are more, not less, likely to want to run their own repository rather than contract it out to someone else.

And via a serendipitous typo, we all contemplated whether working in a repositoire might not be an altogether more rarified and sophisticated career option than working with a repository.

DPTP is of value to anyone with responsibility for digital preservation in an institutional context – its aim is to equip you with the knowledge to effect change in the organisation to allow the right things to happen. (If your primary responsibility is scientific data curation, you may find the DCC’s DC 101 course more applicable.)

Applications need to be in by May 5th – it’s not an onerous process, so don’t delay.

Since it is blogging applications that provide this stucture and behaviour (usually from simple database tables of Posts, Comments, Users, etc), perhaps we should consider making blogging software behave more like an archive. How difficult would that be? Do we need to hire a developer?

One interesting thing about WordPress is the number of uses its simple blog model has been put to. Under-the-hood it is based on a remarkably simple data base schema of about 10 tables and a suite of PHP scripts, functions and libraries that provide the interface to that data. Its huge user-base has contributed a wide variety of themes and additional functions. It can be turned into a Twitter-like microblog (P2 and Prologue) or a fully-fledged social network (WordPress MU, Buddypress).

Another possibility exploited by a 3rd-party plugin is that of using WordPress as an aggregating blog, collecting posts automatically via RSS from other blogs: this seems like a promising basis for starting to develop an archive of blogs, in a blog.

The plugin in question is called FeedWordPress. It uses the Links feature of WordPress as the basis of a list of feeds which it checks regularly, importing new content when it finds it, as Posts within WordPress.

I installed FeedWordPress a while ago on ULCC’s DA Blog, and set it up to import all of the ULCC-contributed posts to JISC-PoWR, i.e. those by Ed Pinsent, Kevin Ashley and myself. I did this because I felt that these contributions warrant being part of ULCC’s insitutional record of its activities, and that DA Blog was the best to place to address this, as things stand.

JISC-PoWR also runs on WordPress, therefore I knew that, thanks to WordPress’s REST-like interface and Cool URIs, it is easy not only to select an individual author’s posts (/author/kevinashley) but also the RSS feed thereof (/author/kevinashley/feed). This, for each of the three author accounts, was all I needed to start setting up FeedWordPress in DA Blog to take an automatic copy each time any of us contributed to JISC-PoWR.The “author” on the original post has been mapped to an author in DA Blog, so posts are automatically (and correctly) attributed. The import also preserves, in custom fields, a considerable amount of contextual information about the posts in their original location.

In many cases, I’ve kept the imported post private in DA Blog. “Introductory” posts for the JISC-PoWR project blog, for example: as editor of DA Blog, I didn’t feel we needed to trouble our readers there with them; nevertheless they are stored in the blog database, as part of “the record” of our activities.

This is, admittedly, a very small-scale test of this approach, but the kind of system I’ve described is unquestionably a rudimentary blog archive, that can be set up relatively easily using WordPress and FeedWordPress – no coding necessary. Content is then searchable, sortable, exportable (SQL, RSS, etc). (Note, by the way, what happens when you use the Search box on the JISC-PoWR blog copy in UKWAC: this won’t happen with this approach!)

For organisations with many staff blogging on diverse public platforms this would be one approach to ensuring that these activities are recorded and preserved. UKOLN, for example, manages its own blog farm, while Brian and Marieke have blogs at WordPress.com (as well as contributing to this one), and Paul Walk appears to manage his own blog and web space. This kind of arrangement is not uncommon, nor the problem of how an institution get a grasp on material in all these different locations (it’s been at the heart of many JISC-PoWR workshop discussions). A single, central, self-hosted, aggregating blog, automatically harvesting the news feeds of all these blogs, might be a low-cost, quick-start approach to securing data in The Cloud, and safeguarding the corporate memory.

There are more issues to address. What of comments or embedded images? Can it handle Twitter tweets as well as blog posts? Does it scale? What of look-and-feel, individual themes, etc? Now we start needing some more robust tests and decisions, maybe even a developer or two to build a dedicated suite of “ArchivePress” plugins. But thanks to the power and Open-ness of  WordPress, and the endless creativity of its many users, we have a promising and viable short-term solution, and a compelling place to start further exploration.

In the Handbook we’ve tried to cover a huge, and sometimes controversial, area in as accessible a way as possible. The workshops, attended by both web-management and records-management professionals from HE institutions, brought  a wide range of concerns and issues to light. It’s been quite a job fitting it all in.

Even as the project progressed, we became aware of new developments in thinking about how to approach the special issues of managing web resources, including everybody’s favourite new fast automatic Web 2.0 applications. We saw the publication of Steve Bailey’s Records Management 2.0 book, TNA’s Web Continuity project, and further web archiving developments at UKWAC. We’ve even heard it whispered in some quarters that approaches to preservation may need a more profound reassessment in the context of the Web and the Cloud. Many of these issues were recorded on the PoWR blog, and we tried to reflect as much of this in the Handbook as possible.

Another recent JISC publication, The Digital Preservation Policies Study by Charles Beagrie Ltd, published at the same time, is complementary in many ways, and reassured us that many of the conclusions we groped towards in the Handbook were not so wide of the mark! Like PoWR, the  Digital Preservation Policies Study identified the necessity of high-level policy engagement as the sine qua non of effective digital preservation.

Digital preservation solutions are undoubtedly partly technical, and the tools being created will enhance digital longevity, but these solutions are also equally dependent on organisational issues. It is important to remember that digital preservation relies on the interaction between the digital preservation environment and wider organisational objectives and procedural issues. These could be financial and staffing issues, collection management, legal obligations, auditing requirements, and other strategies and policies. In this respect, recognition by organisational divisions that digital data is important and key to the successful running of an organisation is crucial.

The Digital Preservation Policies Study, p.11

Among the other recommendations the Study shares with PoWR include:

• Analysis of existing policies and strategies, and how our work can support them even if said polices don’t explicitly refer to preservation or digital assets
• Taking a phased approach – nothing happens all at once. (PoWR recommends pilot projects and working with supportive departments.)
• Careful scoping of preservation requirements. (With regard to web resources, PoWR suggests not everything, not every version, and not forever.)
• Identifying if and where existing systems will do the job
• Consideration of lifecycle, publication, and retention schedules.

The Charles Beagrie survey is a very concise and accessible contribution to the field, and we hope the PoWR Handbook, with its specific focus on established and emerging Web issues, and attention to the detailed and everyday concerns of our many contributors and correspondents, will be similarly useful. We also hope that the work of PoWR will continue in some form, on the blog and perhaps in the form of new projects and workshops, to fill in the gaps we left, and deal with the constantly emerging Web developments. Anyone for PoWR 2.0?

The JISC Digitisation Programme has made a wide variety of valuable resources digitally accessible, including:

• British Newspapers (1620-1900)
• Newsfilm Online
• First World War Poetry
• Newspaper Cartoons
• Welsh Periodicals
• Pre Raphaelite drawings
• East London Theatre Archive

More information about these, and other projects, is available on the JISC Digitisation web page.

The project will review the preservation plans and processes of the sixteen projects funded under Phase 2 of the JISC Digitisation Programme, and identify any medium or long-term access risks to the digitised content. It will also produce recommendations – for individual projects and for JISC as a whole – for processes and strategies to mitigate the risks, and case studies which would be helpful to the broader community.

This is an exciting opportunity for us to apply and extend the experience we have gained working on a range of projects in the field, including the European Visual Archive Market-validation Project (EVAMP) and risk assessments for the recently launched Newsfilm Online project. We will shortly be creating an online home to for the project collaboration and development, and will use DA Blog and the DigiPresSurvey Blog (on JISCInvolve) to keep you updated.

• ‘It was all good. Thank you for the extra bits of reading in the pack too.’
• ‘This course came highly recommended and didn’t disappoint!’
• ‘Even the things that don’t directly apply to me now were useful to know for the future, and to understand the needs of our collaborators.’

We are now studying the feedback with a view to further improving the course and looking at developing some social networking opportunities as well as looking at e-learning. The next DPTP will be in February 2009. Once dates are confirmed we will post them here.