If you haven’t seen it, can I recommend Kristen Snawder’s recent post on the Library of Congress Digital Preservation blog, Digitization is different than digital preservation. Kristen reiterates familiar points about the long-term commitment necessary for serious digital preservation, contrasted with the quick hit of a scanning project. “In the hurry to meet user expectations, institutions may scan large quantities of materials without having a solid plan for preserving the digital images into the future.”

However another recent find on the Web compels me to make an additional point, namely that we might do equally well to differentiate between scanning and digitisation. Anyone can set to work with a scanner and create a bunch of digital images – but that barely scratches the surface of what I think we should be expecting of a digitisation project in 2011.

First and foremost, we need metadata: the more the merrier, but something at least. Even if we expect to come back later and polish it up (once the images can be browsed and examined on screen). In the absence of any established metadata profiles for a project, at least try to cover as many Dublin Core elements as possible – title, creator, date, subject/keywords… Images, in particular, may prove tricky or time-consuming to find again, especially once there are thousands of them on a disk. We should probably keep the metadata in a database, and perhaps additionally store metadata with the objects. This can be as XML or plain text files stored alongside the digital images, or embedded in the files we create (many common file formats – TIFF, JPEG, MPEG, PDF – support metadata embedding, and there are many free tools available to help).

There is yet more, though, that we should be doing, particularly when we are scanning text-based objects (articles, books, magazines, reports, etc). Most importantly, we really should try and extract the text from the image if possible. ^[1]

My recent web find was the teaching blog of Dr Toine Bogers at the Royal School of Library and Information Science (RSLIS) in Copenhagen, Denmark. One fascinating post describes a Lab Session exercise, From OCR To NER, a set of comparatively simple command-line processes to get the most out of a scanned-text project.

Toine’s post walks us through the process. Once the article is scanned, we should apply some OCR. The exercise goes further and also describes the use of tools to clean up and spell-check the resulting OCR’d text. This will, at the very least, result in a separate text file, hopefully containing a fairly accurate version of the article text. Finally, the cleaned-up text can be submited to a Named Entity Recognition service. Toine’s exercise uses NER tools at University of Illinois. (We’ve been using similar functionality provided by OpenCalais and GATE for our AIM25 Open Metadata project.)

Why do all this? The most important, instant, result of this is that we can now easily index our article for full-text searching – in a local repository system, such as EPrints or DSpace provide – and of course by Google. None of this is possible if we leave the scanned image as just that – an image.

Another  side-effect of any successful OCR outcome, is that the text is now free to be re-flowed. This means that we might consider sharing it with users in a variety of forms enhancing usability and accessibility.

It’s important not to confuse preservation formats with formats for access and dissemination. You probably will have your scanned image masters in TIFF, RAW, JPEG2000, PostScript, SVG. None of these are likely to be of much use to your users over the Web. Not only are the formats not widely supported by Web browsers, but most users probably don’t need or want your master image. If it’s a high-resolution scan of a 100 page book, they might be looking at 100Mb download, or worse – slow to load, and probably slow to render and navigate.

Time taken thinking what formats will give users the best experience is time well spent. What platforms might they want to use now and in the foreseeable future? It’s less than 18 months since Kindle3 made e-book readers affordable, and the Ipad made them sexy. E-books look and function very impressively on both platforms (albeit in different ways): for an overview of some of the benefits of the EPUB format, see Martin Fenner’s post Beyond The PDF… is EPUB. PDF outputs may yet have their uses, if users can at least search for text within them. The point is that only with properly digitised text, do these kinds of accessibility options become possible.

Even image collections can also be disseminated as E-books – nice offline items some users might care to flick through on their tablet computers, possibly even smartphones. I’ve demonstrated how we can create OJS XML from EPrints XML on-the-fly with XSLT: since EPUB and Mobi/Kindle are XML-based formats, we should be able to do something similar to create e-books using repository APIs. Also, by using appropriately sized images in dissemination formats (Ipad screen is 1024x768px; Iphone4 is 960x640px) we can not only ship our users a sensibly-sized download, we can protect any capital we may have in the master images, without having to resort to ugly tricks like watermarking. (Giving users full-size, high-res images with embedded watermarks seems to me the worst of all worlds.)

Therefore I’d suggest that, in order to get the best out of a digitisation project, consider what would you like to see at the end of the project – and, more importantly, what would give your users the best experience, or even win you new users? Ask around, do some tests, with users if possible, and get an idea how they want to use the materials and how they will get the best out of them. Maybe there are comparable projects and systems that you admire, with features you’d like to be available for your collection. What about in five or ten years’ time: will your current project outputs help or hinder longer term accessibility goals?

This kind of vision of is essential. Without some conception of the end result, how the materials will be used and managed most effectively, all the scanning in the world isn’t going to amount to a successful digitisation project.

My naïve thinking in the area has been characterised by the assumption that the process is rather static or linear, and that the problem we’re facing is broadly the same every time; migrate data from a format that’s about to become obsolete or unsupported, onto another format that’s stable, supported, and open. MS Word document to PDF or PDF/A…now that, I can understand!

In fact, I learned at least two ways of thinking about formats that hadn’t occurred to me before. One simple one is costs; some formats can cost more to preserve than others. This can be calculated in terms of storage costs, multiplied over time, and the costs associated with migrations to new versions of that format. For example, we’ve tended to pin our faith on the TIFF format for images for many reasons, but there’s a high storage price to be paid for all that wonderful losslessness. This may be one reason why the DP world is looking with more favour on the JPEG2000 format, which is ‘virtually’ lossless and smaller in size.

Secondly, the problems of preserving digital data which doesn’t actually have a specified stable preservation format. Chris Puttick of Oxford Archaeology gave a vivid description of the problems he’s facing with CAD and GIS files, where the data can’t easily be tied to a single format in the first place (nor can a stable format for migration be identified). As the NLA put it on their PADI page, “At present there is little dealing specifically or comprehensively with the preservation of this particular type of data, although some aspects of database preservation are applicable to GIS. Some long term preservation issues include a lack of open source formats and metadata standards, large data volume and complex data objects.” Puttick suggests that his data doesn’t really perform at all unless it’s operated within a very specific environment of hardware and software. How do we preserve an environment? This appears to be quite a distinct preservation problem and much harder to solve than Word to PDF, to put it mildly.

William Kilbride suggested that such cases (and websites too, arguably, because they are time-based) are more like a stream of data – a handy image which conveys something about the dynamic of such information packages, and showing us that it’s much harder to nail them down into a single format. You can never step into the same river twice.

The proprietary nature of RAW formats (an emerging OpenRAW standard notwithstanding) and the relative newness on the block of JPEG 2000 would both tend to bolster the longstanding preference for TIFF, but as Chris’s posts make clear, each preservation project should nevertheless weigh the options based on its own requirements and resources.

If in doubt, the “keep everything” approach is attractive, as ever, but – in spite of the old mantras about ever-cheaper filestore – the implications for storage space and management are potentially very costly once one enters the world of Terabytes and Petabytes. In one example, Sean Martin of BL concludes that

probably only a small amount of additional value is created for the additional expense approaching £200K. This leads to the question “if we had £200K on what would we spend it?” and probably the answer is “not in this way”.

The posts contain many interesting examples of costings, based on the experience of BL, SDSC and others. I won’t even attempt to summarise (Chris’s summary of) them here, but I hope that this post will still be available for me to consult next time I have to dabble in the murky science of DP costings.

In some instances, the case for preserving the RAW image may nevertheless be compelling. One can imagine that the risk of missing a new planet or virus (or identifying a non-existent one), or other potential infelicities in scientific and medical imaging, is not worth contemplating. By contrast, it’s hard to see what value might be added to a collection like the Linnean Society’s by keeping raw camera data alongside the TIFFs, and more than doubling the amount of storage capacity required.

It’s for making decisions like this that the intellectual exercise of identifying significant properties and the needs of all stakeholders – creators, curators and users – seems particularly essential.

It occurred to me that the blog is the perfect place to invite any passing readers to suggest other key readings, so if you know of any other recent, authoritative work in file formats, please leave a link in the comments box below.

I say ‘silently’ because the information about the effect that this update will have is extremely well hidden. Many individuals, and many large organisations, will do as MS recommends and allow their systems to download and install updates like this automatically. They’ll be caught unawares. Others may be more like me and read through the release information to see what the update is going to do beforehand. They would be caught unawares. The information linked to at the front of this post tells you that this is a major security update for Office. It says nothing about losing access to old files. It links to another page which goes into greater detail about what the update contains. That doesn’t mention the issue either, except that under the “known issues” section, it says:

After you install Office 2003 SP3, you may receive an error message when you try to open or to save a file. For more information about this issue, click the following article number to view the article in the Microsoft Knowledge Base:

And even that article doesn’t really list things in detail, although it does mention that older office formats are amongst those affected. The recommended solutions are not ideal either.

Microsoft has belatedly realised that it was a bad idea and released a patch to circumvent the problem. But you need to explicitly request that and many people won’t realise they need it for some time.

There’s been a good deal of discussion on the e-recs list about this, including this remark from Marc Fresko:

Personally, I think this is one of the more “visible” wake-up calls that
should contribute to everyone realising that long-term access to
electronic records, even over less than ten years, cannot be ignored.
Active management is needed. Anything that helps to get this message
across is welcome, so long as it is (as is the case here) practically
harmless.

I agree with Marc that it’s a useful wake-up call. Consider that this could cause problems even to those organisations that had taken explicit steps to ensure continued access:

• You checked when moving to Office 2003 that you could still access your older documents using it.
• You checked the documentation with each update to ensure that it wasn’t going to invalidate your earlier checks

With those precautions, you would still be caught out by a silent update that you cannot uninstall. Only if yours is the sort of organisation that explicitly tests all updates in a safe environment, and specifically includes a check for access to legacy material as part of those tests, would you be alerted to the problem in time.

Microsoft aren’t the only software providers to build in automated update features to their products – the practice is common with both propretary and open-source products. Incidents like this will occur again. It reinforces the belief that one needs to have good control over the application(s) needed to access preserved content as well as the formats used to preserve it, whether one is looking at 5-year retention or 500-year retention.