Keen readers may recall a post from April 2008 about my website-archiving forays working with Web Curator Tool, the workflow database, used for programming Heritrix, the crawler which does the harvesting of websites.

Other UKWAC partners and myself have since found that Heritrix sometimes has a problem, described by some as ‘collateral harvesting’. This means it can gather links, pages, resources, images, files and so forth from websites we don’t actually want to include in the finished archived item.

Often this problem is negligible, resulting in a few extra KB of pages from adobe.com or google.com for example. Sometimes though it can result in large amounts of extraneous material, amounting to several MB or even GB of digital content (for example if the crawler somehow finds a website full of .avi files.)

I have probably become overly preoccupied with this issue, since I don’t want to increase our sponsor (JISC)’s overheads by occupying their share of the server space with unnecessarily bloated gathers, nor clutter up the shared bandwidth by spending hours gathering pages unnecessarily.

Web Curator Tool allows us two options for dealing with collateral harvesting. One of them is to use the Prune Tool on the harvested site after the gather has run. The Prune Tool allows you to browse the gather’s tree structure, and to delete a single file or an entire folder full of files which you don’t want.

The other option is to apply exclusion filters to the title before the gather runs. This can be a much more effective method. The method is to enter a little bit of code in the ‘Exclude Filters’ box of a title’s profile. The basic principle is using the code .* for exclusions. .*www.aes.org.* will exclude that entire website from the gather. .*/images/.* will exclude any path containing a folder named ‘images’.

So far I generally find myself making two types of exclusion:

(a) Exclusions of websites we don’t want. As noted with collateral harvesting, Heritrix is following external links from the target a little too enthusiastically. It’s easy to identify these sites with the Tree View feature in WCT. This view also lets you know the size of the folder that has resulted from the external gathering. This has helped me make decisions; I tend to target those folders where the size is 1MB or larger.

(b) Exclusions of certain pages or folders within the Target which we don’t want. This is where it gets slightly trickier, and we start to look in the log files of client-server requests for instances where the browser is staying in the target, but performing actions like requesting the same page over and over. This can happen with database-driven sites, CMS sites, wikis, and blogs.

I believe I may have had a ‘breakthrough’ of sorts with managing collateral harvesting with at least one brand of wiki, and will report on this for my next post.

In the Handbook we’ve tried to cover a huge, and sometimes controversial, area in as accessible a way as possible. The workshops, attended by both web-management and records-management professionals from HE institutions, brought  a wide range of concerns and issues to light. It’s been quite a job fitting it all in.

Even as the project progressed, we became aware of new developments in thinking about how to approach the special issues of managing web resources, including everybody’s favourite new fast automatic Web 2.0 applications. We saw the publication of Steve Bailey’s Records Management 2.0 book, TNA’s Web Continuity project, and further web archiving developments at UKWAC. We’ve even heard it whispered in some quarters that approaches to preservation may need a more profound reassessment in the context of the Web and the Cloud. Many of these issues were recorded on the PoWR blog, and we tried to reflect as much of this in the Handbook as possible.

Another recent JISC publication, The Digital Preservation Policies Study by Charles Beagrie Ltd, published at the same time, is complementary in many ways, and reassured us that many of the conclusions we groped towards in the Handbook were not so wide of the mark! Like PoWR, the  Digital Preservation Policies Study identified the necessity of high-level policy engagement as the sine qua non of effective digital preservation.

Digital preservation solutions are undoubtedly partly technical, and the tools being created will enhance digital longevity, but these solutions are also equally dependent on organisational issues. It is important to remember that digital preservation relies on the interaction between the digital preservation environment and wider organisational objectives and procedural issues. These could be financial and staffing issues, collection management, legal obligations, auditing requirements, and other strategies and policies. In this respect, recognition by organisational divisions that digital data is important and key to the successful running of an organisation is crucial.

The Digital Preservation Policies Study, p.11

Among the other recommendations the Study shares with PoWR include:

• Analysis of existing policies and strategies, and how our work can support them even if said polices don’t explicitly refer to preservation or digital assets
• Taking a phased approach – nothing happens all at once. (PoWR recommends pilot projects and working with supportive departments.)
• Careful scoping of preservation requirements. (With regard to web resources, PoWR suggests not everything, not every version, and not forever.)
• Identifying if and where existing systems will do the job
• Consideration of lifecycle, publication, and retention schedules.

The Charles Beagrie survey is a very concise and accessible contribution to the field, and we hope the PoWR Handbook, with its specific focus on established and emerging Web issues, and attention to the detailed and everyday concerns of our many contributors and correspondents, will be similarly useful. We also hope that the work of PoWR will continue in some form, on the blog and perhaps in the form of new projects and workshops, to fill in the gaps we left, and deal with the constantly emerging Web developments. Anyone for PoWR 2.0?

As Richard noted in Part I, digital preservation is a “series of managed activities necessary to ensure continued access to digital materials for as long as necessary.” But what sort of digital materials might be in scope for the PoWR project?

We think it extremely likely that institutional web resources are going to include digital materials such as “records created during the day-to-day business of an organisation” and “born-digital materials created for a specific purpose”.

What we want is to “maintain access to these digital materials beyond the limits of media failure or technological change”. This leads us to consider the longevity of certain file formats, the changes undergone by proprietary software, technological obsolescence, and the migration or emulation strategies we’ll use to overcome these problems.

By migration we mean “a means of overcoming technological obsolescence by transferring digital resources from one hardware/software generation to the next.” In contrast, emulation is “a means of overcoming technological obsolescence of hardware and software by developing techniques for imitating obsolete systems on future generations of computers.”

Note also that when we talk about preserving anything, “for as long as necessary” doesn’t always mean “forever”. For the purposes of the PoWR project, it may be worth us considering medium-term preservation for example, which allows “continued access to digital materials beyond changes in technology for a defined period of time, but not indefinitely.”

We also hope to consider the idea of life-cycle management. According to DPC, “The major implications for life-cycle management of digital resources is the need actively to manage the resource at each stage of its life-cycle and to recognise the inter-dependencies between each stage and commence preservation activities as early as practicable.”

From these definitions alone, it should be apparent that success in the preservation of web resources will potentially involve the participation and co-operation of a wide range of experts: information managers, asset managers, webmasters, IT specialists, system administrators, records managers, and archivists.

(All the quotations and definitions above are taken from the DPC’s online handbook.)