I’ve been using Web Curator Tool (WCT) to curate the JISC website collection at UKWA since 2008. I’ve long been aware that the system offered me the opportunity to record a lot of metadata, in tabs called General, Annotations, Groups and Access. It’s a mix of technical metadata (about the gather / website) and descriptive metadata. It’s mainly of value to the curator who wants to keep track of what they’re doing with the website gathering; but WCT also allows us to create some descriptive metadata for exposure. At the bare minimum, we’re required to use Groups; despite its name, this component is actually a simple subject classification scheme, allowing me to tag all my websites with “Higher Education” for example. Once stored in the WCT database and rendered through Wayback Machine, this subject selection translates into this useful view of the collection.

Recently the British Library team approached all the users of the shared WCT tool. It seems that the curators involved in UKWA have been using these metadata fields slightly differently and the BL team have initiated a project to move towards more consistency. The project will involve deciding on definitive interpretations of how to use these fields, followed by a process of cleaning up legacy data stored in the system. Some of it is potentially useful, some of it not so useful; some is legacy from the earlier PANDAS phase of the project, mostly not needed, or entered into the wrong field.

As noted, a lot of this metadata is mainly to do with selection and evaluation decisions, curation information such as changes in status of the site, and as such it’s never been exposed anywhere except within WCT. However, one descriptive field will eventually end up exposed on the UKWA live site, and provide us cataloguer types an opportunity to describe the resources in more detail. It will appear on the Title Entry Page (TEP) for each instance.

I welcome any move towards exposing more descriptive metadata on the UKWA public site. I have always taken the view that the phrase which currently appears alongside a Title “The live site may provide more information” is not really very helpful in the context of a web archive, for three reasons. (1) we don’t want our users clicking away from UKWA; (2) the link to the live site may be dead by now, and; (3) as archivists and curators, I feel strongly that we are the ones who should be providing that “more information” in the shape of a catalogue description of some kind.

The JISC project sites, as a collection, have high evidentiary value as stages in development of very specific tools, services and activities that benefit the UK Higher Education community. The sites by themselves don’t always explain their history or intentions; I would argue that a lot of rich contextual detail about the reasons these sites existed (the JISC programme under which they were developed, the dates, the staff involved, the themes, the outputs) would help interpret the collection to the users and make it more intelligible.

How to archive a website built with a wiki? It’s worth looking into this as increasingly JISC projects are using wikis to manage and report on their projects; of the available brands, MediaWiki is a popular one.

The challenge for me is how to bring in a good copy of a wiki site without causing Web Curator Tool to gather too many pages from it. We don’t want that, because (a) the finished result occupies unnecessary space in the archive and (b) because it takes so long to complete that it can hold up the gather queue in the shared web-archiving service, delaying the work of other UKWAC partners.

I am not technical enough to tell you in great detail what’s causing this, although I sense that it’s something to do with the Heritrix crawler requesting too many pages from the wiki. When you consider that a wiki is database-driven it should not surprise us that it’s creating a lot of its pages on the fly. Secondly, since a wiki is editable by lots of contributors (that’s its core function after all), it presumably means we have numerous past versions of pages also stored somewhere in the wiki labyrinth, and it’s possible that the implacable Heritrix will not cease until it’s faithfully requested and copied every single one of them.

Let’s look at the Repositories Research Team wiki (DigiRep) owned by UKOLN, which I tried to gather five times in 2008. WCT conveniently keeps a history of these attempts, information about which I can still access even if the actual gathered pages have been discarded or archived. The size problems were chronic. Of five 2008 gathers, one was aborted after it had reached a massive 16.87 GB; a second one was rejected at 14.69 GB. I have archived one impression at 5.31 GB, another at 736.26 MB and another at 157.36 MB. Quite large variations there, which was worrying enough in itself.

At first, my workaround was to adjust the Profile Setting in the title to override the maximum number of documents Heritrix can gather. Setting ‘Maximum Documents’ at 10000 worked, but it was not ideal; I suppose all this means is that Heritrix stops when it collects 10,000 pages, whether we have everything we want or not. (I found that the copies in the archive seemed to render OK however).

To get a closer look at what’s going on, I started to browse the Log Files created by WCT (complete records of every single client-server request), which show patterns which I can vaguely understand; when these Log Files are packed with near-identical strings of code I sense that something’s up. For example, a string containing index.php?title=Repositories_Research&action=edit tells us that the wiki is requesting a specific named page, and allowing an edit action on that page. If you multiply that by the number of pages in the wiki, you can see how the problem builds up. (PHP is the script used for MediaWiki’s web scripting engine).

I follow this up by browsing the actual gathered pages in Web Curator Tool using the Tree View. From here I can click on the ‘View’ button to examine a page which I think to be suspect, and compare it with other suspect pages. Lastly, I go back to the live DigiRep site to confirm in my mind what’s happening when certain links are followed.

All the above gave me just about enough information to experiment with exclusion filters. After a certain amount of trial and error, and working with other Media Wiki sites, I arrived at the following exclusion codes which I can add to the Profile Setting:

.*&oldid.*

.*&redirect=no.*

These have the effect of telling WCT to exclude certain pages and actions from Heritrix’s harvesting action. The expectation was that I would lose the discussion / edit / history functions of the wiki in the archive copy.

The title with the above exclusion profile gathered just 63.41 MB and it completed in under ten minutes. I would say that’s an improvement on 16.87 GB. Log Files and the Tree View confirmed the success of this new “slimline” gather. As well as losing the discussion / edit / history functions, we also have eliminated the Toolbox functions, the ‘printable’ views, and the login pages.

This is no great loss at all for our purposes, as scholars who browse the archived copy of DigiRep are not expecting to be able to edit pages, nor join in the discussions, nor browse the history of stored versions of pages. Indeed in a lot of cases, they would require a login to do so. The users simply want to see the results of the DigiRep team’s work.

Keen readers may recall a post from April 2008 about my website-archiving forays working with Web Curator Tool, the workflow database, used for programming Heritrix, the crawler which does the harvesting of websites.

Other UKWAC partners and myself have since found that Heritrix sometimes has a problem, described by some as ‘collateral harvesting’. This means it can gather links, pages, resources, images, files and so forth from websites we don’t actually want to include in the finished archived item.

Often this problem is negligible, resulting in a few extra KB of pages from adobe.com or google.com for example. Sometimes though it can result in large amounts of extraneous material, amounting to several MB or even GB of digital content (for example if the crawler somehow finds a website full of .avi files.)

I have probably become overly preoccupied with this issue, since I don’t want to increase our sponsor (JISC)’s overheads by occupying their share of the server space with unnecessarily bloated gathers, nor clutter up the shared bandwidth by spending hours gathering pages unnecessarily.

Web Curator Tool allows us two options for dealing with collateral harvesting. One of them is to use the Prune Tool on the harvested site after the gather has run. The Prune Tool allows you to browse the gather’s tree structure, and to delete a single file or an entire folder full of files which you don’t want.

The other option is to apply exclusion filters to the title before the gather runs. This can be a much more effective method. The method is to enter a little bit of code in the ‘Exclude Filters’ box of a title’s profile. The basic principle is using the code .* for exclusions. .*www.aes.org.* will exclude that entire website from the gather. .*/images/.* will exclude any path containing a folder named ‘images’.

So far I generally find myself making two types of exclusion:

(a) Exclusions of websites we don’t want. As noted with collateral harvesting, Heritrix is following external links from the target a little too enthusiastically. It’s easy to identify these sites with the Tree View feature in WCT. This view also lets you know the size of the folder that has resulted from the external gathering. This has helped me make decisions; I tend to target those folders where the size is 1MB or larger.

(b) Exclusions of certain pages or folders within the Target which we don’t want. This is where it gets slightly trickier, and we start to look in the log files of client-server requests for instances where the browser is staying in the target, but performing actions like requesting the same page over and over. This can happen with database-driven sites, CMS sites, wikis, and blogs.

I believe I may have had a ‘breakthrough’ of sorts with managing collateral harvesting with at least one brand of wiki, and will report on this for my next post.

At ULCC we’re interested to see if we can align some ‘continuity’ ideas within the context of our PoWR project. Many of the issues facing departmental web and information managers are likely to have analogues in HE and FE institutions, and Web Continuity offers concepts and ways of working that may be worth considering and may be adaptable to a web-archiving programme in a University.

A main area of focus for Web Continuity is integrity of websites – links, navigation, consistency of presentation. The working group on this, set up by Jack Straw, found a lot of mixed practices in e-publication (some use attached PDFs, others HTML pages); and numerous different content management systems in use. No centralised or consistent publication method, in other words.

To achieve persistency of links, Web Continuity are making use of digital object identifiers (DOIs) which can marry a live URL to a persistent identifier. Further, they use a redirection component which is derived from open-source software. It can be installed on common web server applications, eg Apache and Microsoft IIS. This component will “deliver the information requested by the user whether it is on the live website, or retrieved from the web archive and presented appropriately”. Of course, this redirection component only works if the domains are still being maintained, but it will do much to ensure that links persist over time.

They are building a centralised registry database, which is growing into an authority record of Government websites, including other useful contextual and technical detail (which can be updated by Departmental webmasters). It is a means of auditing the website crawls that are undertaken. Such a registry approach would be well worth considering on a smaller scale for a University.

Their sitemap implementation plan involves the rollout of XML sitemaps across government. XML sitemaps can help archiving, because they help to expose hidden content that is not linked to by navigation, or dynamic pages created by a CMS or database. This methodology may be something for HFE webmasters to consider, as it would assist with remote harvesting by an agreed third party.

The intended presentation method will make it much clearer to users that they are accessing an archived page instead of a live one. Indeed, user experience has been a large driver for this project. I suppose that UK Government want to ensure that the public can trust the information they find and that the frustrating experience of meeting dead-ends in the form of dead links is minimised. Further, it does something to address any potential liability issues arising from members of public accessing – and possibly acting upon – outdated information.

Web Curator Tool is a web-based workflow database, one which manages the assignment of permission records, builds profiles for each ‘target’ website, and allows a certain amount of inter-facing with Heritrix, the actual engine that gathers the materials. The open-source Heritrix project is being developed by the Internet Archive, whose access software (effectively the ‘Wayback Machine’) may also be deployed in the new public-facing website when it is launched in May 2008.

Although the idiosyncrasies of WCT caused me some anguish at first, largely through being removed from my ‘comfort zone’ of managing regular harvests, I suddenly turned the corner about two weeks ago. The diagnostics are starting to make sense. Through judicious ticking of boxes and refreshing of pages, I can now interrogate the database to the finest detail. I learned how to edit and save a target so as to ‘force’ a gather, thus helping to clear the backlog of scheduled gathers which had been accumulating, unbeknownst to us, since December. Most importantly, with the help of UKWAC colleagues, we’re slowly finding ways of modifying the profile so as to gather less external material (or reduce collateral harvesting, to put it another way); or extend its reach to capture stylesheets and other content which is outside the root URL.

True, a lot of this has been trial and error, involving experimental gathers before a setting was found that would ‘take’. But WCT, unlike our previous set-up, allows the possibility of gathering a site more than once in a day. And it’s much faster. It can bring in results on some of the smaller sites in less than two minutes.

Now, 200 new instances of JISC project sites have been successfully gathered during March and April alone. A further 50 instances have been brought in from the Jan-Feb backlog. The daunting backlog of queued instances has been reduced to zero. Best of all, over 30 new JISC project websites (i.e. those which started around or after December 07) have been brought into the new system. I’ll be back in my comfort zone in no time…

The morning’s overview of the Methods work was almost all new to me, and extremely interesting. As many attending said (and as Seamus Ross and Susan Hockey said in their informal report on the project’s work) a great deal has been achieved by this group in a short space of time, and it has been done in a way that has engaged many different research groups across the arts and humanities. I’ll certainly be finding time to read the well-illustrated report we all received in more depth soon. And courtesy of Mark Greengrass, I’ve added two new words to my vocabulary: intervenant and performativity.

It was the afternoon that I had really come for, though.Sheila Anderson whisked us through an impassioned and impressive summary of the highlights of AHDS’s achievements, at the subject centres and at the central executive, and we then heard from a range of speakers on how the work of the AHDS had affected them. I thought I was relatively well-informed on what the AHDS had achieved. I learnt today that it had done much, much more. It was wonderful to see Andrew Prescott again, who looked back to the Follett report and forwards to his current role at Lampeter to provide us with a thoughtful and engaging presentation. I will always be grateful to Andrew for his patience and enthusiasm when we worked with him in a small way on helping to preserve the digitised images of the Beowulf manuscript, and his enthusiasm for technology is still undimmed. (How many other university librarians recently switched their desktop from Vista to Ubuntu?) It would be impossible to summarise Barry Smith’s helter-skelter excursion: it had us laughing and thinking at the same time, and involved giraffes and (possibly) imaginary faxes.

And there was a great deal more to which I’ll return to anon. Matthew Woollard sent us away on a positive note, and Malcolm Read expressed (amongst other things) the unusual pleasure of being invited to the closure of a service as opposed to its launch.

I’m glad to say that my colleague Ed Pinsent told us today that we’ve managed to preserve the AHDS web pages as was in UKWAC. Everyone involved with AHDS has got good reason to be proud of its achievements. Good luck to all of you.

ULCC are not actively preserving this blog as yet. It’s backed up, and uses open standards and open source software: I don’t consider there would be any technical obstacles to capturing and preserving it like any other website in UKWAC. I have already archived some JISC blogs successfully, for example, the archived PeerPigeon project site, which contains a link to an archived copy of its own blog. It’s been good to capture a copy of this particular blog (very coincidentally, it also runs on WordPress), which only existed for nine months in 2007 and is starting to wind down. The resource is still live at time of writing, but for how much longer?

It’s also worth noting that within the UKWAC Consortium, major partners The British Library are actively building a collection of blogs (and similar sites using social software), which they call the ‘Digital Lives’ collection.

Some things to resolve with blog preservation, as with any other digital resource you want to preserve, are:

• selecting it for preservation in the first place
• resolving any copyright issues that might prevent its archiving and/or repurposing
• agreeing on a suitable frequency for harvesting of the blog
• ensuring the organisation has sufficient resources to maintain the preserved copies

The web harvesting software UKWAC have used to this point (PANDAS, which sits on top of HTTrack) has not met with any problems harvesting and copying the output from blogs. No problems are anticipated with the new Web Curator Tool, either.

This means that we’re both customers of the BL’s service and service providers, which might mean we get involved in some interesting conversations in the future. But I firmly believe that this dual involvement will be good for the service and for the partners. We’ll have a better insight into how the service is performing and what the users want from it, and are likely to be able to notice, and react to, problems more quickly if and when they arise. (At present, a howl of anguish following a lost piece of work disappearing into the ether travels more quickly down the corridor here than the email which explains what the howl was about.)

The new service still has lots of unknowns, not least its reliability under serious load. We’ve got every reason to have confidence in the tools, but sizing the infrastructure is not what one might call an exact science. I’ve got every reason to believe that my confidence in my colleagues in the infrastructure group is not misplaced, though.

Why haven’t UKWAC got a copy of this site? True, complex sites like this one tend to remain out of the reach of harvesting tools like PANDAS, which is based on HTTrack, and can’t get good results for sites which rely on complex server-side architecture. The site however is still unarchived as far as we know. ULCC’s Joanne Anthony (who had worked as the archivist for the Bernie Grant Trust) was keen to learn if there was any way of submitting the site for consideration to one of the UKWAC partners. There is indeed an online submissions form available, but this merely delivers a message to the UKWAC webmaster, who then forwards the request to the most appropriate partner. It would help considerably if the individual collection policies of each partner were made more manifest and published on the public site. But the visitor to www.webarchive.org.uk will find only a sketchy description of these policies, for example “The British Library will focus on sites of cultural, historical and political importance.”

Among UKWAC partners, the BL and the TNA are known to be directing their energies on certain specialised collection strands. These are given more descriptive paras at http://info.webarchive.org.uk/col.html, yet the underlying pattern or theme of these collections is not apparently obvious. At least three of them – the Tsunami, General Election and London Terrorist attack strands – appear to be based primarily on the fact that the sites are ephemeral and most in danger of loss (regardless of their informational or evidentiary value as records).

It is not clear how a concerned individual, or a member of the DP Community, might be empowered to somehow influence UKWAC’s collection policies for the better. In the case of the Bernie Grant website, Joanne’s interest was to see minority ethnic groups better represented in UK archival collections; but another approach would be to see it within in the larger group of ‘websites funded by Heritage Lottery Funding’. It seems likely there are many such project sites, all with short-term funding and therefore potentially at risk of being removed from cyberspace at any time, yet containing unique digital materials of huge potential cultural value. As a discrete collection of websites, it has parallels with JISC’s collection focus, ie JISC-funded projects which are occupying web space on a similar short-term lease. How can we persuade the relevant funding bodies to ensure their web outputs are archived, as JISC already does?

There are at least two sides to this mammoth task. The first (which I assume is probably relatively easy) involves moving the archive of gathered websites from its current server infrastructure to its new one. The previous hosts, Magus, have decided they can’t see a future in archiving websites. The new host, very coincidentally, is ULCC; our infrastructure services recently won the contract to provide a home for the large quantities of stored websites.

The second migration aspect, which involves complexities I’m glad I don’t have to deal with, involves moving the publisher and website profiles across from the PANDAS database to the Web Curator Tool (WCT) database. WCT, as fate would have it, is being jointly developed by the BL and the National Library of Zealand, and it will become our weapon of choice for all future web-harvesting activities. It’s certainly a more sophisticated piece of software than the clunky, web-object driven PANDAS, and appears able to handle the concept of one publisher owning more than one title (something which always baffled PANDAS).

The planned migration moves have been causing consternation to many of the UKWAC partners, particularly those who have been storing unprocessed gathers in the Magus ‘Temporary Drive’ for a long time. We at ULCC have been assisting with managing that process for months. Kevin Ashley devised a simple script that could query this drive, and report back on the occupancy broken down by website number, with figures on file sizes and dates. Ed Pinsent, by querying PANDAS, was able to match website numbers to their owners, thus providing a handy set of reports on information that was otherwise unavailable. (PANDAS wasn’t able to see these unprocessed gathers, for some reason; Magus wouldn’t run a script to report on them because they’d never been asked to, and it would probably have incurred additional charges anyway.)

The JISC occupancy of the temp drive has been negligible however. This is mainly because we have been so efficient in processing our completed gathers, and using the ftp connection has allowed us to look more closely at failed gathers. Additionally, JISC’s requirements are such that (unlike other partner members) we have rarely had to gather entire websites, instead concentrating on a few pages that constitute a JISC Project.