CERN Computer Centre

CERN Computer Racks

Last week I was in a very welcoming Geneva, exactly in the European Organization for Nuclear Research (CERN) to meet other partners working on BlogForever and to have several Invenio workshops. I felt very lucky to be in the hub of such an organization and to see how many young international students are getting the opportunity to be in the forefront of high physics research.

The Globe at CERN

CERN is home of the world’s biggest and most powerful particle accelerator, the Large Hadron Collider (LHC). This machine is installed in a 27 km circumference tunnel. The LHC records around 15 petabytes per year. All the data is stored in their vast computer centre, where open access and sharing has been the driving principle since their foundation in 1954 and an inspirational environment for the Web to be born there.

Invenio’s workshops showed us that their electronic document management system is robust and versatile, targets the management of more than 1.2 million documents and it can be used in 19 different languages. Its content is clean and complete. In just their High Energy Physics domain, they have around 700 collections and approximately 20K queries a day. As well Invenio is used for special programs like the UNESCO funded digital repos in Africa and EU funded projects like D4Science and OpenAIRE.

Me at CERN

In the case of BlogForever and Invenio, plenty of work to be done by the Invenio Team at the User and Document Service Group. At the moment, they have more than 30 readily available Python modules that can be adapted to the case of preserving huge amount of blogs. From the point of view of my work with repositories as part of the Digital Archives and Repositories Team at ULCC, I was inspired by Invenio’s advance search engines; indexing and ranking methods.

In a more personal level, if you are ever crossing the border between France and Switzerland near Geneva, get Tram 18 and hop off at CERN to see their Microcosm and Globe exhibitions.

Recently the British Library team approached all the users of the shared WCT tool. It seems that the curators involved in UKWA have been using these metadata fields slightly differently and the BL team have initiated a project to move towards more consistency. The project will involve deciding on definitive interpretations of how to use these fields, followed by a process of cleaning up legacy data stored in the system. Some of it is potentially useful, some of it not so useful; some is legacy from the earlier PANDAS phase of the project, mostly not needed, or entered into the wrong field.

As noted, a lot of this metadata is mainly to do with selection and evaluation decisions, curation information such as changes in status of the site, and as such it’s never been exposed anywhere except within WCT. However, one descriptive field will eventually end up exposed on the UKWA live site, and provide us cataloguer types an opportunity to describe the resources in more detail. It will appear on the Title Entry Page (TEP) for each instance.

I welcome any move towards exposing more descriptive metadata on the UKWA public site. I have always taken the view that the phrase which currently appears alongside a Title “The live site may provide more information” is not really very helpful in the context of a web archive, for three reasons. (1) we don’t want our users clicking away from UKWA; (2) the link to the live site may be dead by now, and; (3) as archivists and curators, I feel strongly that we are the ones who should be providing that “more information” in the shape of a catalogue description of some kind.

The JISC project sites, as a collection, have high evidentiary value as stages in development of very specific tools, services and activities that benefit the UK Higher Education community. The sites by themselves don’t always explain their history or intentions; I would argue that a lot of rich contextual detail about the reasons these sites existed (the JISC programme under which they were developed, the dates, the staff involved, the themes, the outputs) would help interpret the collection to the users and make it more intelligible.

During the ArchivePress project at ULCC, we briefly considered the data and metadata generally made available with blogs and blog posts. As ArchivePress focused on the representations of blogs in newsfeeds, we examined the metadata that is generated in common, and exposed in the newsfeeds of three of the most common blog platforms, WordPress, Blogger and TypePad. Blogger and Typepad prefer the Atom newsfeed format; WordPress (particularly WordPress.com) prefers RSS (though it can be made to publish Atom feeds too). This analysis was done, about a year ago, things may have changed, but here is a summary of what we found.

For each Blog, the following core information is available in the feeds:

For each Post, we established that the following core information is available in the newsfeeds:

One interesting point we noted was that neither Blogger nor Typepad published a link to a Comments Feed for each post. This made our work on ArchivePress more difficult since it was predicated on being able to easily identify the Comments feed for each post, and harvest new Comments as they were published. Obviously for blogs generated other than by WordPress, this was not going to be so easy. (Our ace developer Emanuele found some workarounds, but that’s another story.)

I think this offers us an interesting overview of the core of standard, structured blog data and metadata, in three of the leading blog platforms. This is the data structure and metadata profile that is maintained in blog databases, in one of its native forms, and I’d expect it to be present in all blog platforms, since it arguably represents the essence of blogs. I hope this will be useful background when considering the core models for data and metadata handling that will be developed for BlogForever.

Is there a particular blog or blogger you would like to see included in the BlogForever survey? We invite you to use this form to nominate them, and we will try to ensure that the blog is reviewed or the blogger contacted to participate in our survey.

At an atomic level, a “blog” comprises “blog posts”, which are continually added to the blog corpus: that is the dynamic essence of a blog, and distinguishes it from old-fashioned, largely static Websites and hypertexts in which little content changed between major update iterations, which process was probably more akin to “publishing a new edition” in the world of non-digital publications.

The blog also displays, as part of its frame, other graphical and functional elements (sidebars, widgets, “blogrolls”, etc) which may themselves contain dynamically updated, constantly changing information. These can be added, removed, amended and rearranged at will by the blog author/editor. Blog posts that were “published” in the context of one set of framing elements, will persist through subsequent versions of that framework.

Similarly with design (layout, colours, mastheads, etc), though the persistence tends to be longer, the informal nature of blogs means that these may be easily changed by the blog editor/author, and are thus more volatile than a typical “corporate” website. Again, blog posts may persist, unchanged in themselves, through many iterations of the blog site design and layout.

A simple view of blog elements and their temporal relationship

This very simplified visualisations suggests where we might start conceptualising key elements of a blog. It indicates that they iterate over time, but in the cases of Design, Posts and Widgets (as we’ll call them for brevity), according to independent schedules. While Posts and Comments persist in the online view of a blog, designs and widget arrangements are overwritten.

With my earlier ArchivePress project we deliberately overlooked preservation of the blog’s framing elements, and (given the much smaller scope of that project) established an acceptable rationale for doing so. The challenge for BlogForever is to find a solution to  precisely these issues. Unless we were simply to adopt the snapshot approach of Heritrix-based web archiving initiatives (e.g. Wayback/archive.org, UK Web Archive), we need to ensure the BlogForever repository supports a degree of granularity that can capture, describe and preserve atomic blog objects in a way that reflects the particular interdependencies, in order to understand and preserve them authentically, and permit the many possible authentic and valid “time slice” views and analyses that users of the archive will need.

(I appreciate, by the way that these objects themselves are compound objects, so not strictly “atomic”: but the same is also true of atoms, as our CERN colleagues can attest!)

1. Consider your organisational requirements, drivers and level at which you want to do it, and create a selection policy that matches this. Consider legal framework.
2. Understand the technology that copies websites (harvesters) and how websites themselves behave. I talked about aspects of the dynamic web that sometimes trip up a harvester – CMS, wiki, databases.
3. Consider how (and indeed whether) you want to offer access to the collection, and whether metadata is needed.
4. Build a programme for web archiving, adapting existing methodologies as needed – e.g. Institutional vs Individual. What other services exist, and can they do it for you?

In the middle, we had an excellent case study from Dave Thompson at the Wellcome Trust, and his experiences strongly reflected many of the themes of the course. Like many organisations, they don’t have one single reason for collecting web archives, and the future value of these collections is something we can’t yet see (due to its closeness with the live web).

We were all impressed by the people attending the course, all from a variety of backgrounds and projects, coming with widely different expectations of how they would be managing their web content. National libraries and business archives were represented, but also the arts; the Tate Gallery are doing interesting work in time-based media and specialist works of art that manifest themselves over the web. How to capture that content, and make it perform in the future?

The DPTP recognises the value of participation and sharing experiences, which we can all learn from. When I was holding forth on the concept of three possible points of capture for web content, I was very pleased to hear a proposal for a fourth possible method from our Swiss delegate Daniel Spichty. There were also numerous questions about exactly what it is that Content Management Systems do, which suggested to me I need to learn more about the inner workings and preservation implications of such systems.

I’m also pleased we were able to offer a printed copy of the PoWR Handbook to all those attending – in advance of the official launch of the book, which will take place at IWMW 2010 on 12 July.

The challenge for me is how to bring in a good copy of a wiki site without causing Web Curator Tool to gather too many pages from it. We don’t want that, because (a) the finished result occupies unnecessary space in the archive and (b) because it takes so long to complete that it can hold up the gather queue in the shared web-archiving service, delaying the work of other UKWAC partners.

I am not technical enough to tell you in great detail what’s causing this, although I sense that it’s something to do with the Heritrix crawler requesting too many pages from the wiki. When you consider that a wiki is database-driven it should not surprise us that it’s creating a lot of its pages on the fly. Secondly, since a wiki is editable by lots of contributors (that’s its core function after all), it presumably means we have numerous past versions of pages also stored somewhere in the wiki labyrinth, and it’s possible that the implacable Heritrix will not cease until it’s faithfully requested and copied every single one of them.

Let’s look at the Repositories Research Team wiki (DigiRep) owned by UKOLN, which I tried to gather five times in 2008. WCT conveniently keeps a history of these attempts, information about which I can still access even if the actual gathered pages have been discarded or archived. The size problems were chronic. Of five 2008 gathers, one was aborted after it had reached a massive 16.87 GB; a second one was rejected at 14.69 GB. I have archived one impression at 5.31 GB, another at 736.26 MB and another at 157.36 MB. Quite large variations there, which was worrying enough in itself.

At first, my workaround was to adjust the Profile Setting in the title to override the maximum number of documents Heritrix can gather. Setting ‘Maximum Documents’ at 10000 worked, but it was not ideal; I suppose all this means is that Heritrix stops when it collects 10,000 pages, whether we have everything we want or not. (I found that the copies in the archive seemed to render OK however).

To get a closer look at what’s going on, I started to browse the Log Files created by WCT (complete records of every single client-server request), which show patterns which I can vaguely understand; when these Log Files are packed with near-identical strings of code I sense that something’s up. For example, a string containing index.php?title=Repositories_Research&action=edit tells us that the wiki is requesting a specific named page, and allowing an edit action on that page. If you multiply that by the number of pages in the wiki, you can see how the problem builds up. (PHP is the script used for MediaWiki’s web scripting engine).

I follow this up by browsing the actual gathered pages in Web Curator Tool using the Tree View. From here I can click on the ‘View’ button to examine a page which I think to be suspect, and compare it with other suspect pages. Lastly, I go back to the live DigiRep site to confirm in my mind what’s happening when certain links are followed.

All the above gave me just about enough information to experiment with exclusion filters. After a certain amount of trial and error, and working with other Media Wiki sites, I arrived at the following exclusion codes which I can add to the Profile Setting:

.*&oldid.*
.*&diff.*
.*&limit.*
.*&direction.*
.*Recentchanges.*
.*/Special.*
.*\?title=Special.*
.*&action=edit.*
.*&action=history.*
.*&section.*
.*&redlink.*
.*&printable=yes.*
.*&redirect=no.*

These have the effect of telling WCT to exclude certain pages and actions from Heritrix’s harvesting action. The expectation was that I would lose the discussion / edit / history functions of the wiki in the archive copy.

The title with the above exclusion profile gathered just 63.41 MB and it completed in under ten minutes. I would say that’s an improvement on 16.87 GB. Log Files and the Tree View confirmed the success of this new “slimline” gather. As well as losing the discussion / edit / history functions, we also have eliminated the Toolbox functions, the ‘printable’ views, and the login pages.

This is no great loss at all for our purposes, as scholars who browse the archived copy of DigiRep are not expecting to be able to edit pages, nor join in the discussions, nor browse the history of stored versions of pages. Indeed in a lot of cases, they would require a login to do so. The users simply want to see the results of the DigiRep team’s work.

Keen readers may recall a post from April 2008 about my website-archiving forays working with Web Curator Tool, the workflow database, used for programming Heritrix, the crawler which does the harvesting of websites.

Other UKWAC partners and myself have since found that Heritrix sometimes has a problem, described by some as ‘collateral harvesting’. This means it can gather links, pages, resources, images, files and so forth from websites we don’t actually want to include in the finished archived item.

Often this problem is negligible, resulting in a few extra KB of pages from adobe.com or google.com for example. Sometimes though it can result in large amounts of extraneous material, amounting to several MB or even GB of digital content (for example if the crawler somehow finds a website full of .avi files.)

I have probably become overly preoccupied with this issue, since I don’t want to increase our sponsor (JISC)’s overheads by occupying their share of the server space with unnecessarily bloated gathers, nor clutter up the shared bandwidth by spending hours gathering pages unnecessarily.

Web Curator Tool allows us two options for dealing with collateral harvesting. One of them is to use the Prune Tool on the harvested site after the gather has run. The Prune Tool allows you to browse the gather’s tree structure, and to delete a single file or an entire folder full of files which you don’t want.

The other option is to apply exclusion filters to the title before the gather runs. This can be a much more effective method. The method is to enter a little bit of code in the ‘Exclude Filters’ box of a title’s profile. The basic principle is using the code .* for exclusions. .*www.aes.org.* will exclude that entire website from the gather. .*/images/.* will exclude any path containing a folder named ‘images’.

So far I generally find myself making two types of exclusion:

(a) Exclusions of websites we don’t want. As noted with collateral harvesting, Heritrix is following external links from the target a little too enthusiastically. It’s easy to identify these sites with the Tree View feature in WCT. This view also lets you know the size of the folder that has resulted from the external gathering. This has helped me make decisions; I tend to target those folders where the size is 1MB or larger.

(b) Exclusions of certain pages or folders within the Target which we don’t want. This is where it gets slightly trickier, and we start to look in the log files of client-server requests for instances where the browser is staying in the target, but performing actions like requesting the same page over and over. This can happen with database-driven sites, CMS sites, wikis, and blogs.

I believe I may have had a ‘breakthrough’ of sorts with managing collateral harvesting with at least one brand of wiki, and will report on this for my next post.

At ULCC we’re interested to see if we can align some ‘continuity’ ideas within the context of our PoWR project. Many of the issues facing departmental web and information managers are likely to have analogues in HE and FE institutions, and Web Continuity offers concepts and ways of working that may be worth considering and may be adaptable to a web-archiving programme in a University.

A main area of focus for Web Continuity is integrity of websites – links, navigation, consistency of presentation. The working group on this, set up by Jack Straw, found a lot of mixed practices in e-publication (some use attached PDFs, others HTML pages); and numerous different content management systems in use. No centralised or consistent publication method, in other words.

To achieve persistency of links, Web Continuity are making use of digital object identifiers (DOIs) which can marry a live URL to a persistent identifier. Further, they use a redirection component which is derived from open-source software. It can be installed on common web server applications, eg Apache and Microsoft IIS. This component will “deliver the information requested by the user whether it is on the live website, or retrieved from the web archive and presented appropriately”. Of course, this redirection component only works if the domains are still being maintained, but it will do much to ensure that links persist over time.

They are building a centralised registry database, which is growing into an authority record of Government websites, including other useful contextual and technical detail (which can be updated by Departmental webmasters). It is a means of auditing the website crawls that are undertaken. Such a registry approach would be well worth considering on a smaller scale for a University.

Their sitemap implementation plan involves the rollout of XML sitemaps across government. XML sitemaps can help archiving, because they help to expose hidden content that is not linked to by navigation, or dynamic pages created by a CMS or database. This methodology may be something for HFE webmasters to consider, as it would assist with remote harvesting by an agreed third party.

The intended presentation method will make it much clearer to users that they are accessing an archived page instead of a live one. Indeed, user experience has been a large driver for this project. I suppose that UK Government want to ensure that the public can trust the information they find and that the frustrating experience of meeting dead-ends in the form of dead links is minimised. Further, it does something to address any potential liability issues arising from members of public accessing – and possibly acting upon – outdated information.

To warm up my brain before the afternoon’s PoWR workshop on preserving web resources, I sampled James Currall’s enjoyable discussion of web archiving, and also a talk on Institutional Repositories by Stephanie Taylor of UKOLN and RRT.

In The Tangled Web is but a Fleeting Dream … but then again…, James Currall covered the essentials of web archiving in a clear and engaging way, drawing comparisons between the survival of WWI soldiers’ diaries, and the blogs of present-day servicemen in Iraq. Another example given was the trials and tribulations of the website for the Lockerbie Trial Briefing Unit: outsourced, host ceased trading, domain name lapsed, original website content dependent on outdated Coldfusion and Access environment. Thankfully a remote-harvested, static HTML image of the site does survive.

Why have there not been more conspicuous successes in web archiving in the past two decades? Partly because it can be difficult to decide exactly whether and when a website should be treated as an authentic (and authenticable) record, a publication channel, or a publication itself – among other things. Partly because there are always waves of innovation continually washing up more interesting things to do. Partly because archiving is a policy issue, yet is generally addressed as purely a techn(olog)ical problem (where it is addressed at all).

It was reassuring, as ever, to hear it said again that there is not one single tool that addresses all possible web preservation issues (behaviour, dynamic content, scripts, versioning, emerging standards, etc.); that depending on the Internet Archive is at best a partial and risky solution; and that “whatever you do is likely to be imperfect”.

James put the chamber’s electronic voting systems to entertaining and informative use with a number of snap votes: would, for example, present-day soldiers’ blogs still be available in 90 years’ time? Of course I could confidently vote ‘yes’, knowing that the publication of the JISC PoWR handbook is barely months away! Other ad hoc vox pops revealed that the audience was about as familiar with OAIS as with the Book of Ezra.

James is director of the Espida Project, and this was a timely reminder that we must consider the relevance of that project’s work on assessing and controlling costs to the guidance we’re assembling for the forthcoming PoWR handbook. James and I share what must be a fairly unusual distinction of citing Thomas Carlyle in support of our cause in recent presentations. Unlike James, however, I don’t bear such an uncanny likeness to the great man.

Stephanie Taylor’s talk, Institutional Repositories: Asset or Obstacle?, gave us a brief history of the Insitutional Repository, and a well-paced explanation of the value and purpose of IRs – rather more substantial, thankfully, than my Bluffer’s Guide To IRs last year. I was initially intrigued that her presentation might be leading to the conclusions that IRs are “obstacles”, but I quickly realised the alarmist title was rhetorical in nature. The talk was, among other things, an engaging appeal to the better nature of web managers and other techies to appreciate the value of, and issues faced by librarians.

In fact, the emergence of repositories, and the recognition of their place among institutional information systems, is a watershed in the evolution of electronic resources. Leaving it to researchers and teachers to manage what they create – variously on websites, blogs, Google Docs, thumb drive, or what you will – is no more sensible now than it ever was, if we want to ensure that they are consistently managed and accessible, let alone think about their preservation over the longer term.

There are many ways to approach it, none the only right and proper way. You may add extra value to it, through your choice of software or implementation (in-house or outsourced), or by using Web-Two-Oh-ish features and services. Institutional Repositories are only “one of many small conversations going on in different ways in different mediums”, but the need for an institution to manage valuable academic outputs in an orderly way is unarguable.

However one tweet that flew over the Twitosphere during the afternoon suggests that there may be more to do in assessing the pros and cons of different approaches – particularly the merits or otherwise of Google’s omnipresent panaceas. Mike Ellis commented: “I’ve never been clear why it is that institutions would trust a repository vendor more than someone like Google….?”

I won’t write up the JISC-PoWR workshop – the results of that will be on the JISC-PoWR blog - except to say that Marieke succinctly summarised the many web preservation issues we’ve accumulated, while Brian found effective waysto draw out issues and concerns around the growing range of Web 2.0 applications finding favour among staff and students at our institutions; and we were pleased that both Stephanie and James were able to contribute to the discussion.

And then it was off to the Aberdeen City Art Gallery for a glass of wine or three and some inspirational art….

Not everything on the web needs to be kept. And there’s more than one way to go about keeping it – often it’s just the information that needs to survive, and the particular way it is presented on a web site today is not, of itself, worthy of long-term preservation. Yet there’s a lack of knowledge about *how* to preserve web resources, and even when people know how to do it, for some reason it just doesn’t happen. That’s not a situation I feel comfortable with.

Thus, we’re pleased to be cooperating with UKOLN who are leading a short JISC project to produce guidelines on the preservation of web resources in UK academic institutions. We have a memorable acronym (POWR) and a project-specific blog at JISC Involve, a relatively new service which allows for the creation of JISC project blogs. We’ll be doing most of the project blogging there rather than here on DABlog. In using services like JISC Involve, we’ll (hopefully) go some way towards understanding whether such services create or solve problems in the area of resource persistence.

Through workshops and through the blog, we’re keen to gain input from everyone with an interest in this problem, particularly from the web managers who are most likely to be affected by the guidelines we’re charged with producing. Do contribute your thoughts.

Web Curator Tool is a web-based workflow database, one which manages the assignment of permission records, builds profiles for each ‘target’ website, and allows a certain amount of inter-facing with Heritrix, the actual engine that gathers the materials. The open-source Heritrix project is being developed by the Internet Archive, whose access software (effectively the ‘Wayback Machine’) may also be deployed in the new public-facing website when it is launched in May 2008.

Although the idiosyncrasies of WCT caused me some anguish at first, largely through being removed from my ‘comfort zone’ of managing regular harvests, I suddenly turned the corner about two weeks ago. The diagnostics are starting to make sense. Through judicious ticking of boxes and refreshing of pages, I can now interrogate the database to the finest detail. I learned how to edit and save a target so as to ‘force’ a gather, thus helping to clear the backlog of scheduled gathers which had been accumulating, unbeknownst to us, since December. Most importantly, with the help of UKWAC colleagues, we’re slowly finding ways of modifying the profile so as to gather less external material (or reduce collateral harvesting, to put it another way); or extend its reach to capture stylesheets and other content which is outside the root URL.

True, a lot of this has been trial and error, involving experimental gathers before a setting was found that would ‘take’. But WCT, unlike our previous set-up, allows the possibility of gathering a site more than once in a day. And it’s much faster. It can bring in results on some of the smaller sites in less than two minutes.

Now, 200 new instances of JISC project sites have been successfully gathered during March and April alone. A further 50 instances have been brought in from the Jan-Feb backlog. The daunting backlog of queued instances has been reduced to zero. Best of all, over 30 new JISC project websites (i.e. those which started around or after December 07) have been brought into the new system. I’ll be back in my comfort zone in no time…

ULCC are not actively preserving this blog as yet. It’s backed up, and uses open standards and open source software: I don’t consider there would be any technical obstacles to capturing and preserving it like any other website in UKWAC. I have already archived some JISC blogs successfully, for example, the archived PeerPigeon project site, which contains a link to an archived copy of its own blog. It’s been good to capture a copy of this particular blog (very coincidentally, it also runs on WordPress), which only existed for nine months in 2007 and is starting to wind down. The resource is still live at time of writing, but for how much longer?

It’s also worth noting that within the UKWAC Consortium, major partners The British Library are actively building a collection of blogs (and similar sites using social software), which they call the ‘Digital Lives’ collection.

Some things to resolve with blog preservation, as with any other digital resource you want to preserve, are:

• selecting it for preservation in the first place
• resolving any copyright issues that might prevent its archiving and/or repurposing
• agreeing on a suitable frequency for harvesting of the blog
• ensuring the organisation has sufficient resources to maintain the preserved copies

The web harvesting software UKWAC have used to this point (PANDAS, which sits on top of HTTrack) has not met with any problems harvesting and copying the output from blogs. No problems are anticipated with the new Web Curator Tool, either.

Why haven’t UKWAC got a copy of this site? True, complex sites like this one tend to remain out of the reach of harvesting tools like PANDAS, which is based on HTTrack, and can’t get good results for sites which rely on complex server-side architecture. The site however is still unarchived as far as we know. ULCC’s Joanne Anthony (who had worked as the archivist for the Bernie Grant Trust) was keen to learn if there was any way of submitting the site for consideration to one of the UKWAC partners. There is indeed an online submissions form available, but this merely delivers a message to the UKWAC webmaster, who then forwards the request to the most appropriate partner. It would help considerably if the individual collection policies of each partner were made more manifest and published on the public site. But the visitor to www.webarchive.org.uk will find only a sketchy description of these policies, for example “The British Library will focus on sites of cultural, historical and political importance.”

Among UKWAC partners, the BL and the TNA are known to be directing their energies on certain specialised collection strands. These are given more descriptive paras at http://info.webarchive.org.uk/col.html, yet the underlying pattern or theme of these collections is not apparently obvious. At least three of them – the Tsunami, General Election and London Terrorist attack strands – appear to be based primarily on the fact that the sites are ephemeral and most in danger of loss (regardless of their informational or evidentiary value as records).

It is not clear how a concerned individual, or a member of the DP Community, might be empowered to somehow influence UKWAC’s collection policies for the better. In the case of the Bernie Grant website, Joanne’s interest was to see minority ethnic groups better represented in UK archival collections; but another approach would be to see it within in the larger group of ‘websites funded by Heritage Lottery Funding’. It seems likely there are many such project sites, all with short-term funding and therefore potentially at risk of being removed from cyberspace at any time, yet containing unique digital materials of huge potential cultural value. As a discrete collection of websites, it has parallels with JISC’s collection focus, ie JISC-funded projects which are occupying web space on a similar short-term lease. How can we persuade the relevant funding bodies to ensure their web outputs are archived, as JISC already does?

There are at least two sides to this mammoth task. The first (which I assume is probably relatively easy) involves moving the archive of gathered websites from its current server infrastructure to its new one. The previous hosts, Magus, have decided they can’t see a future in archiving websites. The new host, very coincidentally, is ULCC; our infrastructure services recently won the contract to provide a home for the large quantities of stored websites.

The second migration aspect, which involves complexities I’m glad I don’t have to deal with, involves moving the publisher and website profiles across from the PANDAS database to the Web Curator Tool (WCT) database. WCT, as fate would have it, is being jointly developed by the BL and the National Library of Zealand, and it will become our weapon of choice for all future web-harvesting activities. It’s certainly a more sophisticated piece of software than the clunky, web-object driven PANDAS, and appears able to handle the concept of one publisher owning more than one title (something which always baffled PANDAS).

The planned migration moves have been causing consternation to many of the UKWAC partners, particularly those who have been storing unprocessed gathers in the Magus ‘Temporary Drive’ for a long time. We at ULCC have been assisting with managing that process for months. Kevin Ashley devised a simple script that could query this drive, and report back on the occupancy broken down by website number, with figures on file sizes and dates. Ed Pinsent, by querying PANDAS, was able to match website numbers to their owners, thus providing a handy set of reports on information that was otherwise unavailable. (PANDAS wasn’t able to see these unprocessed gathers, for some reason; Magus wouldn’t run a script to report on them because they’d never been asked to, and it would probably have incurred additional charges anyway.)

The JISC occupancy of the temp drive has been negligible however. This is mainly because we have been so efficient in processing our completed gathers, and using the ftp connection has allowed us to look more closely at failed gathers. Additionally, JISC’s requirements are such that (unlike other partner members) we have rarely had to gather entire websites, instead concentrating on a few pages that constitute a JISC Project.