How to archive a website built with a wiki? It’s worth looking into this as increasingly JISC projects are using wikis to manage and report on their projects; of the available brands, MediaWiki is a popular one.

The challenge for me is how to bring in a good copy of a wiki site without causing Web Curator Tool to gather too many pages from it. We don’t want that, because (a) the finished result occupies unnecessary space in the archive and (b) because it takes so long to complete that it can hold up the gather queue in the shared web-archiving service, delaying the work of other UKWAC partners.

I am not technical enough to tell you in great detail what’s causing this, although I sense that it’s something to do with the Heritrix crawler requesting too many pages from the wiki. When you consider that a wiki is database-driven it should not surprise us that it’s creating a lot of its pages on the fly. Secondly, since a wiki is editable by lots of contributors (that’s its core function after all), it presumably means we have numerous past versions of pages also stored somewhere in the wiki labyrinth, and it’s possible that the implacable Heritrix will not cease until it’s faithfully requested and copied every single one of them.

Let’s look at the Repositories Research Team wiki (DigiRep) owned by UKOLN, which I tried to gather five times in 2008. WCT conveniently keeps a history of these attempts, information about which I can still access even if the actual gathered pages have been discarded or archived. The size problems were chronic. Of five 2008 gathers, one was aborted after it had reached a massive 16.87 GB; a second one was rejected at 14.69 GB. I have archived one impression at 5.31 GB, another at 736.26 MB and another at 157.36 MB. Quite large variations there, which was worrying enough in itself.

At first, my workaround was to adjust the Profile Setting in the title to override the maximum number of documents Heritrix can gather. Setting ‘Maximum Documents’ at 10000 worked, but it was not ideal; I suppose all this means is that Heritrix stops when it collects 10,000 pages, whether we have everything we want or not. (I found that the copies in the archive seemed to render OK however).

To get a closer look at what’s going on, I started to browse the Log Files created by WCT (complete records of every single client-server request), which show patterns which I can vaguely understand; when these Log Files are packed with near-identical strings of code I sense that something’s up. For example, a string containing index.php?title=Repositories_Research&action=edit tells us that the wiki is requesting a specific named page, and allowing an edit action on that page. If you multiply that by the number of pages in the wiki, you can see how the problem builds up. (PHP is the script used for MediaWiki’s web scripting engine).

I follow this up by browsing the actual gathered pages in Web Curator Tool using the Tree View. From here I can click on the ‘View’ button to examine a page which I think to be suspect, and compare it with other suspect pages. Lastly, I go back to the live DigiRep site to confirm in my mind what’s happening when certain links are followed.

All the above gave me just about enough information to experiment with exclusion filters. After a certain amount of trial and error, and working with other Media Wiki sites, I arrived at the following exclusion codes which I can add to the Profile Setting:

.*&oldid.*

.*&redirect=no.*

These have the effect of telling WCT to exclude certain pages and actions from Heritrix’s harvesting action. The expectation was that I would lose the discussion / edit / history functions of the wiki in the archive copy.

The title with the above exclusion profile gathered just 63.41 MB and it completed in under ten minutes. I would say that’s an improvement on 16.87 GB. Log Files and the Tree View confirmed the success of this new “slimline” gather. As well as losing the discussion / edit / history functions, we also have eliminated the Toolbox functions, the ‘printable’ views, and the login pages.

This is no great loss at all for our purposes, as scholars who browse the archived copy of DigiRep are not expecting to be able to edit pages, nor join in the discussions, nor browse the history of stored versions of pages. Indeed in a lot of cases, they would require a login to do so. The users simply want to see the results of the DigiRep team’s work.

On December 8th I was invited by Dr Saad Eskander, the Director of the National Library and Archives of Iraq, as his guest to an awards ceremony he was having at the British Library.

The story behind this is that in 2006 I met him in Abu Dhabi speaking about his work and I subsequently asked him to write a blog for our community of archivists and librarians about his day to day life. This blog was picked up by almost all main stream press around the world. Here is an article about him in the International Herald Tribune. In an interview with Fran Monks (in her website on How to make a difference ) Saad spoke about how through his own free will he decided to return to his home city of Baghdad in 2003 after 23 years of absence, of which 13 had been spent as an academic in the UK. After Sadaam had been removed, Saad and a group of Iraqi artists, writers and academics from the UK returned to their country to see what they could do. All of the group except for Saad returned to London almost immediately because they were so shocked by the security situation. Saad alone was prepared to risk his life in order to assist with the rebuilding of the nation that he loves. Saad believes that the future of Iraq and Baghdad must be non-sectarian and democratic and have equal rights for all citizens, including women. This he practises in the Library which he has restored. Saad described his shock on first seeing the library: “95% of the contents had been either destroyed or looted. Everything had been burnt and even the marble had melted. Everything was covered in soot and the stench was almost unbearable.”

At the dinner after the ceremony were many of the great and the good from UK with specialities in the Middle East. I was an incongruous figure among them…as this obviously does not include me…I also sat beside Ann Clwyd who is a Welsh MP and Gordon Browns’ special envoy on Human Rights in Iraq. A lot of interest in the digital preservation training we do here and also the potential of VLEs being developed to facilitate learning in this area. It is seen as priority for the INLA to learn about how to manage their digital surrogates as their collections of archives is largely made up of these surrogates from the British Library as well as other places.

Chris Rusbridge has already done an excellent job of some near-real-time reporting on the sessions, and I’m not going to try to replicate that level of detail in this post. As a first-time attendee at iPres, I was impressed by the professional mix attending, which took in hard-core computer science, digital preservation and curation folk, repository managers and those from the traditional custodial professions. In that respect it was very reminiscent of the early DLM-Forums, which were eye-opening for me when I attended the first one in 1996. But it was also interesting to observe that, just as DLM was dominated by archivists and records managers, iPres is a very library-oriented event. For example, those who expressed a desire for a Europe-wide event bringing together all those with an interest in digital preservation didn’t seem to be aware that the DLM-Forums existed.

One positive observation (of many) is that there is more reassuring news on the oft-vexed issue of IPR barriers to digital preservation. At the close of day 1, we heard a summary of the findings of the international survey on the impact of copyright law on digital preservation. That indicated that the UK had one of the strictest set of constraints of all the countries looked at – in terms of who is permitted to carry out certain acts in the name of preservation and what those acts are. Other countries have more relaxed exemptions and that doesn’t appear to be causing the major rightsholdfers any significant financial loss. That should give us hope for some change in the law in the UK at least. And Horst Foster, making the keynote speech opening day 2, appeared to echo this at the European level, implying that the case for change had been made and accepted, although he was notably cautious about making any promises as to when this change might come about.

That’s an improvement on the situation in Europe a few years ago, though, when I was at one of a number of expert panels helping the European Commission to frame the forthcoming research questions and challenges in the digital preservation arena. At that time we were all warned off mentioning the (C) word at all – it seemed to have a somewhat toxic flavour. It’s really heartening to see that things have changed.

One should add a note of caution, however. After Adrienne Muir had commented favourably on how Australian law allows institutions such as the national library and archives to bypass DRM systems in order to preserve material, Colin Webb injected a note of caution. It is apparently still illegal to manufacture or to import a device to Australia which allows a DRM system to be bypassed. But if the national library happens to find such a device on its premises, it can use it without fear of breaking the law. Still some way to go, then, before the law is ‘joined up and working’ – the strapline of iPres2008.

Not the book by George Perec but actually a costing model for digital preservation developed by the British Library. LIFE, (Life Cycle Information for E-Literature) is a collaboration between University College London (UCL) and the British Library. The Project has developed a methodology to model the digital lifecycle and calculate the costs of preserving digital information for the next 5, 10 or 100 years. For the first time, organisations can apply this process and plan effectively for the preservation of their digital collections.

Currently the LIFE Project is in its second phase (“LIFE²”), an 18 month project running from March 2007 to August 2008. LIFE tries to develop a costing model which can identify costs at each stage, 1 year, 10 years and 100 years. It wants to show the long term consequences of accessioning collections and can be used for curatorial decisions.

The event was kicked off by Helen Shenton, who asked us not to crack any more Second Life jokes, and introduced us all to LIFE and its evolution. To read more, have a look here.

Paul Courant of the University of Michigan gave a keynote speech and noted how history ‘gets longer by the minute’. He noted that market forces may drive preservation for a while but that material which is seen as unmarketable will not be taken care of, in crude economic terms. It is vitally important that people who control resources understand how important digital preservation is.

Throughout the day we heard case studies from national institutions in Denmark and the British Library, as well as the newspaper project within the British Library. All of them had used LIFE and gave feedback regarding their experience with it. Some thought it would be useful to use OAIS terms to ease understanding and communication and would like to see a breakdown in some of the generic functional entities. One case study involving film noted that it would be more economical to preserve film in digital format as opposed to 105 mm films. Professor Bo-Christer Björk who was asked to check the model related that he didn’t identify any significant flaws with the methodology of LIFE. He hoped that convergence of a few file formats would, in the future, reduce costs. He could see how very rare formats could create high costs in the long run.

Paul Ayris of UCL questioned whether libraries should, instead of pulling readers into their institutions, be pushing material out to the public. He referred to the Blue Ribbon task force on substantial digital preservation and access (BRTF -SDPA) which he is involved in. Launched by the National Science Foundation (NSF) and the Andrew W. Mellon Foundation, the Task Force’s two-year mission is to develop a viable economic sustainability strategy to ensure that today’s data will be available for further use, analysis and study. Members will convene a broad set of international experts from the academic, public and private sectors who will participate in quarterly discussion panels. The group will then publish two substantial reports with their findings, including a final report in late 2009 that will include a set of actionable recommendations for digital preservation.

LIFE seems to be workable; however, costing takes time, evidence for costing takes time to gather, and it appears to work best within national institutions and Institutional Repositories where preservation priorities have been clearly stated; and there must be a stable budget.

Web Curator Tool is a web-based workflow database, one which manages the assignment of permission records, builds profiles for each ‘target’ website, and allows a certain amount of inter-facing with Heritrix, the actual engine that gathers the materials. The open-source Heritrix project is being developed by the Internet Archive, whose access software (effectively the ‘Wayback Machine’) may also be deployed in the new public-facing website when it is launched in May 2008.

Although the idiosyncrasies of WCT caused me some anguish at first, largely through being removed from my ‘comfort zone’ of managing regular harvests, I suddenly turned the corner about two weeks ago. The diagnostics are starting to make sense. Through judicious ticking of boxes and refreshing of pages, I can now interrogate the database to the finest detail. I learned how to edit and save a target so as to ‘force’ a gather, thus helping to clear the backlog of scheduled gathers which had been accumulating, unbeknownst to us, since December. Most importantly, with the help of UKWAC colleagues, we’re slowly finding ways of modifying the profile so as to gather less external material (or reduce collateral harvesting, to put it another way); or extend its reach to capture stylesheets and other content which is outside the root URL.

True, a lot of this has been trial and error, involving experimental gathers before a setting was found that would ‘take’. But WCT, unlike our previous set-up, allows the possibility of gathering a site more than once in a day. And it’s much faster. It can bring in results on some of the smaller sites in less than two minutes.

Now, 200 new instances of JISC project sites have been successfully gathered during March and April alone. A further 50 instances have been brought in from the Jan-Feb backlog. The daunting backlog of queued instances has been reduced to zero. Best of all, over 30 new JISC project websites (i.e. those which started around or after December 07) have been brought into the new system. I’ll be back in my comfort zone in no time…

I don’t think I could begin to do justice in a few words to the wide-ranging debate at the JISC/BL/DPC Workshop on Significant Properties at the British Library on Monday: I’d rather leave it to others to analyse the significant outcomes in more detail, or to further discussions like the one started by Chris Rusbridge (our cucumber-cool chairman on the day) on the DCC Blog. Suffice it to say there was a sack of food for thought in all the presentations, and lots of opportunities to wonder “now why didn’t I think of that?”

I think the most enjoyable presentation was that of Cal Lee from UNC Chapel Hill, who managed to lift us with a few laughs about his work – with Microsoft Word file format specifications – just as we were beginning to flag at the end of a long day of considering many things from many angles. The Moving Images study also looks like it may have useful information for our deliberations on standards and formats for PRIMO. If I wasn’t so taken with the presentation on E-learning objects, that was chiefly because I gave it – a mind-bending task to give our account of not only Learning Objects but also Significant Properties in 15 mins. But the slides look nice on the DPC website. (All credit to Ed for the excellent diagrams.)

Neil Grindley and the rest of the JISC/DPC/BL organisers did well to run it so smoothly and facilitate the discussion, and it was a rare opportunity to catch Andrew Wilson on a flying visit to this hemisphere and hear his first hand account of NAA’s approaches and models. I can’t remember if I’ve been to the British Library Conference Centre before or not, but it is certainly an impressive space, and with super-slick AV facilities.

Why haven’t UKWAC got a copy of this site? True, complex sites like this one tend to remain out of the reach of harvesting tools like PANDAS, which is based on HTTrack, and can’t get good results for sites which rely on complex server-side architecture. The site however is still unarchived as far as we know. ULCC’s Joanne Anthony (who had worked as the archivist for the Bernie Grant Trust) was keen to learn if there was any way of submitting the site for consideration to one of the UKWAC partners. There is indeed an online submissions form available, but this merely delivers a message to the UKWAC webmaster, who then forwards the request to the most appropriate partner. It would help considerably if the individual collection policies of each partner were made more manifest and published on the public site. But the visitor to www.webarchive.org.uk will find only a sketchy description of these policies, for example “The British Library will focus on sites of cultural, historical and political importance.”

Among UKWAC partners, the BL and the TNA are known to be directing their energies on certain specialised collection strands. These are given more descriptive paras at http://info.webarchive.org.uk/col.html, yet the underlying pattern or theme of these collections is not apparently obvious. At least three of them – the Tsunami, General Election and London Terrorist attack strands – appear to be based primarily on the fact that the sites are ephemeral and most in danger of loss (regardless of their informational or evidentiary value as records).

It is not clear how a concerned individual, or a member of the DP Community, might be empowered to somehow influence UKWAC’s collection policies for the better. In the case of the Bernie Grant website, Joanne’s interest was to see minority ethnic groups better represented in UK archival collections; but another approach would be to see it within in the larger group of ‘websites funded by Heritage Lottery Funding’. It seems likely there are many such project sites, all with short-term funding and therefore potentially at risk of being removed from cyberspace at any time, yet containing unique digital materials of huge potential cultural value. As a discrete collection of websites, it has parallels with JISC’s collection focus, ie JISC-funded projects which are occupying web space on a similar short-term lease. How can we persuade the relevant funding bodies to ensure their web outputs are archived, as JISC already does?

There are at least two sides to this mammoth task. The first (which I assume is probably relatively easy) involves moving the archive of gathered websites from its current server infrastructure to its new one. The previous hosts, Magus, have decided they can’t see a future in archiving websites. The new host, very coincidentally, is ULCC; our infrastructure services recently won the contract to provide a home for the large quantities of stored websites.

The second migration aspect, which involves complexities I’m glad I don’t have to deal with, involves moving the publisher and website profiles across from the PANDAS database to the Web Curator Tool (WCT) database. WCT, as fate would have it, is being jointly developed by the BL and the National Library of Zealand, and it will become our weapon of choice for all future web-harvesting activities. It’s certainly a more sophisticated piece of software than the clunky, web-object driven PANDAS, and appears able to handle the concept of one publisher owning more than one title (something which always baffled PANDAS).

The planned migration moves have been causing consternation to many of the UKWAC partners, particularly those who have been storing unprocessed gathers in the Magus ‘Temporary Drive’ for a long time. We at ULCC have been assisting with managing that process for months. Kevin Ashley devised a simple script that could query this drive, and report back on the occupancy broken down by website number, with figures on file sizes and dates. Ed Pinsent, by querying PANDAS, was able to match website numbers to their owners, thus providing a handy set of reports on information that was otherwise unavailable. (PANDAS wasn’t able to see these unprocessed gathers, for some reason; Magus wouldn’t run a script to report on them because they’d never been asked to, and it would probably have incurred additional charges anyway.)

The JISC occupancy of the temp drive has been negligible however. This is mainly because we have been so efficient in processing our completed gathers, and using the ftp connection has allowed us to look more closely at failed gathers. Additionally, JISC’s requirements are such that (unlike other partner members) we have rarely had to gather entire websites, instead concentrating on a few pages that constitute a JISC Project.