Rory and I met with Richard Gartner and Gareth Knight at CeRch today, to catch up with their investigations into using GATE and OpenCalais to process the EAD outputs from AIM25.

Results look very encouraging. OpenCalais, in particular, generates a post-processing set of identified entities (personal names, place names, corporate names) which Richard G has then created regular expressions to locate these in the body of the EAD and wrap in appropriate EAD tags (<persname> etc).

This suggests that the way forward for enhancing the existing data entry processes for AIM25 will involve dispatching the EAD-compliant data entered by collections manager to OpenCalais, and returning the data, with enhanced markup, for checking by the submitter. This hook should be easy enough to insert for manual, form-based entry; for batch entry processes we will need to assess whether any significant delays are introduced.

We’ve also started to consider ideas for a URI scheme for the entities identified. Our current working hypothesis is that this will involve defining a “data” namespace for AIM25, binding to http://data.aim25.ac.uk/. Within that we can develop a structure along the lines /person, /place, /corporate_body, and append our unique IDs for each entity. Further research is necessary, particularly into the recommendations of the Cabinet Office recommendations for Designing URI Sets for the UK Public Sector.

These URIs can then be used in identifier attributes for our EAD elements (<persname>, etc.), and thence easily transformed into an RDFa format for the Web-based HTML rendering of the AIM25 catalogues.

Next steps include further investigating how to implement and assert relationships between our entities and other open datasets (e.g. our_entity is_the_same_as your_entity). And how to make the authority data, duly marked-up, available as open metadata.

Rory and I can now start to consider suitable approaches to embedding this in our development copy of the existing AIM25 system, and we’ll continue to liaise closely with CeRch for advice on  the relative merits of Gate and OpenCalais processing, and guidance on URI implementation.

It was great to take part in last week’s Repositories Support Project event at Sheffield Cathedral. The theme of the day, organised by Jackie Wickham and the RSP team, was “Doing It Differently” and it covered a wide range of repository-related themes. I took along an updated and expanded version of the presentation I made to SHERPA-LEAP repository managers. I covered the same topics, but in preparing the presentation, I was amazed how many more things there were to talk about a year on.

Stephanie Taylor gave an excellent overview of the repository scene, and I hope I followed it up with useful ideas about making repositories more user-friendly or just generally useful to users. Other talks went off into less well trodden areas, though no less interesting: Pat Lockley impressed again with his enthusiastic description of Xpert; Joss Winn described his further adventures in WordPress land; and Stephanie Meece described the challenges of non-textual repositories at UAL. My ears pricked up when Jason Hoyt of Mendeley mentioned that an imminent upgrade to Mendeley will be able to identify OA sources for papers, which might signal it’s time for me to finally catch up with Mendeley (dissertation starts next year!). I didn’t catch the final speakers as I had to catch my train, but I commend to you Vicki McGarvey’s post on the SHARE project blog at Nottingham Trent University.

I tried to keep things simple by steering clear of all the complicated issues in repository management – OA, OAI-PMH, copyright, advocacy, REF, RIM, etc – and just focus on simple UI enhancements that might improve a user’s experience of the repository, and effective use of features like RSS feeds and statistics, with examples from all over the world of institutional and specialist repositories. Which features a repository manager might choose, if any, is up to them and their own circumstances, but my aim was to ensure they are at least aware of what’s possible – as evidenced by what’s been done in many repositories around the country.

Although I focused on EPrints installations, I think nearly everything I demonstrated ought to be feasible in other platforms. Overloading an abstract page with features like “Share this on Facebook/Twitter”, QR Codes, or metadata export in RSS/JSON/CSV and more, should be a very easy way to enhance the user experience of repositories. As I suggested, adding buttons to support “the latest thing” users may be finding useful, is generally not difficult. A “Send This Paper To My Kindle” button, for example, seems so trivial I might even try it myself.

I had a long list of ideas/examples to show: for anyone who didn’t have time to copy down the small print, they were:

• SNEEP
• Lincoln EPrints
• Language Box
• MePrints
• Hum Box
• ULCC Publications Archive
• UCL EPrints
• IRStats
• Repository Stats using Google Analytics (presentation by Graham Triggs at OR10)
• E-Space at Manchester Metropolitan University
• Framework for Linking Inline Semantic Metadata
• Oxford University Research Archive
• MERLIN

Inspired by Rob Sanderson’s lightening talk at dev8D on Memento I decided to go for the bounty offered for writing a memento client. My tack was to enable a mediawiki instance to handle the Accept-Date protocol using an existing plugin. Then to write a little PICT tool that supplied a user interface by which users could specify a date and browse their PICT enabled mediawiki “from the past”… spooky!

Thanks to getting nowhere near the deadline (those involved with this project will be grinning at that), I got nowhere near the prize, but I did finish a prototype plugin and threw it up on a mediawiki instance for another project: CLASM-demo. CLASM is the name of project not a piece of conjurers’ onomatopoeia, so click on that link to see the PICT-memento client in action. A wiki with more pages would have been a better example, but at least it does have a lot of revisions.

Beacon cited through fog

Who would have thought that reference management could be so interesting? We spent a  very informative and enjoyable Thursday in snowy Milton Keynes, at the Innovations in Reference Management (#IRM10) event (part of the OU/JISC TELSTAR project). All thoroughly blogged by Owen Stephens, and tweeted by many.

Owen Stephens and Jason Platts of OU described the outputs of the TELSTAR project, which integrates the OU’s Moodle VLE with Refworks. This means that students using the VLE can move seamlessly between their reading lists and Refworks, locating resources, maintaining consistency of style and generating bibliographies easily.

Paul Stainthorp of Lincoln University described some exciting, bleeding-edge uses of Yahoo Pipes to mashup data from Refworks, OPAC, and Amazon. Arguably even more bleeding-edge was the presentation by Euan Adie from Nature Publishing, who showed us Help Me Igor, a reference manager plugin for Google Wave. Speakers from CiteULike and Mendeley also gave us fascinating insights into their respective social-tinged bibliographic management offerings.

Perhaps unsurprisingly, Kevin and I brought to the table the theme of web preservation. With reference to our work with JISC-PoWR, UKWAC and ArchivePress, we reminded anyone who hasn’t heard our spiel already that there are many important, valuable and eminently citable web resources, notably blogs by academic researchers, that are at risk of disappearing – making references to them virtually useless.

Authors may not be responsible for ensuring their readers can access the resources they reference, but we think they should at least give them a fighting chance of doing so! We  therefore proposed that students and researchers should be encouraged to locate and cite copies of web resources in stable web archives (such as the UK Web Archive) rather than “in the wild”.

We also discussed the idea that persistent collections of web resources could be created at the institutional level, whether that were an open archive of blog posts by a university’s researchers, or a closed repository where researchers can store copies of the web resources they cite.

One of the strong themes that emerged in discussion was the need for information literacy/digital skills training at all levels to address current tools and trends in reference management; and to re-assert the purpose, value and nature of citation in online digital environments

An interesting suggestion also made was that reference management tools are becoming a natural part of the environment, just as email has: is provision of specialised applications by universities an “aberration”?

I’m inclined to think not, after all it was clear from the workshop that there’s still a need to support ongoing study and research effectively, and scope to develop and validate new approaches.  Microsoft Word may now include reference management features, but that doesn’t obviate the need to educate people in how to use them effectively, and why.

We’re very grateful to Owen for including us in his programme: this is a fascinating area, where e-learning, libraries, preservation and publishing collide, and I’m sure we haven’t heard the last of it.

FOTE 2009 in Second Life

We used the opportunity to include a short presentation about our CLASM project, and I shared the platform for one session with James Ballard, our resident Learning Technologist and ace Moodle hacker. The full video and slides (with audio) of our talk are available from the FOTE website; the slides are also on Slideshare.

I was particularly pleased to make contact with Jane Secker of LSE, who knows more than most about CLA, and I am looking forward to discussing some issues with her, as we try to refine the work done on the CLASM plugins and produce a finished package. Jane also published an excellent account of the day’s events on her blog

The audience was a bit different from the JISC Information Environment crowd I’ve made presentations to before, so my talk was a very high-level overview of repository work in the sector, with a few ideas about where trends and technology seem to be leading us. One particular advantage I see is that interoperability between web applications should enable us to focus on using the “right” tools – portfolio, VLE, blog, repository, etc, maybe even VW – at each stage of the institutional/educational workflow, rather than using over-ambitious and over-complicated systems that try to do everything. “Small pieces loosely joined,” and all that.

Unfortunately, while the slides on the FOTE website include audio, the video there doesn’t include the slides, which robs the talk of some context. I have, by some dark means, managed to create a new version which combines the video and slides and upload it to YouTube. (To keep it short and relevant to DA Blog readers, I’ve only included my part of the presentation.)

Many interesting discussions about repository approaches for the arts followed, both in the workshops, plenary debates, and over the rather smashing buffet lunch. One interesting direction the discussions took was in suggesting that while endeavours like KULTUR and PRIMO provide examples of ways to develop repositories for visual and performing arts, they don’t offer any kind of ready-made application for institutions wanting to create their own repositories with a minimum of fuss and cost. Is it possible that the benefits of such projects (particularly JISC-funded projects) would be greater if the outputs generated a reusable product rather than just a script or a recipe? Bill agreed to discuss this idea further within RSP, and I look forward to following it up soon.

The JISC funded SNEEP project (Social Networking Extensions for EPrints) – part of the original JISC rapid innovation programme – aimed to provide a set of social networking tools for EPrints repositories. It ran for 6 months and ended in May 2008. Since the rather low key publication of the resultant EPrints plugin interest and uptake has been slowly but surely gathering momentum.

Today I am pleased to announce a couple of significant SNEEP related developments. Firstly , thanks to my colleague Ben Wheeler here at ULCC, SNEEP 0.3.2 released this week offers an automagic installer. This does away with the (slightly tortuous) manual install procedure that we suspect discouraged all but the hardier EPrints hac… I mean administrators.

You can download SNEEP 0.3.2 and/or read Ben’s post to the EP-tech mailling list. The download page is also a good place to see SNEEP in action.

PICT

I am also pleased to announce a new project (funded as part of the 2009 JISC rapid innovation programme) that aims to build on the SNEEP work to provide SNEEP-ish services to a broader range of web resources. The goal of the PICT project (Platform Independent Community Toolbox) is a lightweight javascript tool that can be deployed across an number of web resources (not just a repository) to encompass the web-based real estate of a given research community and provide that community with collaborative tools available at the on-line research coalface.

Effectively PICT will allow resource owners to offer

• tags
• comments
• notes
• other goodies

from their web page. The data gathered by these tools will be managed by a PICT server (probably run by a community-minded resource owner) and be available for cross referencing with other resources in a PICT community.

If all that is a bit difficult to picture, rest assured that demos will appear throughout the course of the project that should help to clear the murk.

I began the day by chairing the final meeting of RPAG(the repositories and preservation advisory group.) We had a short meeting mainly to follow up on discussions we had been having on how the group had operated and how JISC might make use of advisory bodies in future. Those who expressed an opinion all felt it had been useful to them, but we all had concerns about how our time, and the JISC Executive’s time, might have been used more effectively. Future advisory groups may try to split responsibility for some areas into smaller working groups. All were agreed that the face-to-face meetings were invaluable, but we weren’t all agreed on which online technology would be best to use in between times. Enthusiasts for tools like ideascale were matched by those who found them unusable.

The meeting proper opened with some background and perspective from Rachel Bruce and Neil Grindley of JISC and myself. I tried – partly seriously, but without much expectation of accuracy – to give a one-line summary of what each project set out to do. But there were two things I meant to say which I failed to do. One was to look forward to the theme of day 2 (Value) and stress that repositories are not ends in themselves, but need to be thought of in terms of value, impact and benefits to someone. The second point I omitted was to remind us that , for innovation projects, failure in one sense can still mean success, as long as we understand the nature of the failure and are able to use it to improve and adapt future work. Not achieving what you set out to do is disappointing. Analysing the reasons for that and making sure others are aware of them can be of great value.

But it was the rest of the event that provided greatest interest. The discussion sessions on text mining, research data, teaching and learning repositories and more; presentations from projects from stakeholder, developer and other perspectives; posters and demos from many of the projects; and the fever of activities in the ideas room, which deployed technology ranging from post-it notes upwards to catalyse, capture and refine ideas from the attendees. These activities gave the event much more of a participatory feel – everyone became a contributor rather than being a consumer.

I learned a few things over the course of a day or two, most of them unexpected. David Flanders (via Chris Rusbridge) passed on the neat idea of feeding funding proposals through Wordle before marking them. That’s what ULCC’s AIDA project looked like. Perhaps you ought to try the same with your proposals prior to submitting them?

I learned that talking unprepared and unscripted to a video camera doesn’t produce great results unless you’ve had practice or training – neither of which I’ve had. I knew that in an abstract sense and now have the unfortunate experience to back it up. But Andy McGregor and Dave Flanders did capture some other people talking far more sense than I did and far more clearly, and you can see the results on the dev8D youtube channel.

Andrew Prescott’s overview of the Welsh Repository Network provided us with the surprising finding that smaller institutions are more, not less, likely to want to run their own repository rather than contract it out to someone else.

And via a serendipitous typo, we all contemplated whether working in a repositoire might not be an altogether more rarified and sophisticated career option than working with a repository.

Photo by Chad Miller

On the 5 March I attended the London launch of the rebranding of ‘TASI’ to ‘JISC Digital Media’. Tables were decked with everything from canapés & wine, to a variety of AV and photographic media on display (on separate tables of course!). Although the former ‘TASI’ was always a JISC-funded venture, it’s now more prominently self-evident in its newly rebranded name.

As of August this year, JISC Digital Media will become part of a consortium of JISC advisory services that aim to provide joined-up solutions for clients. Other aligned services include JISC InfoNet, JISC TechDis, JISC Legal Information, Procureweb and JISC Netskills.

JISC Digital Media’s official brief is “to ensure that digital media resources being created, used and managed within the further and higher education community meet the teaching, learning and research needs of individuals and institutions within the UK.” The recently expanded service now also provides expertise in moving images and sound. (In fact, as I blog, a couple of members of our very own Digitisation team are attending their new training course on Audio Production).

Speakers at the launch touched upon some specific aspirations for the Service, and a few points of interest stood out:

• JISC Digital Media are keen for the HE and FE sector to use the JISC Digital Media blog and share expertise across the sector;
• Would like to adopt more web 2.0 technologies, for example, skype-based e-learning that could support some aspects of practical training;
• Their emphasis will be on helping the HE/FE sector to use images for teaching;
• There is a recognised need that more must be done to help the FE sector.

Of course, with any newly rebranded organisation, comes a new-look website http://www.jiscdigitalmedia.ac.uk/ Nice bold colours and user friendly too! …So farewell dear ‘TASI’ [now a dirty word that incurs a fine if spoken out-loud by its own staff], and ‘hello’ to the new and improved Advisory Service: ‘JISC Digital Media’.

What’s prompted me to write about it now is the news that the outputs from that workshop are now visible, and the ongoing process of revising and amending them is taking place in a far more public forum on pbwiki. repinf.pbwiki.com is somewhere you should visit if you are, in the words of its homepage:

…. interested in:

1. developing coordinated action plans for specific areas of repository development

2. pursuing those plans

3. coordinating that activity internationally

An international workshop in March 2009 kicked off this process, which is now open to anyone willing to contribute.

If you have an opinion on things like interoperable identifiers, citation services, streamlining deposit workflows or (most contentiously) international repository organisations, you need to take a look at these materials. The workshop that produced them was a curious and mixed event, but it certainly had some positive features. It brought together interested experts from around the world to consider things that need doing with repositories that can only happen through joined-up international action. We did our best to focus on things that could be done in a reasonable timescale and that would produce clear benefits. At the end a group of funders – many of whom clearly weren’t quite sure what was expected of them – spent an hour or two considering the ideas that came out of the 4 workshop groups and voiced their own opinions about them. Some of the ideas drew widespread support from public and commercial organisations, whilst others were not yet clearly developed enough, or were still too parochial. Generally, there was a clear willingness to take action, but some of the plans needed more work before funders could act. The wiki is the way that that work will be done (with the contentious exception noted above.)

The idea of getting joined-up thinking between doers, thinkers and funders has succeeded. Anyone can read the material on the repinf wiki site, and anyone can edit it once they ask for a userid. Do it. Do it now.

Last Thursday’s Libraries Of The Future (LOTF) event at Oxford University has been well covered elsewhere, so I’ll just note a few key themes as I inferred them. LOTF is a JISC-sponsored campaign begun last year, and continued by means of online social networking (chez Ning) and a JISCInvolve blog, as well as F2F events like this.

If you’d nodded off for even only a couple of years, LOTF might well seem like the product of some post-information-meltdown future, with live Twitter streams projected behind the speakers, and the occasional glimpse of the parallel Second Life auditorium. All the same, one thing we didn’t see, in Real Life or Second Life, were shelves of books, or any books at all for that matter, apart from the JISC Collections catalogue on the registration desk. (By the way, if anyone knows of any shelf-for-shelf library reconstructions iSL, please send me a link.)

On change: Sarah Thomas astutely observed that change only looks fast if you are standing still: the only way to manage it is to be part of it. She proffered sound advice that the past and future – old ways and new ways – should not be set in opposition: we’ll achieve best results by understanding and integrating the best of traditional and innovative approaches to information management. Chris Batt compared the LOTF meeting with a 15th Century gathering of the Society of Scribes, discussing the emergence of “something big”. On that occasion, the “smart” ones, as Chris observed, bit the bullet and set about designing typefaces or investing in their own presses, leaving the rest to soldier on in an ever more marginalised business. This was a more original historical analogy than the usual hackneyed account of the Alexandrian library, though we didn’t escape the day without at least one mention of that. Peter Murray-Rust (“speaking in HTML, not powerpoint“) also had a compelling message: “If you are not aggessively trying to change the world, you will not end up with the Library of The Future.”

On the book: Sarah pointed out that, while the traditional book may still be alive and kicking, we cannot escape or deny the increasing reliance on digital resources, and a need for more and better digitisation projects and online finding aids. Chris picked up this theme, citing Susan Greenfield’s analysis of the information (r)evolution that is replacing “people of the book” with “screen people”: someone born at the start of the 21st Century may have little if any nostalgia for a paper book. (That said, I’m still inclined to think that the dedicated e-Book readers are a conceptual dead-end, except to the extent that they inform the evolution of laptops and other general purpose mobile devices.) From Santiago de la Mora we learned of the potential of Google Books to bring back to life “in-copyright-out-of-print” books.

On libraries and librarians: One particularly interesting observation was the extent to which digitisation breaks down the traditional boundaries between institutions: for example, by undertaking digitisation, a museum in effect is creating a ‘library’ of its digitised objects. Web technology makes it possible for almost any place to become a library, and it is erroneous to conceive of future libraries in terms of their bricks-and-mortar predecessors; libraries themselves must adapt their physical structures and the services they provide. In HE the library of the future will be at the heart of much greater integration across services, departments and institutions, and the trend towards OA research outputs and shared storage (e.g. UK Research Reserve ) is already clear.

On public knowledge: All the speakers expressed commitment to open public knowledge. Peter Murray-Rust cited endeavours like OpenStreetmap , NCBI PubChem, Sourceforge , inter alia, undermining the proprietary nature of data and information. However small, “each individual can do whatever they like on the web… don’t talk… do it!” Peter also aligned himself against Wikipedia’s detractors, and asked why universities aren’t actively helping Wikipedia. Robert Darnton drew audible gasps when he suggested that the best way to accelerate Open Access was for libraries to cancel their subscriptions to paid-for journals. Mining a similar vein, Chris Batt described the job of the “knowledge” (sic) sector as defining, mediating, managing and leading, in creating the public landscape of a Learning Society. The mission must be to remove all barriers to acccess: a universal right to knowledge and integration of knowledge into everyday life, empowering citizens to want to learn. That should involve the kind of creative approach we see in ventures like LibraryThing, rather than some of the more pedestrian public sector online efforts of the past.

I’d be inclined to echo the views on the CILIP blog, that the need for new information skills and literacy training – both of and by librarians – might have been addressed more directly. The nature of librarians of the future was raised by Chris Batt, and implicit in several comments through the course of the afternoon, but at the end of the day I felt we had dealt with many of the new information paradigms, but much less how “informaticians” are going to share that enormous dynamic richness with new and old generations.

And then it was off to the Bodleian Divinity School for a glass or three of Château JISC…

This will be a six-month project with a double-edged purpose: to develop a SWORD plugin for Moodle, so that it can interact, platform independently, with common repository applications like EPrints and DSpace; and to explore and demonstrate the use of that plugin for managing Copyright Licensed materials in Moodle courses.

The issue of managing CLA materials for VLE courses was drawn to our attention on several occasions by colleagues from other institutions, and the superior bibliographic features of e-repositories seem to offer a promising approach to managing these objects effectively for tutors, students and library staff, while making them available within a VLE, in accordance with CLA terms and conditions.

CLASM will be developed by the same team that developed SNEEP, one of the first JISC Rapid Innovations projects, and we will also be working closely with ULCC’s E-learning team, responsible for our Moodle and Mahara service. This should be a particularly rewarding and fruitful collaboration, since there is huge potential to improve the integration between these three critical educational applications – repositories, VLEs and e-Portfolios.

You can find out more about Phil and James’s adventures in e-Learning at our recently revamped sister blog: El Blog.

Post scriptum, 18th March.

I also learned that, among the other successful bids to the JISC’s Information Environment/e-Research call, were: MERLIN, a text-mining initiative for Institutional Repositories, led by UCL Library; and PhilPapers,  a project led by the Institute of Philosophy, to extend the impressive Philosophy portal currently hosted by ANU. ULCC will be contributing, as partners, to both of these exciting projects.

Rating a JISC bid

Marking

Markers – at least those outside the JISC Executive – get anywhere between 1 and 10 bids to look at, and something like 2 weeks in which to do the marking. As well as the bids, they’ll get some guidance for markers and the original text of the call. They’ll also see a log of all bids received, which as well as assigning each bid a unique ID, tells them how much each strand is over-subscribed (that is, how much is being asked for against how much is available.) And they’ll have access to a closed email list to clarify issues with other markers and the Executive. In my experience these lists are not much used these days, although they can prove useful to clarify ambiguities in the call or the marking criteria. Some are probably using Twitter for this now, which is unfortunate as it means that not all the markers will be aware of the discussions.

At an early stage, markers are meant to double-check that they don’t have any conflicts of interest in the bids assigned to them. A bid in which your own institution is a partner definitely constitutes such a conflict, and the Executive usually spot these in advance. But they arise for other reasons, and every marking episode I’ve been involved with has had at least one person realising that they have a conflict, often very late in the process.

Then we begin the process of reading, evaluating, and assigning marks. I suspect everyone has a different approach to this, but the outputs are the same. We score each criterion on a 5-point scale that’s really a 10-point scale (because it has half marks.) The criteria include such things as “Appropriateness to the call” and “value” and the lowest rating implies that the bid fails that criterion in a way which just can’t be fixed, whereas the highest implies that it exceeds expectations significantly in a number of ways. Markers also make comments on each criterion to explain the thinking behind their marking. These comments will form the bulk of the feedback that you will receive if you ask for it.

It’s worth noting that you can (and should) ask for this feedback even if your bid is successful. Sometimes your programme manager will offer it to you unasked. Few bids are perfect, and most bids contain something of value. The feedback can tell you what you need to improve but it will usually also tell you what you did well. Both are good to know!

As well as the criteria-based marking, we are also asked to rate the bid overall as A, B or C. As the picture above shows, this means that we strongly recommend funding, weakly recommend it, or do not recommend funding. These marks are the most significant ones once all the bids are considered at the evaluation stage.

The marking process is now all done via the web, although some find this frustratingly slow if they have to copy their marking information from some other source to the web forms.

The Evaluation Panel

The evaluation process usually involves a face-to-face meeting of all the markers, or just those from the Executive. The exact conduct of the meeting will depend on the size and complexity of the call, the number of bids and the number of projects to be funded. Every bid will have been marked by at least 3 people. Typically, one that scores AAA will be approved without further discussion, and one that scores CCC is likely to be rejected without significant discussion, although the panel will make sure that there’s sufficient information in the comments to provide feedback for the CCC bid.

What happens next depends to a great extent on the degree of competition, the quality of the proposals and the way the evaluation panel chair chooses to work. If there are many more proposals than can be funded, it’s not unusual to try to pick off further outliers – ones that stand out from the rest as being particularly strong or weak – before examining the rest in detail. The marks in each criterion will often come into play here, either to choose between two bids with equal recommendations, or to compare (say) an ABC with a BBB. Markers may be asked to justify or clarify their comments, and opinions do change as the result of discussion at this stage. The Executive will also want to bring other considerations into the process – either to ensure a range of different types of project are funded, or to ensure that funding goes to a range of institutions. Similarly, bids are sometimes approved subject to (agreed) change. Scores of 3 or below for any criteria imply that the marker sees problems that can be, and should be, corrected before funding is approved. If one institution receives funding for a number of related projects, they may be asked to look for economies of scale between them.

When there’s less or even no competition (with the number of bids being less than or equal to the number of projects desired) then the evaluation will have a different focus. It will still be necessary to eliminate projects that are too weak to receive funding, even if that means that some funding will be unspent. (Sometimes that funding can be reallocated to another stream where it can be better spent.) For those that can be funded, any concerns that the markers had need to be turned into guidance for the programme managers or the projects themselves. The budget may need to be made clearer, the dissemination plan improved, or the project may need to take account of the work of a related project, for instance.

The challenge for me is how to bring in a good copy of a wiki site without causing Web Curator Tool to gather too many pages from it. We don’t want that, because (a) the finished result occupies unnecessary space in the archive and (b) because it takes so long to complete that it can hold up the gather queue in the shared web-archiving service, delaying the work of other UKWAC partners.

I am not technical enough to tell you in great detail what’s causing this, although I sense that it’s something to do with the Heritrix crawler requesting too many pages from the wiki. When you consider that a wiki is database-driven it should not surprise us that it’s creating a lot of its pages on the fly. Secondly, since a wiki is editable by lots of contributors (that’s its core function after all), it presumably means we have numerous past versions of pages also stored somewhere in the wiki labyrinth, and it’s possible that the implacable Heritrix will not cease until it’s faithfully requested and copied every single one of them.

Let’s look at the Repositories Research Team wiki (DigiRep) owned by UKOLN, which I tried to gather five times in 2008. WCT conveniently keeps a history of these attempts, information about which I can still access even if the actual gathered pages have been discarded or archived. The size problems were chronic. Of five 2008 gathers, one was aborted after it had reached a massive 16.87 GB; a second one was rejected at 14.69 GB. I have archived one impression at 5.31 GB, another at 736.26 MB and another at 157.36 MB. Quite large variations there, which was worrying enough in itself.

At first, my workaround was to adjust the Profile Setting in the title to override the maximum number of documents Heritrix can gather. Setting ‘Maximum Documents’ at 10000 worked, but it was not ideal; I suppose all this means is that Heritrix stops when it collects 10,000 pages, whether we have everything we want or not. (I found that the copies in the archive seemed to render OK however).

To get a closer look at what’s going on, I started to browse the Log Files created by WCT (complete records of every single client-server request), which show patterns which I can vaguely understand; when these Log Files are packed with near-identical strings of code I sense that something’s up. For example, a string containing index.php?title=Repositories_Research&action=edit tells us that the wiki is requesting a specific named page, and allowing an edit action on that page. If you multiply that by the number of pages in the wiki, you can see how the problem builds up. (PHP is the script used for MediaWiki’s web scripting engine).

I follow this up by browsing the actual gathered pages in Web Curator Tool using the Tree View. From here I can click on the ‘View’ button to examine a page which I think to be suspect, and compare it with other suspect pages. Lastly, I go back to the live DigiRep site to confirm in my mind what’s happening when certain links are followed.

All the above gave me just about enough information to experiment with exclusion filters. After a certain amount of trial and error, and working with other Media Wiki sites, I arrived at the following exclusion codes which I can add to the Profile Setting:

.*&oldid.*
.*&diff.*
.*&limit.*
.*&direction.*
.*Recentchanges.*
.*/Special.*
.*\?title=Special.*
.*&action=edit.*
.*&action=history.*
.*&section.*
.*&redlink.*
.*&printable=yes.*
.*&redirect=no.*

These have the effect of telling WCT to exclude certain pages and actions from Heritrix’s harvesting action. The expectation was that I would lose the discussion / edit / history functions of the wiki in the archive copy.

The title with the above exclusion profile gathered just 63.41 MB and it completed in under ten minutes. I would say that’s an improvement on 16.87 GB. Log Files and the Tree View confirmed the success of this new “slimline” gather. As well as losing the discussion / edit / history functions, we also have eliminated the Toolbox functions, the ‘printable’ views, and the login pages.

This is no great loss at all for our purposes, as scholars who browse the archived copy of DigiRep are not expecting to be able to edit pages, nor join in the discussions, nor browse the history of stored versions of pages. Indeed in a lot of cases, they would require a login to do so. The users simply want to see the results of the DigiRep team’s work.

Keen readers may recall a post from April 2008 about my website-archiving forays working with Web Curator Tool, the workflow database, used for programming Heritrix, the crawler which does the harvesting of websites.

Other UKWAC partners and myself have since found that Heritrix sometimes has a problem, described by some as ‘collateral harvesting’. This means it can gather links, pages, resources, images, files and so forth from websites we don’t actually want to include in the finished archived item.

Often this problem is negligible, resulting in a few extra KB of pages from adobe.com or google.com for example. Sometimes though it can result in large amounts of extraneous material, amounting to several MB or even GB of digital content (for example if the crawler somehow finds a website full of .avi files.)

I have probably become overly preoccupied with this issue, since I don’t want to increase our sponsor (JISC)’s overheads by occupying their share of the server space with unnecessarily bloated gathers, nor clutter up the shared bandwidth by spending hours gathering pages unnecessarily.

Web Curator Tool allows us two options for dealing with collateral harvesting. One of them is to use the Prune Tool on the harvested site after the gather has run. The Prune Tool allows you to browse the gather’s tree structure, and to delete a single file or an entire folder full of files which you don’t want.

The other option is to apply exclusion filters to the title before the gather runs. This can be a much more effective method. The method is to enter a little bit of code in the ‘Exclude Filters’ box of a title’s profile. The basic principle is using the code .* for exclusions. .*www.aes.org.* will exclude that entire website from the gather. .*/images/.* will exclude any path containing a folder named ‘images’.

So far I generally find myself making two types of exclusion:

(a) Exclusions of websites we don’t want. As noted with collateral harvesting, Heritrix is following external links from the target a little too enthusiastically. It’s easy to identify these sites with the Tree View feature in WCT. This view also lets you know the size of the folder that has resulted from the external gathering. This has helped me make decisions; I tend to target those folders where the size is 1MB or larger.

(b) Exclusions of certain pages or folders within the Target which we don’t want. This is where it gets slightly trickier, and we start to look in the log files of client-server requests for instances where the browser is staying in the target, but performing actions like requesting the same page over and over. This can happen with database-driven sites, CMS sites, wikis, and blogs.

I believe I may have had a ‘breakthrough’ of sorts with managing collateral harvesting with at least one brand of wiki, and will report on this for my next post.

In the Handbook we’ve tried to cover a huge, and sometimes controversial, area in as accessible a way as possible. The workshops, attended by both web-management and records-management professionals from HE institutions, brought  a wide range of concerns and issues to light. It’s been quite a job fitting it all in.

Even as the project progressed, we became aware of new developments in thinking about how to approach the special issues of managing web resources, including everybody’s favourite new fast automatic Web 2.0 applications. We saw the publication of Steve Bailey’s Records Management 2.0 book, TNA’s Web Continuity project, and further web archiving developments at UKWAC. We’ve even heard it whispered in some quarters that approaches to preservation may need a more profound reassessment in the context of the Web and the Cloud. Many of these issues were recorded on the PoWR blog, and we tried to reflect as much of this in the Handbook as possible.

Another recent JISC publication, The Digital Preservation Policies Study by Charles Beagrie Ltd, published at the same time, is complementary in many ways, and reassured us that many of the conclusions we groped towards in the Handbook were not so wide of the mark! Like PoWR, the  Digital Preservation Policies Study identified the necessity of high-level policy engagement as the sine qua non of effective digital preservation.

Digital preservation solutions are undoubtedly partly technical, and the tools being created will enhance digital longevity, but these solutions are also equally dependent on organisational issues. It is important to remember that digital preservation relies on the interaction between the digital preservation environment and wider organisational objectives and procedural issues. These could be financial and staffing issues, collection management, legal obligations, auditing requirements, and other strategies and policies. In this respect, recognition by organisational divisions that digital data is important and key to the successful running of an organisation is crucial.

The Digital Preservation Policies Study, p.11

Among the other recommendations the Study shares with PoWR include:

• Analysis of existing policies and strategies, and how our work can support them even if said polices don’t explicitly refer to preservation or digital assets
• Taking a phased approach – nothing happens all at once. (PoWR recommends pilot projects and working with supportive departments.)
• Careful scoping of preservation requirements. (With regard to web resources, PoWR suggests not everything, not every version, and not forever.)
• Identifying if and where existing systems will do the job
• Consideration of lifecycle, publication, and retention schedules.

The Charles Beagrie survey is a very concise and accessible contribution to the field, and we hope the PoWR Handbook, with its specific focus on established and emerging Web issues, and attention to the detailed and everyday concerns of our many contributors and correspondents, will be similarly useful. We also hope that the work of PoWR will continue in some form, on the blog and perhaps in the form of new projects and workshops, to fill in the gaps we left, and deal with the constantly emerging Web developments. Anyone for PoWR 2.0?

This event serendipitously precedes the DPC/DCC/JISC/RSP workshop on repository preservation, up the road in Euston on Friday 12th: expect to see a few familiar faces cropping up at both events!

On a related note, check out also  Les Carr’s recent blog post, The Value That Repositories Add, which includes a fascinating presentation (intended for the SPARC conference in Baltimore) demonstrating the many ways that repository software can potentially add value for its users – statistics, bibliographies, personalisation, mailing lists, news feeds, and all sorts of Web 2.0 mashupery. As Les points out, not all repositories provide all these facilities, but all are possible with the current generation of repository software, and only a little bit of extra technical wizardry. Our own resident Eprints wizard is busy working on some of them even as I write…

The JISC Digitisation Programme has made a wide variety of valuable resources digitally accessible, including:

• British Newspapers (1620-1900)
• Newsfilm Online
• First World War Poetry
• Newspaper Cartoons
• Welsh Periodicals
• Pre Raphaelite drawings
• East London Theatre Archive

More information about these, and other projects, is available on the JISC Digitisation web page.

The project will review the preservation plans and processes of the sixteen projects funded under Phase 2 of the JISC Digitisation Programme, and identify any medium or long-term access risks to the digitised content. It will also produce recommendations – for individual projects and for JISC as a whole – for processes and strategies to mitigate the risks, and case studies which would be helpful to the broader community.

This is an exciting opportunity for us to apply and extend the experience we have gained working on a range of projects in the field, including the European Visual Archive Market-validation Project (EVAMP) and risk assessments for the recently launched Newsfilm Online project. We will shortly be creating an online home to for the project collaboration and development, and will use DA Blog and the DigiPresSurvey Blog (on JISCInvolve) to keep you updated.

We also heard from Murray Weston of BUFVC, whose self-evident enthusiasm and passion for his subject was clearly key to the project’s success. Murray made a strong case that video content is the poor relation in national collections (I think recorded sound runs it a close second, but they are both worthy of more attention.) Many millions are spent on the copyright libraries and they work as part of a system which allows us to use our local library as a portal to a vast resource of printed publications via inter-library loan. For film and video, by contrast, there is no system of legal deposit for material, hence no guarantee that material will be preserved for the nation. Even for the material that does exist, loan agreements are generally absent, and we must visit places such as the NFTVA to view material.

The collection itself is impressive and we were given a good grasp of the breadth and depth of the material available. But the part of the day that made a lasting impression on me was the presentation from Kevin Burden of the University of Hull. They were funded by JISC in what’s known as an assisted takeup project – one which helps the community to exploit the results of another project, be it software, guidelines or digital resources such as NFO. Their work in developing examples of teaching and learning using the Newsfilm Online archive was inspiring, all the more so because they were doing this before the service was fully populated with material. Kevin’s presentation used clips from workshops in a way which illustrated the very techniques he was promoting, and the matrix of use cases – although still only partially populated – is a simple and effective tool which will help others to make use of this resource. I understand that JISC are interested in using this model for other collections, and this can only be a good thing.

And finally… The memory that many may take away is not only of the wealth of material, but of the impact it will have on Malcolm Read, who confessed at the opening to not having owned or watched a television ‘since Bamber Gascoigne looked young.‘ Scholars have generally dated this period to the early 1970s.

There will be quite a few changes from the current beta version. Many of these are the result of Professor Katharine Ellis’s extensive advocacy and consultation, among both the musical research community, who will be the system’s users, and among the repositories community, at events like the JISC Rights workshop, and the Repository Fringe in Edinburgh. Others have been made possible as a result of the experience we’ve gained on other Eprints projects, like Linnean Online, Their Past Your Future and SNEEP.

As you can see from the screenshot, we’ve freshened up the overall look, and Rory has had considerable success with the embedded Flash player, which is going to make both video and audio content in PRIMO far more usable and accessible than the present, download-oriented system.

There are more changes and enhancements and a snag list as long as your arm, but we’re confident that when PRIMO gets its full and final launch in January, it will be a major improvement. There is also exciting new content currently being reviewed by the Committee, that will showcase the potential value of PRIMO to its community, and, we hope, encourage other musical researchers to contribute their own research-in-practice.

Chris Rusbridge has already done an excellent job of some near-real-time reporting on the sessions, and I’m not going to try to replicate that level of detail in this post. As a first-time attendee at iPres, I was impressed by the professional mix attending, which took in hard-core computer science, digital preservation and curation folk, repository managers and those from the traditional custodial professions. In that respect it was very reminiscent of the early DLM-Forums, which were eye-opening for me when I attended the first one in 1996. But it was also interesting to observe that, just as DLM was dominated by archivists and records managers, iPres is a very library-oriented event. For example, those who expressed a desire for a Europe-wide event bringing together all those with an interest in digital preservation didn’t seem to be aware that the DLM-Forums existed.

One positive observation (of many) is that there is more reassuring news on the oft-vexed issue of IPR barriers to digital preservation. At the close of day 1, we heard a summary of the findings of the international survey on the impact of copyright law on digital preservation. That indicated that the UK had one of the strictest set of constraints of all the countries looked at – in terms of who is permitted to carry out certain acts in the name of preservation and what those acts are. Other countries have more relaxed exemptions and that doesn’t appear to be causing the major rightsholdfers any significant financial loss. That should give us hope for some change in the law in the UK at least. And Horst Foster, making the keynote speech opening day 2, appeared to echo this at the European level, implying that the case for change had been made and accepted, although he was notably cautious about making any promises as to when this change might come about.

That’s an improvement on the situation in Europe a few years ago, though, when I was at one of a number of expert panels helping the European Commission to frame the forthcoming research questions and challenges in the digital preservation arena. At that time we were all warned off mentioning the (C) word at all – it seemed to have a somewhat toxic flavour. It’s really heartening to see that things have changed.

One should add a note of caution, however. After Adrienne Muir had commented favourably on how Australian law allows institutions such as the national library and archives to bypass DRM systems in order to preserve material, Colin Webb injected a note of caution. It is apparently still illegal to manufacture or to import a device to Australia which allows a DRM system to be bypassed. But if the national library happens to find such a device on its premises, it can use it without fear of breaking the law. Still some way to go, then, before the law is ‘joined up and working’ – the strapline of iPres2008.

As usual, a JISC gathering is a great opportunity to meet others working in the field – plenty of familiar faces and some new ones. JISC-sponsored innovations move on at a head-spinning pace in all the areas we are directly or indirectly involved in, from digital preservation and archives to repositories and e-learning. With JISC-PoWR in mind, I was especially interested to meet our former ULCC/RSC colleague Sarah Sherman, who is now working on the APT-STAIRS project, investigating the use of Google Docs for students, teachers and researchers. Definitely a preservation angle here, that I hope we’ll be able to follow up.

Plenty about the conference, and more, on the JIF08 blog; if contemporaneous twittering is your bag, check out twemes.com/jif08.

ULCC are currently leading a project (sponsored by JISC) called Assessing Institutional Digital Assets, or AIDA. We’re looking for institutions in the HFE sector in the United Kingdom who would like to help us, by participating as a case study for this project.

The idea is that you would complete a guided self-assessment task which we hope will make things clearer in relation to you and your digital assets. We plan to do this around June-July 2008.

For this, we have drafted a self-assessment toolkit which would help determine your institution’s current capacity for digital asset management. It will help you assess your institution’s ability / readiness for digital asset management. Based on that assessment of readiness and maturity, later project outputs will provide recommendations on appropriate steps to take to improve digital asset management for you. (We’re approaching different institutions who are likely to be at different stages of maturity). The toolkit can be found at http://aida.jiscinvolve.org/toolkit/.

The process and the outputs of this project may be of some benefit to you. We think that the tools, guidance and case studies will help institutions understand how to take small steps forward to improve institutional maturity in regard to digital asset management and preservation concerns.

We’re looking to work with the following information experts: records managers, librarians, digital librarians, data curators, repository managers, information managers, digital asset managers, web masters, archivists, and others. Our guess is that there is no single person in the institution who can do the entire self-assessment, so it may turn into a team effort. There is also the possibility of on-site support or remote support from ULCC. We’re able to provide some financial support, via our JISC funding, to a small number of case-study sites.

If you’re interested in participating, we are looking at starting around June or July 2008, depending on availability of yourself and your staff. We expect the work to take an absolute maximum of eight days, but our hope and expectation is that it will be less for many institutions. At this stage, we are looking for participation from UK Higher Education institutions only, although comments from others are welcome.

Further information is available at our project website.

Contacts: Ed Pinsent (e.pinsent@ulcc.ac.uk) / Patricia Sleeman (p.sleeman@ulcc.ac.uk)