Over the next year I’m creating a plan for investigating and implementing digital curation micro-services in some form. A first step in this project is a learning environment on the topic.

http://www.netvibes.com/stevemtzn#curation_micro-services_news

Archivematica is an open source digital preservation platform using the micro-services design idea “that allows users to process digital objects from ingest to access in compliance with the ISO-OAIS functional model”.

The MetaArchive Cooperative has released a Guide to Distributed Digital Preservation:

This volume is devoted to the broad topic of distributed digital preservation, a still-emerging field of practice for the cultural memory arena. Replication and distribution hold out the promise of indefinite preservation of materials without degradation, but establishing effective organizational and technical processes to enable this form of digital preservation is daunting. Institutions need practical examples of how this task can be accomplished in manageable, low-cost ways.

Here’s a quick chart that specifies the amount of disk space a single scanned 8.5″ x 11″ master file will take up. Pixels per inch (PPI) and bit levels were taken largely from the BCR best practices document. These estimates were generated with PhotoShop CS4.

Gary McGath has an interesting post up over at File Formats Blog arguing for a little wiggle room within file format preservation:

JHOVE 1.x takes a “one strike, you’re out” approach to compliance, with certain exceptions. … JHOVE2 improves on this by introducing the concept of assessment. … What I’m proposing here is that a metric of assessment should be reconstructability, the degree to which a file can be reconstructed in spite of non-standard aspects in its content or structure.

Found this posted earlier last week via Digital Koans:

Digital Project Staff Survey of JPEG 2000 Implementation in Libraries

The data is presented in an xls, that includes all survey questions and comments from those who volunteered information. What I thought was most interesting was that only 60% of respondents (out of 173 total) were using JPEG2000 in any way. The followup question was “if not, why not”, and the general themes include the lack of tools, adoption, and understanding of the format.

For digital preservation, or archival uses, only 20% were making use of the format. There are a lot of very interesting tidbits in this spreadsheet.

David Rosenthal has a detailed and useful analysis of the value of open format specifications for preservation:

If we plot the quality achieved by a newly created renderer for a format against the cost of creating it we will get an S curve. A certain amount of money is needed to get to a barely functional renderer. Beyond that, quality increases rapidly at first but after a while the law of diminishing returns sets in. Getting from 99% to 99.9% is very expensive; the cost of getting to 100% is infinite. … It seems clear that preserving the specification for a format is unlikely to have any practical impact on the preservation of documents in that format. If, during the currency of the format, it acquires an open source renderer there is no significant risk of ever ending up without a functional renderer. The need for a new one to be created from the specification is extremely unlikely ever to arise. If that unlikely event ever happened, it is hard to believe that resources on the scale needed to do the job would be available. And in the unlikely event that they were, it is unreasonable to believe that the combination of the preserved specification and the available resources would be enough to create a renderer that would satisfy those who reject Open Office because of minor rendering flaws.

Don’t let the perfect be the enemy of the good.

Clearly, formats with open source renderers are, for all practical purposes, immune from format obsolescence. …

[emphasis mine]

—

Addendum:

Chris Rusbridge makes some good points and offers a summary of the DCC email thread

Found this useful idea by Chris Rusbridge from a year-end review by Jill Hurst-Wahl:

It’s not just the language that makes digital preservation unconvincing to the decision maker. Part of the problem is that digital preservation describes a process, and not an outcome. … So I would argue that outcome-related phrases like “long term accessibility” or “usability over time” are better than the process-oriented phrase “digital preservation”.

This makes good sense to me, sorry I missed it earlier in the year.

Found via the Washington University DLS blog, here’s an article from the NYT on the challenges faced by the National Archives when it comes to the massive data dump the Bush administration is about to unload.

The archives invoked its emergency plan to deal with problems in transferring two types of electronic files: a huge collection of digital photographs and the “records management system,” which provides an index to most of the textual records generated by Mr. Bush and his staff members in the last eight years. … If the electronic records of the Bush White House total 100 terabytes of information, as archives officials estimate, that would be about 50 times the volume of electronic records left behind by the Clinton White House in 2001 and some five times the contents of all 20 million catalogued books in the Library of Congress … [the] agency was expecting to receive 20 to 24 terabytes of e-mail alone from the Bush White House.

That’s an incredible amount of material for the archives to ingest. Since the administration, the Vice President’s office in particular, is not providing details on the types of materials being provided, what this amounts to is the National Archives trying to implement an OAIS solution with a hostile data provider. The reason they’re hostile is because the data has political implications in the short run, but the archives needs to preserve the material for the long run. In the next year or so, I hope the archives will write and present a case study what they eventually ended up doing.

Reading a press release out of UCSD, this comment  from the Blue Ribbon Task Force on Sustainable Digital Preservation and Access [pdf] caught my eye:

A recent study by the International Data Corporation (IDC) said that in 2007, the amount of digital data began to exceed the amount of storage to retain it, and will continue to grow faster than storage capacity from here on. The IDC study predicts that by 2011, our “digital universe” … will be 10 times the size it was in 2006.

That sounds about right, and is a little alarming. The University of Montana Mansfield Library recently digitized a run of Salish-Kootenai tribal newspapers called the Char-Koosta News.  While this was a relatively small project, it still amounted to the creation of 26,633 files at 253 GB of storage (and that doesn’t include what’s available on the public website.) If we are able to ramp up production 10x by 2011, we’re looking at an additional 2.5TB of data and possibly 270k files that will need to preserved, both compressed and uncompressed, from that point forward. It’s a bit staggering.

The report has a list of coming challenges seem spot on to me:

• Inadequacy of funding models to address long-term access and preservation needs.
• Confusion and/or lack of alignment between stakeholders, roles, and responsibilities with respect to digital access and preservation.
• Inadequate institutional, enterprise, and/or community incentives to support the collaboration needed to reinforce sustainable economic models.
• Complacency that current practices are “good enough.”
• Fear that digital access and preservation is too big to take on.

The last two bulletts will be the two I’ll be focusing on initially here at worrk.