Over the next year I’m creating a plan for investigating and implementing digital curation micro-services in some form. A first step in this project is a learning environment on the topic.

http://www.netvibes.com/stevemtzn#curation_micro-services_news

Archivematica is an open source digital preservation platform using the micro-services design idea “that allows users to process digital objects from ingest to access in compliance with the ISO-OAIS functional model”.

The MetaArchive Cooperative has released a Guide to Distributed Digital Preservation:

This volume is devoted to the broad topic of distributed digital preservation, a still-emerging field of practice for the cultural memory arena. Replication and distribution hold out the promise of indefinite preservation of materials without degradation, but establishing effective organizational and technical processes to enable this form of digital preservation is daunting. Institutions need practical examples of how this task can be accomplished in manageable, low-cost ways.

Here’s a quick chart that specifies the amount of disk space a single scanned 8.5″ x 11″ master file will take up. Pixels per inch (PPI) and bit levels were taken largely from the BCR best practices document. These estimates were generated with PhotoShop CS4.

Gary McGath has an interesting post up over at File Formats Blog arguing for a little wiggle room within file format preservation:

JHOVE 1.x takes a “one strike, you’re out” approach to compliance, with certain exceptions. … JHOVE2 improves on this by introducing the concept of assessment. … What I’m proposing here is that a metric of assessment should be reconstructability, the degree to which a file can be reconstructed in spite of non-standard aspects in its content or structure.

Found this posted earlier last week via Digital Koans:

Digital Project Staff Survey of JPEG 2000 Implementation in Libraries

The data is presented in an xls, that includes all survey questions and comments from those who volunteered information. What I thought was most interesting was that only 60% of respondents (out of 173 total) were using JPEG2000 in any way. The followup question was “if not, why not”, and the general themes include the lack of tools, adoption, and understanding of the format.

For digital preservation, or archival uses, only 20% were making use of the format. There are a lot of very interesting tidbits in this spreadsheet.

David Rosenthal has a detailed and useful analysis of the value of open format specifications for preservation:

If we plot the quality achieved by a newly created renderer for a format against the cost of creating it we will get an S curve. A certain amount of money is needed to get to a barely functional renderer. Beyond that, quality increases rapidly at first but after a while the law of diminishing returns sets in. Getting from 99% to 99.9% is very expensive; the cost of getting to 100% is infinite. … It seems clear that preserving the specification for a format is unlikely to have any practical impact on the preservation of documents in that format. If, during the currency of the format, it acquires an open source renderer there is no significant risk of ever ending up without a functional renderer. The need for a new one to be created from the specification is extremely unlikely ever to arise. If that unlikely event ever happened, it is hard to believe that resources on the scale needed to do the job would be available. And in the unlikely event that they were, it is unreasonable to believe that the combination of the preserved specification and the available resources would be enough to create a renderer that would satisfy those who reject Open Office because of minor rendering flaws.

Don’t let the perfect be the enemy of the good.

Clearly, formats with open source renderers are, for all practical purposes, immune from format obsolescence. …

[emphasis mine]

—

Addendum:

Chris Rusbridge makes some good points and offers a summary of the DCC email thread

Found this useful idea by Chris Rusbridge from a year-end review by Jill Hurst-Wahl:

It’s not just the language that makes digital preservation unconvincing to the decision maker. Part of the problem is that digital preservation describes a process, and not an outcome. … So I would argue that outcome-related phrases like “long term accessibility” or “usability over time” are better than the process-oriented phrase “digital preservation”.

This makes good sense to me, sorry I missed it earlier in the year.

Reading a press release out of UCSD, this comment  from the Blue Ribbon Task Force on Sustainable Digital Preservation and Access [pdf] caught my eye:

A recent study by the International Data Corporation (IDC) said that in 2007, the amount of digital data began to exceed the amount of storage to retain it, and will continue to grow faster than storage capacity from here on. The IDC study predicts that by 2011, our “digital universe” … will be 10 times the size it was in 2006.

That sounds about right, and is a little alarming. The University of Montana Mansfield Library recently digitized a run of Salish-Kootenai tribal newspapers called the Char-Koosta News.  While this was a relatively small project, it still amounted to the creation of 26,633 files at 253 GB of storage (and that doesn’t include what’s available on the public website.) If we are able to ramp up production 10x by 2011, we’re looking at an additional 2.5TB of data and possibly 270k files that will need to preserved, both compressed and uncompressed, from that point forward. It’s a bit staggering.

The report has a list of coming challenges seem spot on to me:

• Inadequacy of funding models to address long-term access and preservation needs.
• Confusion and/or lack of alignment between stakeholders, roles, and responsibilities with respect to digital access and preservation.
• Inadequate institutional, enterprise, and/or community incentives to support the collaboration needed to reinforce sustainable economic models.
• Complacency that current practices are “good enough.”
• Fear that digital access and preservation is too big to take on.

The last two bulletts will be the two I’ll be focusing on initially here at worrk.

I’ve been asked by a small-sized museum / special library to make a suggestion on what kind of equipment a start-up digitization studio would need while operating under a budget of approximately $3,200. This studio would need the cabability of digitizing photos, and possibly some text, post-processing the scans, hosting the images on an existing web server, and then preserving the master scans. It’s an interesting project, and one that can easily be accomplished within a tight budget. By putting the recommendation online, I’m hoping that others will find this and make further recommendations on the following:

Here are the criteria for the digitization studio:

1. Budget of $3,200
2. Manageable by a staff of basically one, with minimal to no technical support
3. Capable of digitizing photos & text according the best practices per the BCR Digital Imaging document
4. Capable of long-term preservation of master files
5. Capable of public access to the resulting web-ready scans

Here are some recommendations using the lowest prices (found via Google Product Search) as of this writing:

Scanning

Computer-Baseline: $600

What’s amazing to me is that a baseline studio-ready PC can be had for as little as $600, maybe less. While it’s easy to spend more, as long as it has a minimum of 2GB memory and 100GB+ hard drive you’re good to go. Upgrade options here include a larger monitor and more memory. Macs will be more expensive, but can be a good solution.

Scanner-Baseline: $735

PlusTek OpticBook 4600: I’ve used the OpticBook 3600 previously and was very happy with it. It offers a scanning area that is good enough for 8 1/2″ x 11″ photos while also being configured to drape books over the side in a way that won’t break the binding and still allows you to get deep into the gutter of the page.

Upgrade option – Epson 10000XL Photo: $2,500. This scanner has excellent dynamic range, which for photos is paramount but comes at a price. Other options include a digital camera, but if you go in this direction you’ll need to worry about lighting and dealing with RAW files. For a start-up studio that may be more trouble than it’s worth.

Post-Processing and Quality Control

Image-processing baseline: $176

Photoshop CS3 is probably your best option. The latest version is CS4, but for a start-up studio the earlier version is going to work fine. This software will be used to run the scanner for capturing master images and to create derivative files that are web-ready. Price: $150

Alternately, the open-source software Gimp is an excellent solution. It will do almost everything that Photoshop can do. The only downside is if your just learning image manipulation, most of the tutorials available are for Photoshop. Price: Free

Tutorials: Ben Willmore’s Photoshop CS3 Studio Techniques is an invaluable resource. It will walk you through everything you need to know to make web-ready derivatives: straightening, cropping, adjusting levels, sharpening, resizing, etc. Price: $26

File Name Editing: Bulk Rename Utility is a good resource for renaming an entire directory-worth of files. This is an invaluable resource and free.

Image Quality Control: I find IrfanView to be an excellent, light-weight, and fast image viewer. This is crucial if you need to browse through a couple thousand images quickly and easily. It also has the capability to do some simple image editing, like cropping. Price: Free

Optical Character Recognition: OCR will give you the capability to convert images of text to actual text. This will allow you to create PDFs as well as text files of the scans. Most scanners come with the Sprint version of Abbyy FineReader, which is fine for starting out. Upgrade to the Pro version is a good idea when the studio expands its capabilities. Price: Free ($179 for an upgrade to Pro)

Public Image Presentation

Hosting Baseline: Free (as in “free puppy”)

Greenstone is open-source hosting software designed with the small institution in mind. As long as you have a web server to place files on, you’re good to go. Alternately, you can go with CONTENTdm, which has either a hosted option or a version that you host yourself. Fees for this can be quite steep, so moving to this later may be an option. One caveat: if technical support is not available to you, getting content you’ve placed on Greenstone into CONTENTdm is not trivial. Plan on rebuilding your collections from scratch with any migration. Price: Free

Long-Term Preservation

Preservation Baseline: $749

This is about ensuring that the master files created in a digitization studio can be kept indefinitely, at least on a bit bit level. Migration, emulation, etc. is another issue entirely. (Most istitutions are taking the attitude that we’ll burn that bridge when we come to it.) For a small institution without a network, what’s needed is redundant storage that’s easy to maintain. Data Robotics has an elegant solution in its Drobo product. It accepts cheap and easily purchased hard drives which it monitors for you to guard against bit rot (which is inevitable, eventually). If a drive goes bad, you simply replace it and Drobo manages the reformatting and transfer for you. With this setup two copies would be kept: one on the scanning computer, and one on the Drobo. If it’s also possible to keep another copy on a network, that would help spread the risk.

Grand Total: $2,260

That’s about $1,000 below budget, which will give you room to maneuver. Upgrading from here I’d recommend getting as large a monitor as possible, upgrading the ram of the computer to 4GB, and possibly finding a scanner somewhere between the OpticBook and the Epson.

Comments and suggestions are welcome.