TRAC: C1.1: Well-supported core infrastructure

C1.1 Repository functions on well-supported operating systems and other core
infrastructural software.

The requirement specifies “well-supported” as opposed to manufacturer-supported or other similar phrases. The level of support for these elements of the infrastructure must be appropriate to their uses; the repository must show that it understands where the risks lie. The degree of support required relates to the criticality of the subsystem involved. A repository may deliberately have an old system using out-of-date software to support some aspects of its ingest function. If this system fails, it may take some time to replace it, if it can be replaced at all. As long as its failure does not affect mission-critical functions, this is acceptable. Systems used for internal development may not be protected or supported to the same level as those for end-user service.

Evidence: Software inventory; system documentation; support contracts; use of strongly community supported software (i.e., Apache).



At the foundation of ICPSR's core technology infrastructure is Red Hat Linux. Linux, of course, is a very widely deployed open source variant of UNIX, and Red Hat is a world leader in supporting Linux. ICPSR had previously used proprietary operating systems, but moved all of its systems to Red Hat Linux in the decade.

Moving up in the stack of our core technology infrastructure we use several pieces of software from the Apache Software Foundation, a very large community of developers and users. In addition to the flagship HTTP Server, we also use the tomcat servlet container for all of our Java-based web applications, the Solr search engine which is built atop the Lucene Java search technology, and the cocoon framework for rendering XML into other formats.

ICPSR also builds and maintains a suite of custom software for key business processes, such as our Data Deposit Form (ingest), our download system (access), and our data processing systems (data management, ingest, access). While this software is necessarily proprietary, it is written in common, modern software languages such as Perl and Java, which have wide support in the community.

Like many, many enterprises, ICPSR uses Oracle as its database system. Given the large installed base of Oracle across the world, ICPSR views this as a well-supported platform. We believe we can continue to use Oracle as long as the University of Michigan continues to make it freely available to us. Further, we make use of only the most basic elements of Oracle, and if required, it would be straight-forward, but not insignificant, to migrate our content to any other relational database technology, such as postgres or mysql. Our one highly customized use of Oracle, and therefore the one that carries the most risk, is the OracleText-based search engine for the Child Care site, is scheduled to be replaced in early 2010.

And finally, we are actively migrating our own proprietary archival storage system to the Fedora system, which is supported by the newly created DuraSpace organization. Other than internally created systems, Fedora probably has the smallest community of support of any of our major technology systems, but because it is open source and gaining traction in the community, we believe its level of support will continue to grow over time. And further, because the underlying content resides in plain XML files, even with a sudden and catastrophic loss of Fedora, it would still be possible to migrate content to another system.

Trustworthy Repositories Audit & Certification: Criteria and Checklist:

We're working through the Center for Research Libraries (CRL) Trustworthy Repositories
Audit & Certification (TRAC) Criteria and Checklist. My primary responsibility is to review Section C, which focuses on technology. Having looked through the TRAC document several times now, my sense is that ICPSR has a pretty good story to tell, and the main task is to take the time to tell the story. I thought it might be interesting to begin a series of blog posts, one per item from Section C in the TRAC checklist.

I have three goals for the posts:

1. Tell ICPSR's story for each item so that we have transparent evidence and documentation in each area
2. Share ICPSR's evidence and documentation for others who are working through the TRAC document, or who are thinking about starting; I think it's often easier to work with something that someone has already begun v. starting from scratch
3. Solicit input from others who have completed TRAC certification: What have we missed?

There are ten items in Section C.1 and I'll see if I can't work through one item/week between now and the end of the calendar year. Look for the first item later this week.

And if you're working through TRAC yourself, I'd love to hear any advice you have (either as Comments here or via email).

More Grant News: IMLS National Leadership Grants

Micah Altman and Gary King of Harvard's IQSS put together a proposal to the IMLS to extend work on a LOCKSS-based archival replication system we started as part of our Library of Congress-funded Data-PASS project, and it looks like the IMLS will be funding it. A nice summary of recent grants is available on the IMLS web site, and also on the Library Journal web site. It wouldn't be right of me to mention the LOCKSS-based system without also mentioning the Roper Center and the Odum Institute as the other key players in building the initial prototype. It was a real team effort.

And while it isn't technology-oriented....

ICPSR picked up its own IMLS grant for rescuing "at risk" social science data. George Alter is the PI, and his project is another follow-on that leverages our Data-PASS partnership.

IASSIST 2010 Call For Papers

In case you haven't seen this already, here's the call for papers for the next IASSIST meeting:

IASSIST 2010

Social Data and Social Networking:
Connecting Social Science Communities across the Globe
1-4 June 2010
Ithaca, NY, USA

IASSIST 2010, the 36th Annual Conference of the International Association for Social Science Information Service and Technology (IASSIST) will be hosted by the Cornell Institute for Social and Economic Research (CISER) and Cornell University Library (CUL) and will be held at Cornell University, in Ithaca, New York, USA, on 1-4 June 2010.

The theme of this year's conference is Social Data and Social Networking: Connecting Social Science Communities across the Globe. Social science has begun to feel the impact of the dramatic shift in communication patterns globally, where social networking and other digital media trends are changing how social scientists study the world around them. This theme is intended to stimulate discussion about the impact of social networking on the creation, collection, sharing, storage, preservation, dissemination, confidentiality, licensing of, and access to data. Of particular interest is how social connectivity has facilitated multi-site and cross-national social science research.

A webform for submission of proposals will be available on the conference web site: http://ciser.cornell.edu/IASSIST/ from 12 October 2009.


Deadline for submission: 30 November 2009.

Notification of acceptance: 1 February 2010.

For more information about the conference, including travel and accommodation, see the attached PDF Call for Papers or visit the conference web site at:
http://ciser.cornell.edu/IASSIST/ .


IASSIST is an international organization of professionals working in and with information technology and data services to support research and teaching in the social sciences. Typical workplaces include data archives/libraries, statistical agencies, research centers, libraries, academic departments, government departments, and non-profit organizations, see http://www.iassistdata.org for further information.

As usual, I suspect that ICPSR will submit several presentations and papers for the conference. I think an update on either (or both) of the recent tech-oriented grants might be appropriate: using the cloud to deliver sensitive and under-utilized data, and using Fedora to preserve and delivery social science datasets and documentation.

Designing Storage Architectures for Digital Preservation - Day Two

[ Due to a combination of my own stupidity and the way in which Blogger does (or doesn't!) do auto-save, many of my notes for the second day disappeared sometime between DCA and DTW. So a very abbreviated set of notes for Day Two. ]

The first session, Data Integrity, began at 9:00am with a series of vendor presentations.

1. Henry Newman, Instumental: small market for digital preservation-quality systems. Disk density and transfer rate have outpaced reliability with disks. Still need tapes due to low power and high capacity and high reliability. HSM lacks broad market acceptance. Asserts that loss of a single bit is catastrophic. Mismatch between preservation requirements and main market requirements.
2. David Rosenthal, LOCKSS. Different usage patterns between content going into archives, and content retrieved on a regular basis. Designs and thinking need to take account of this.
   
3. Ray Clarke, Sun Microsystems. Draws distinction between backups v. archiving (preservation). Content growth exponential. Most data in archives is used infrequently. Asserts the tape "continues to make sense" for preservation: power, portability, etc. Humans introduce most errors, and so need to future-proof.
4. Mike Mott, IBM, spoke how some loss in some contexts is acceptable. Need to hit the "utility" number. Shared some stories from the past about needing to solve error detection and correction problems in the end-to-end system, not just within each component.
5. Tim Harder, EMC, High-Assurance and Integrity Layer. "Law of Large Numbers is not on your side." Described approach similar to LOCKSS and DuraCloud. Use sampling to validate correctness of data.
6. Paul Rutherford, Isilon, failures = disk drive, controller/node, human. Do not trust storage. Do not trust yourself. Need to recover from failures fast enough. Overall system must be available in face of failure in large components. "RAID is dead." Not good enough. "We called it 'grid' before 'cloud'."
   

Designing Storage Architectures for Digital Preservation - Day One

This is the first day of a two-day workshop on storage architectures for digital preservation. The workshop is hosted by the Library of Congress at the Churchill Hotel in Washington, DC. There are about eighty or so attendees, many from the LoC itself, but also many "tool makers" (Sun, EMC, Seagate, Cisco, etc.) and "data stewards" (ICPSR, MetaArchive, HathiTrust, etc). My apologies in advance where I have misunderstood or misquoted a speaker below.

The workshop began at noon with a luncheon, followed by a brief Opening/Welcome session. This moved quickly to a 90-minute session, Storage for Digital Preservation: Panel of Case Studies from Users, which began at 1:15pm. There were eight, seven-minute presentations:

1. Thomas Youkel, Library of Congress, cited some numbers about the amount of content ingested by the LoC, and the amount they project they will ingest during the next two years. One interesting figure is that the LoC ingested 24.7 TB during the week of June 2, 2009. He also described data integrity as a key challenge, and workflow, content management, and migration as secondary challenges.
   
2. David Minor, San Diego Supercomputer Center, gave a brief overview of the Chronopolis project: three partners (SDSC, NCAR, UMIACS), 50TB of storage at each node. SRB is the content transport system; BagIt is the content container; and, ACE is the content integrity system. ICPSR, the California Digital Library, the MetaArchive, and one other organization I missed are the content providers. Chronopolis Next Generation is seeking additional storage partners (nodes), migration tools, and connecting to other storage networks, such as the MetaArchive's private LOCKSS network.
3. Bill Robbins, Emory University, described how the MetaArchive was using an Amazon EC2 system as its central "properties server" to solve the (political? procedural?) issues of Emory serving as the "master node" for the MetaArchive. Bill had a good quote: "We're not cheap. We're 6x cheap, and that's not so cheap." Bill expressed general satisfaction with EC2, but wished the documentation was better.
4. Andy Maltz, Academy of Motion Pictures Arts and Sciences, reference the Digital Dilemna in his talk about the requirements his organization has for digital preservation solutions: (1) last 100 years; (2) survive benign neglect; (3) at least as good as photochemical; and, (4) cost less than $500/TB/year. Andy also referenced the phrase "Migration is broken" from a 2007 SNIA report. He cited some figures: a movie consumes 2-10PB of storage, and Hollywood produces about one move/day. He finished with a brief description of StEM, an NDIIPP-sponsored project.
5. Laura Graham, Library of Congress, described the LoC's efforts to preserve websites. The Internet Archive does the crawling, and the Wayback Machine is the delivery mechanism. A system at the LoC acts as archival storage. Wish list includes fewer manual steps in the system, and less of a need to copy files around quite so much.
6. John Wilkin and Corey Snavey, Hathitrust (and the University of Michigan Library), gave a brief overview of Hathitrust. They're leveraging the OAIS reference model, plugging in modular solutions wherever possible. 185TB of storage today. Focus is on the "published record." Corey asserted that data stewards will need to be able to rely more and more on the storage solution (trust) in order to succeed in their missions.
7. Jane Mandelbaum, Library of Congress, was the proxy for a very brief overview of the DuraCloud effort from DuraSpace. DuraCloud is essentially a middle layer between a variety of cloud storage providers and data stewards.
   
8. Jimmy Lin, representing Cloudera, described Cloudera as the RedHat for Hadoop. Jimmy went on to talk a bit about Hadoop, HDFS, and MapReduce, and how Cloudera might be a very attractive platform for connecting "compute" to storage.

The session concluded with a general conversation about key issues in digital preservation: trust; costs; not knowing what bits will be considered valuable up-front; and, how frequency of access is unknown. David Rosenthal had a good line: "You have to get used to the idea of losing stuff." There's no magic bullet that will keep lots of bits around for a long time without any loss.

My main comment during the session was that software and hardware and even power are not the big costs of digital preservation (at least at ICPSR); the big costs are people, and the processes that require people.

After a short break, we began the next session, Storage Products & Future Trends: Vendor Perspectives, at 3:15. Again, the format was a seven-minute presentation:

1. Art Pasquinelli, Sun Microsystems, spoke about the Sun PASIG, an d described how researchers were looking to IT and libraries for their digital preservation needs, and how that was simply not working.
   
2. Mike Mott, IBM, asked how we define a "document" in a digital world, and thought that we would see the end of Moore's Law (in storage) by 2013 unless there was a new technological breakthrough. Mike also described a new paradigm in architecture: a river v. a building.
3. Dave Anderson, Seagate, spoke on how he expected some trends to end (approx 40%/year increase each year in capacity + 20%/year increase each year in transfer rate); how solid state disk uptake has been slower than expected; and, how the change in disk form factor from the desktop (3.5") to the laptop (2.5") will shift the industry.
4. Tim Harder, EMC, described a new "compute + storage" solution called Atmos, and how they are betting big on x86 technology + virtualization, and off-the-shelf gear bundled with software. EMC has also founded a cloud division.
   
5. Paul Rutherford, Isilon, expects SATA, 3.5" form-factor disks, and block-level access to disappear, replaced by SAS, 2.5" form-factor disks, and file-level (or object-level) access. He said "I hate the cloud" and didn't think it would be used as the sole source for important data.
6. Kevin Ryan, Cisco Systems, gave a high-level overview of Cisco's "unified fabric" vision which sounds like it consists of a single, lossless, open pipe for all sorts of bits: network, data, NAS, SAN, etc.
7. Raymond Clarke, Sun Microsystems, gave a similar type of talk, but about the Sun Cloud which struck me as an umbrella term for an integrated solution using lots of Sun's open technologies: Solaris, Java, ZFS, MySQL, etc.

The session ended with another general conversation about trends. We then had a brief close-out (with homework!) to end the day about 4:45pm.

More Good News: ICPSR Wins NSF Grant

There's more good news for ICPSR on the funding and technology front. I learned last week that our NSF proposal to the EAGER (EArly-concept Grants for Exploratory Research) program was funded. In the proposal, Shared Digital Technologies for Data Curation, Preservation, and Access: A Proof of Concept, we describe a series of objects (Content Models, Service Definitions, Service Deployments) that we will build as an exemplar on how one might store social science datasets and documentation in a Fedora-type repository.

We have been exploring Fedora quite actively since the release of version 3, and this is a wonderful opportunity to move our work from the wings and into center stage. The majority of our efforts so far have been focused more on transferring our legacy content (the keepsakes) from file-based storage into object-based storage, but the work from this grant will be much more forward-looking. This will be a chance to renew work on the early content modeling work I reported on back in May.

My sense is that between this grant and the NIH Challenge Grant, we'll need to look for a reasonably senior systems analyst/developer before the end of the calendar year. (We may even need to add two.)