Cold, Dark, and Lonely: An Archive Moves On-Line.

Carol Minton Morris of DuraSpace called me the other day with some good news: She told me that a short piece I wrote about Fedora and ICPSR would be published in their blog. The piece is called Cold, Dark, and Lonely: An Archive Moves On-Line.

While my colleagues at ICPSR have been alarmed by the title and suggested I seek immediate therapy for what must be an overwhelming foreboding of dread, the title was actually a poor riff on Thomas Friedman's Hot, Flat and Crowded tag. At least I think it was. (Maybe I should make that call after all.....)

Carol has also invited me to participate in the Sun/DuraSpace/SPARC webinar next week, All About Repositories. Should be a lot of fun!

ICPSR Job Posting in Technology - Cloud Computing Developer

We've posted the following position on the U-M employment site. (The site is just awful, but don't let that scare you off.)

We've listed it as a two year appointment to match the NIH Challenge Grant, but we've had a lot of success keeping staff employed quite happily and busily by generating more and more grant activity.

Job ID:	34671
Job Title:	Cloud Computing Developer
Job/Career Family:	Information Technology
Country:	United States of America
State:	Michigan
City:	Ann Arbor
Job Type:	Regular
Full Time:	Yes
Date Posted:	Oct 07 2009
Minimum Salary:	0
Maximum Salary:	0
Salary Frequency:	Annual
Job Description and Responsibilities:	Market Title: Systems Analyst Senior Working Title: Cloud Computing Developer FLSA: Exempt Salary Range: $70,000 - $80,000 depending on qualifications and experience of selected candidate Hours/Week: 40 Hours Shift/Hours/Days: Regular Business ** Please note this is a two year term limited appointment** The Inter-university Consortium for Political and Social Research (ICPSR), established in 1962, is an integral part of the international infrastructure of social science research. ICPSR's unique combination of data resources, user support, and training in quantitative methods make it a vital resource for fostering inquiry and furthering the social sciences. ICPSR maintains and provides access to a vast archive of social science data for research and instruction. A unit within the Institute for Social Research at the University of Michigan, ICPSR is a membership-based organization, with over 600 member colleges and universities around the world. A Council of leading scholars and data professionals guides and oversees the activities of ICPSR. ICPSR offers a work environment that is a combination of the best aspects of a small nonprofit or business, established within a university setting. ICPSR is small enough that each person can make a difference, yet large enough to offer a variety of career opportunities. We have a relaxed, collegial atmosphere that fosters communication and networking within and between departments. We are family-friendly, offering flexibility with work hours, and we have a diverse staff that enriches the workplace with their skills and experience. ICPSR offers a competitive total compensation package providing full access to the University of Michigan benefits. More information can be found about ICPSR at www.icpsr.umich.edu. The ICPSR computing environment consists of Windows desktop workstations and UNIX servers. The desktop workstations run typical business applications such as Microsoft Office, but also run statistical software such as SAS and SPSS. The UNIX servers are based on the Intel/Linux platform and include Oracle databases, World Wide Web server software such as Apache, and a number of other major systems (e.g., tomcat, cocoon). Responsibilities Build a prototype secure data computing environment using public utility computing (as provided by the Amazon Elastic Computing Cloud's EC2) at the Inter University Consortium for Social and Political Research (ICPSR) that will provision an analytic computing instance that conforms to the underlying security requirements for data distributed under restricted use agreements and meets the analytic needs of end users and their research teams. Test the performance, security and usability of both the provisioning infrastructure and the analytic computing interface. Standard methods of testing system performance and security will be used as well as independent security assessments through white hat hacking. This position reports to the Assistant Director, Computer & Network Services, ICPSR, but projects will be assigned and priorities designated by the Project Principal Investigator. Note: Part of this job may require some work outside normal working hours to analyze and correct critical problems that arise in ICPSR's 24 hours per day operational environment. Duties -Works with technical staff to design, implement, and support cloud based and/or virtualized computing platforms for both internal and external users. -Creates automated process requiring little manual input for the creation of virtualized computer instances, user accounts, and data access. -Analyzes, proposes and designs the implementation of security interfaces in systems, applications, and network software. -Participates in the evaluation of proposed systems, applications, and network software to determine security, data integrity, and usability implications. Assess risks to data security and identify countermeasures, plan and implement technologies. -Provides third-level technical support for desktop and network systems, both virtualized and non-virtualized.
Job Requirements:	-Bachelor's degree in computer science, information systems, or equivalent combination of education and experience. -Experience with Cloud computing and provisioning (preferably with Amazon Elastic Computing Cloud). - 4+ years experience with collecting and documenting business requirements from users; then researching, designing, implementing, and supporting computing systems to meet those requirements. - 5+ years experience and expertise with installing, configuring, and programming Windows software in both virtualized and non-virtualized settings. -Experience and expertise in industry security training, such as SANS GIAC, or have work experience in security consulting or network security. -Experience with social science concepts, social science data, and analysis methods, and statistical applications (SAS, SPSS, etc) preferred. -Experience with Wise Package Studio or other MSI-packaging software preferred. -Ability to explain complex technical concepts to non-technical users and stakeholders. -Excellent customer service skills and customer-oriented focus. -Attentiveness to detail. -Excellent writing skills (writing samples will be required) -Ability to work independently while meeting deadlines, communicating issues, and providing detailed project status updates. -Ability to work within a team.

TRAC: C1.2: Backup infrastructure

C1.2 Repository ensures that it has adequate hardware and software support for backup functionality sufficient for the repository’s services and for the data held, e.g., metadata associated with access controls, repository main content.

The repository needs to be able to demonstrate the adequacy of the processes, hardware and software for its backup systems. Some will need much more elaborate backup plans than others.

Evidence: Documentation of what is being backed up and how often; audit log/inventory of backups; Trustworthy Repositories Audit & Certification: Criteria and Checklist validation of completed backups; disaster recovery plan—policy and documentation; “firedrills”—testing of backups; support contracts for hardware and software for backup mechanisms.

---

ICPSR has extensive documentation and infrastructure to support its core access functions even when a catastrophic failure disables its primary location in Ann Arbor, Michigan. The documentation - planning documents and instructions - reside in a Google Group, and all members of the IT team, and two of ICPSR's senior staff outside of IT are members of the group. The process has been used twice in 2009, once as a test, and once when the Ann Arbor site suffered a power failure.

ICPSR has a less well documented, but fairly prosaic, backup solution in place. All non-ephemeral content at ICPSR resides on a large Network Attached Storage (NAS) appliance. The IT team has configured the NAS to "checkpoint" each filesystem once per day, and each checkpoint is retained for 30 days. Checkpoints provide a read-only, self-serve backup system for those instances where a member of the staff has inadvertently damaged or destroyed non-archival content.

Further, we write all filesystems to a tape library, which is located in a different machine room than the NAS. Every two weeks tapes are removed from the tape library, and stored in yet a different building. We retain the last four weekly backups, and the last twelve monthly backups. The system is exercised on an infrequent, but regular basis when we restore files that were damaged or destroyed beyond the thirty day checkpoint window.

Finally, unlike "working" files where all copies reside locally, and where we retain only one year of content, our archival storage solution consists of copies in at least four locations. The master copy (1) is on the NAS; a copy (2) is written to tape each month; a copy (3) is synchronized daily with the San Diego Supercomputer Center's storage grid; and, a copy (4) is synchronized daily with the MATRIX Center at Michigan State University. Furthermore, archival content collected prior to 2009 has also been copied into the Chronopolis project storage grid, which adds two additional copies.

One area with room for improvement would be regular "fire drills" where we would attempt to retrieve a random number of random objects from an arbitrarily selected archival storage location.

DuraCloud

I attended a webinar on DuraSpace last Wednesday. As a big fan of "the cloud" I was very interested to hear about what's been built, how it could be used, and a roadmap of the future. I learned a little bit about all three topics on the webinar.

Gina Jones from the Library of Congress hosted the meeting, and the main speaker was Michele Kimpton.

DuraCloud is being built as an OSGi container sitting on top of cloud storage providers. Customers can view DuraCloud as a buying club for lower prices, and for easing the burden of learning the administrative and software interfaces of each cloud provider.

DuraCloud is starting a pilot project with four cloud providers: (1) Amazon, (2) EMC, (3) Rackspace, and (4) Sun. They are also working actively to add Microsoft as a fifth cloud provider. They have two content providers signed up for the pilot: the New York Public Library, and the Biodiversity Heritage Library.

The NYPL has 800k objects and 50TB of content. They'd like to use DuraSpace to make a copy of their materials, and to transform content from TIFF format to JPEG2000. The JPEG2000 images would then be pulled back out of the cloud to local storage at the NYPL.

The BHL has 40TB of content, and is hoping to use DuraCloud to distribute its content across multiple locations (US, EU), and as a platform for hosting computational intensive data mining.

The pilot is running through the end of the calendar year, and DuraSpace intends to have a pricing model in place by Q2 2010, and to launch a production service in Q3.

In response to a question from a participant, Michele indicated that the focus was NOT on securing sensitive data, but rather on hosting public data with open access. So DuraCloud might be a good bet for some of the content ICPSR delivers on its web site, for example, but not for medical records, confidential data, etc.

OR Meeting - Live Chat Session - Technology & Preservation at ICPSR

Next Wednesday I'll be participating in a "live chat" session with Nancy McGovern. I thought I might seed the technology portion of the conversation with these topics:

1. Fedora Repository software
   
2. OpenID authentication for the ICPSR Web site
   
3. Cloud services from Amazon Web Services

Since we only have 30 minutes, and since the agenda is for both technology and preservation, I don't want to include too many topics.

Anything important I may have left out?

Teaching With Data Launched

ICPSR launched a new portal this week - TeachingWithData. This is part of ICPSR's grant to build a National Science Digital Library (NSDL) Pathway dedicated to social science data and teaching. I won't say too much more about the content in this post, but will instead focus on our technology selection.

The two main technical requirements were: (1) Fedora-based, and (2) fast deployment. We think we've achieved both with the site.

We implemented the site on an Amazon Web Services (AWS) Elastic Computing Cloud (EC2) instance; this allowed us to stand-up a base platform very quickly. We now have nearly a dozen instances running in AWS. Some are delivering production services, like our search technology and this Pathway, and others are used for science experiments and front-ends for storage. We've found that it's more convenient to synchronize and organize content in AWS by sticking a Linux instance in front so that we can use tools like rsync. Using Simple Storage Service (S3) directly is less convenient.

After looking at a variety of stacks that sit on top of Fedora, we selected Muradora for three main reasons:

1. Open source software
2. Active development community
3. Best technology fit with other technology platforms in use at ICPSR

Selecting Muradora allowed us to stand up the portal relatively quickly, and other time we'll evaulate how well the system is meeting our needs.

Enclave in the Clouds blog

My colleagues and I have started a separate blog to report progress on our NIH Challenge Grant. It's called Enclave in the Clouds, and the first post is from Felicia LeClere, PI on the grant.