Provenance metadata and the OAIS Receive Submission

The FLAME (File-Level Archival Management Engine) project continues to articulate functional requirements for the software system.  So far the process has looked something like this:

1. Select one of the functional areas of OAIS
2. Drill down into one of the sub-functions within that area
3. Enumerate a list of high-level statements that should be true for that sub-function
4. Translate those high-level statements into medium-level specifications for the software

For example, we tackled one such area before the recent holiday break:

1. Ingest
2. Receive Submission
3. The producer provided basic provenance information at deposit

Of course, this raises the question:  What do we consider "basic provenance information" at the time of deposit?  What information can we collect from the deposited content, and what information do we need to collect from the person performing the deposit?

Here is the draft list we created:

a. FLAME should enable the ability to transfer digital content to ICPSR through web-based file upload

i. Uploading files should NOT require MyData authentication
ii. The act of uploading files serves as a signature for the transfer

b. FLAME should enable the ability to document receipt of content transferred to ICPSR through non-electronic means

i. Date package arrived (required)
ii. Shipping company (required)
iii. Tracking ID number (required)
iv. Other details about the shipment (optional)

c. FLAME should capture the following provenance information from the content provider:

i. Self-reported identity of the content provider, or identity from MyData profile (for electronic transfer)

1. Name of depositor (required)

ii. Self-reported contact information for the depositor, or from MyData profile

1. Email-address (required)
2. Telephone number (optional)
3. Mailing address (optional)

iii. Self-reported descriptive provenance information from the depositor

1. Name or title of deposit (required)
2. Summary or description of the deposit (optional)
3. Name of organization that sponsored the research, or "not applicable" (required)
4. Number of ID of the grant or contract, or "not applicable" (required)

d. FLAME should capture the following provenance information from the files after each content transfer:

i. Date and time at which each file is received
ii. Checksum of each file
iii. MIME type of each file
iv. Original name of each file
v. Packaging information (e.g., file was part of a Zip archive)

What do you think basic provenance information should include?  Does our list look like it captures everything one could reasonably expect to collect at the time of deposit?

TRAC: A3.9: Self-assessment and certification

A3.9 Repository commits to a regular schedule of self-assessment and certification and, if certified, commits to notifying certifying bodies of operational changes that will change or nullify its certification status.

A repository cannot self-certify because an objective, external measurement using a consistent and repeatable certification process is needed to ensure and demonstrate that the repository meets and will likely continue to meet preservation requirements. Therefore, certification is the best indicator that the repository meets its requirements, fulfills its role, and adheres to appropriate standards. The repository must demonstrate that it integrates certification preparation and response into its operations and planning.

Evidence: Completed, dated audit checklists from self-assessment or objective audit; certificates awarded for certification; presence in a certification register (when available); timetable or budget allocation for future certification. 

---

Like a few of the other A-group TRAC requirements, this one really operates at the uppermost level of the organization, and so it is difficult to address it from the IT perspective.

HOWEVA..... One barrier to implementing a regular certification cycle are some fundamental questions:

Where do I find a list of consultants or analysts that can grant "TRAC certification" to my repository? 

Which organization sanctions those consultants and analysts? 

What does it mean - precisely - to be "TRAC certified?" 

Are there different levels of TRAC certification, much like FISMA levels? 

If I'm already FISMA certified, does that automatically grant TRAC certification for certain items (especially in section C)?

And so on.

It seems like there is a business opportunity here.

For instance, if ICPSR asserted that it was now in the business of reviewing TRAC requirements for organizations, and a team of ICPSR analysts would either certify your data archive as TRAC compliant or would identify clear action items required to become compliant, would that be a useful thing?  Or would other organizations rise up to say, "Hey, who are you, ICPSR, to be granting certifications?"

How should this work?

December 2011 deposits at ICPSR

Chart.

# of files	# of deposits	File format
98	13	application/msword
67	4	application/octet-stream
207	25	application/pdf
192	8	application/vnd.ms-excel
44	5	application/x-sas
86	13	application/x-spss
4	4	application/x-zip
2	1	image/jpeg
2	1	image/tiff
12	1	message/rfc8220117bit
7	7	text/html
23	2	text/plain; charset=iso-8859-1
69	5	text/plain; charset=unknown
468	13	text/plain; charset=us-ascii
7	2	text/rtf
1	1	text/x-c++; charset=us-ascii
1	1	text/x-c; charset=unknown
10	3	text/x-c; charset=us-ascii
4	1	text/xml

Lots of plain text and Excel this month, and not so much from the conventional stats packages.  The usual set of C and C++ bogons that are undoubtedly plain text.  And a large number of files where we could not identify the content (octet-stream) which tells me that we either received lots of binary data, or we are starting to see a new format that our MIME detector can't figure out.

ICPSR web availability through December 2012

Web availability in December was looking very, very good through most of the month.  We had seen only a single noteworthy event the entire month, and that resulted only in a few minutes of downtime.  (As happens from time to time, a member site was scraping our web pages, presumably to collect the metadata we publish.  And while professional scrapers like Google, Yahoo, and the other search engines scrape gently and non-intrusively, this is not often the case with less experienced scrapers.)

Of course, December is always a tricky month here at ICPSR.  Snow storms.  Ice storms.  Power outages.  I can't remember the last time that my entire team was able to take off the entire week between Christmas and New Years (like the rest of the U-M) without having to come into the office to troubleshoot a problem.

And this year was no different.

We started to see sporadic up/down alerts from the U-M network monitoring system on the morning of December 30.  It looked like our production web server was working OK overall, but having some problems. When we tried to load the home page from home, the page wouldn't load.  And when we tried to login (via ssh) from home, the connection timed out.  It looked as if everything was down even though the monitoring system said it was OK.

We found we could log into other systems on campus, and then use those as a launch pad to get to ICPSR.  All of our systems were up, but none seemed reachable from systems off campus.  This explained why the U-M monitoring system didn't through more alarms earlier.

Then we noticed this:

http://status.its.umich.edu/outage.php?id=73300

(I think this link works even from off-campus.)

We then worked with the campus network engineers to draw their attention to the problem that was affecting us.  Unfortunately it was kind of helpful to have the ICPSR web site be unavailable from off-campus as a test case; we would know the network was fixed when the web site was available again.

All in all not a horrible month for availability, but we moved from 99.9% on Dec 29 to 99.5% by the end of Dec 30.

TRAC: A3.8: Information integrity measurements

A3.8 Repository commits to defining, collecting, tracking, and providing, on demand, its information integrity measurements. 

The repository must develop or adapt appropriate measures for ensuring the integrity of its holdings. The mechanisms to measure integrity will evolve as technology evolves, but currently include examples such as the use of checksums at ingest and throughout the preservation process. The chain of custody for all of its digital content from the point of deposit forward must be explicit, complete, correct, and current. The repository must demonstrate that the content it has matches the content it received, e.g., with an implemented registry function that documents content from submission onward. Losses associated with migration and other preservation actions should also be documented and made available to relevant stakeholders. (See C1.5 and C1.6.)

If protocols, rules, and mechanisms are embedded in the repository software, there should be some way to demonstrate the implementation of integrity measurements.

Evidence: An implemented registry system; a definition of the repository’s integrity measurements; documentation of the procedures and mechanisms for integrity measurements; an audit system for collecting, tracking, and presenting integrity measurements; procedures for responding to results of integrity measurements that indicate digital content is at risk; policy and workflow documentation. 

---

ICPSR operates very differently than a conventional archive, and it really shows when one looks at this TRAC requirement.

A typical workflow for us looks like this:

1. Receive some content in formats like SAS and Word
2. Preserve that content "as is" at the bit-level
3. Completely re-do all of the data and documentation, preserving the intellectual content (modulo disclosure concerns), but reorganizing it all
4. Produce normalized and ready-to-use content based on the re-do
5. Preserve the normalized content forever

So at the file-level we track all of the original deposits and all of the content we produce, and we test the integrity of each file every week.  Since my team inherited the responsibility to manage archival storage n 2006 I've never seen a problem that wasn't traced back to a transient error that took place as content was being copied into archival storage, and where the solution wasn't solved when the ICPSR staff member re-ran the copy.

We also track the chain of custody at the aggregate level, assigning each "deposit" and "study" to both a workgroup and an individual, and by linking deposits to studies (and vice-versa).  We have internal systems to manage both deposits and studies, and they include mechanisms whereby a data manager can edit metadata, assign key dates, and enter diary entries, not unlike a trouble ticket or help desk system.

Systems Architect Senior job posting @ ICPSR

We've posted another job opening on my team at ICPSR.  I'll include the text from the job description below, but the short-lived link to the U-M job site is http://umjobs.org/job_detail/65035/systems_architect_senior.

In brief we're looking for someone with deep experience building production operational environments for web applications who can apply those skills to the ICPSR environment.  We currently have a mix of stuff running on real hardware at ICPSR and virtual hardware in Amazon's EC2, and we have a mix of legacy Perl CGI code and newer Java-based web applications.  A mix of myself and Steve Burling (who retires at the end of January) have been the main architects of the environment, but it has grown so much over the past 5-10 years that it has become a full-time job.

Here are the details:

Systems Architect Senior

Job Summary

The Inter-university Consortium for Political and Social Research (ICPSR), the world's largest archive of digital social science data, is now accepting applications for a Systems Architect Senior. ICPSR is a unit within the Institute for Social Research at the University of Michigan. ICPSR's data are the foundation for thousands of research articles, reports, and books. Findings from these data are put to use by scholars, policy analysts, policy makers, the media, and the public. This position reports to the Assistant Director, Technology Operations, Computer and Network Services.

Responsibilities*

Responsibilities
This position is responsible for the design, implementation, maintenance, and regular management of ICPSRs web systems development, staging, production, and disaster recovery operational environments. This consists of several distinct platforms, including local physical hardware and virtual systems hosted in Amazons Elastic Computing Cloud (EC2). The successful candidate will also work closely with the Assistant Director, Software Development, Computer and Network Services to define and implement functional requirements.

One, this position will select, install, and manage integrated development environment (IDE) software on developer workstations, and the underlying software repository. The incumbent systems are Eclipse and CVS, respectively.

Two, this position will select, install, manage, and maintain the testing, staging, and production platform environments used by ICPSR to deploy and test new web applications. The incumbent web application server is Apache Tomcat, sometimes run as a stand-alone web server and sometimes as a client to Apache Httpd. The incumbent server platform is a mix of local, physical servers and Elastic Computing Cloud (EC2) instances running within Amazon Web Services (AWS). ICPSR has interest in exploring a more complete. cloud-based web application platform such as AWS Elastic BeanStalk.

Three, this position will manage the AWS-hosted replica of ICPSRs production web environment. This includes building and maintaining tools that synchronize software, static content, and database content between the production environment and the replica environment.

Four, the over-arching responsibility of this position is to maintain and improve ICPSRs capacity for delivering high-availability, high-performance web-based services, managing the tension between the desire to have well-defined, documented, predictable deployments and business processes and the desire to have fast moving, fluid, and flexible deployments and business processes.

Required Qualifications*

BS in Computer Science, Computer Engineering, or at least eight years of experience with designing and managing complex web application hosting environments
Two or more years of experience with J2EE application servers (such as Tomcat)
Two or more years of experience with virtualization products or services, such as Amazon Web Services
In-depth expertise with RedHat Enterprise Linux 5 and 6
In-depth knowledge of networking principles and network support
In-depth knowledge of Web technologies (Apache, Tomcat)
Experience operating an RDBMS (Oracle, MySQL)
Experience working with monitoring tools, control software (CVS , Subversion, Perforce), and build tools (Make , Ant)
Strong understanding of web application architectures
Enthusiastic self-starter who works well with other team members
Excellent inter-personal skills with the ability to communicate clearly to peers, vendors, customers, and colleagues

Desired Qualifications*

MS in Computer Science or Computer Engineering
At least five years of experience with J2EE application servers (such as Tomcat)
Experience with Veritas products (Veritas Netbackup)
Storage experience (EMC)
Experience as an Infrastructure Engineer in a high availability environment
Expertise working with monitoring tools , version control software (CVS , Subversion, Perforce), and build tools (Make , Ant)

Underfill Statement

This position may be underfilled at a lower classification depending on the qualifications of the selected candidate.

U-M EEO/AA Statement

The University of Michigan is an equal opportunity/affirmative action employer.

Tech@ICPSR takes another holiday

Dear Loyal Readers:

We return to our normal antics later this week.

Signed,

Tech@ICPSR