For crying out cloud it has been an interesting weekend

As some folks may know, ICPSR makes pretty extensive use of ICPSR's cloud service, and so it has been interesting, and even a little painful, to watch the recent multi-day outage play out.

The timing of the outage wasn't that bad for us.  To the extent we use the Amazon cloud to perform research as part of our NIH "cloud grant" losing access over a long weekend wasn't that bad.  At worst we lost one day (Friday) and one afternoon (Thursday) where we wouldn't have been able to do everything we might have wanted.

We also got lucky that our production web site didn't suffer any problems - and wasn't scheduled for any maintenance - this weekend.  We run our replica system in Amazon's cloud, and had we lost the main site here at ICPSR, we would have been in very bad shape.

We also keep an encrypted copy of our holdings in Amazon's cloud, and over the past few days we haven't been able to keep it sync'd with master copies from here.  But since we have so many copies in so many different locations, this wasn't all that worrisome.  In fact, if we synchronized content weekly instead of daily, and if we always performed the synchronization on, say, Wednesday, the outage would have been a non-issue for this purpose.  (But, of course, if the weekly synchronization was performed on Saturdays instead of Wednesday, we'd be drifting even more out of sync.)

The primary pain point was with a production web site we run exclusively in the cloud, TeachingWithData.org.  For all practical purposes this site was off-line from Friday morning (EDT) until the middle of the day on Saturday.  That's not good.  But it's also the case that the service was never designed for 24 x 7 production, and so it's not surprising that it could suffer an extended outage.  Building in fault-tolerance costs more.

The biggest lesson for us is that we need to make sure that some of our oldest, most long-lived instances are moved to EBS-backed instances.  This is something that we've been meaning to do for some time, and this serves as a reminder of why it would be good to make it a higher priority.

March deposits at ICPSR

March 2011 deposits at ICPSR:

# of files	# of deposits	File format
56	20	application/msword
9	1	application/octet-stream
114	28	application/pdf
49	11	application/vnd.ms-excel
30	6	application/x-sas
31	15	application/x-spss
3	2	application/x-stata
2	2	application/x-zip
9	3	text/plain; charset=iso-8859-1
5	3	text/plain; charset=unknown
231	18	text/plain; charset=us-ascii
6	3	text/rtf
1	1	text/x-c; charset=us-ascii

Nothing too unusual here.  Lots of plain-text, SPSS, SAS, PDF, and MS Office formats (like Word and Excel).  Still need to update our system so that we bucket purported C source code files (text/x-c) as plain-text instead.

TRAC: B3.4: Demonstrating effctive preservation planning

B3.4 Repository can provide evidence of the effectiveness of its preservation planning.

The repository should be able to demonstrate the continued preservation, including understandability, of its holdings over a number of years, given the age of the repository and its holdings.

This could be evaluated at a number of degrees and depends on the specificity of the designated community(ies). If a designated community is fairly broad, an auditor could represent the test subject in the evaluation. More specific designated communities could require significant efforts. If judgment must be exercised as to whether adequate efforts have been made, it must be justified in detail.

Evidence: Collection of appropriate preservation metadata; proof of usability of randomly selected digital objects held within the system; demonstrable track record for retaining usable digital objects over time.

---

There are a couple of different stories we might tell related to this TRAC requirement.

One would be the "proof is in the pudding" story where we might assert that, of course, ICPSR has effective preservation planning:  We still preserve and distribute content that we originally collected decades ago.  In fact, if you spend enough time wandering around the less traveled portions of our collections, you'll likely find content which is very old, but which is still reasonably useful.  (It might be lacking formats for the modern stats packages, and possibly even setups, but there will always be documentation.)  This speaks to the "track record" reference above.

Another possible answer is to point to the regular audits we perform each week where we check the digital signature of each item on disk to the signature in a database.  This allows us to spot problems quickly, and that is a really important factor in allowing us to do a good job in managing problems.  This doesn't really prove usability per se, but it does demonstrate that the content (plain text data and its documentation) has not become corrupted.

And yet another possible answer is to address the collection of appropriate preservation metadata.  Each item (file) has a unique ID (reference); a digital signature (fixity); co-location with its related content (context); and, a MIME type (representation).  And for all of our newer content post-dating the deposit system, we have very good provenance information.  (Older stuff is more of a mixed bag of good provenance information, especially prior to the 1990's.)

Is it April already?

Things are really humming at ICPSR these days.  I've been so busy with a new grant application, the recent Council meeting, a major infrastructure project, and more that I've been neglecting the blog.  I hope to take care of that oversight very soon, and plan to queue up several more posts tomorrow.  But for now...

I gave a talk at ICPSR yesterday on "the cloud."  I used Prezi for creating and playing the "slides," and you can find a link to a copy of it here.  And I will also embed it here:

Cloud Computing - ICPSR Lunch and Learn on Prezi

I used the iPad 2 to play the presentation, and I think it worked pretty well.  The Prezi iPad app does not support video or audio at this time, and that meant I could not include a few clips I had planned to use.  If the Prezi guys can fix this up, I think Prezi and the iPad will be my new main platform for giving talks.

TRAC: B3.3: Updating plans in response to monitoring

B3.3 Repository has mechanisms to change its preservation plans as a result of its monitoring activities.

The repository must demonstrate or describe how it reacts to information from monitoring, which sometimes requires a repository to change how it deals with the material it holds in unexpected ways. Plans as simple as migrating from format X to format Y when the registries show that format X is no longer supported are not sufficiently flexible—other events may have made format Y a bad choice. The repository must be prepared for changes in the external environment that may make its current plan (to migrate from X to Y in 10 years) a bad choice as the time to implement draws near. The repository should be able to show that it can revise long-range plans in light of changing circumstances.

Another possible response to information gathered by monitoring is for the repository to create additional Representation Information and/or PDI.

Evidence: Preservation planning policies tied to formal or information technology watch(es); preservation planning or processes that are timed to shorter intervals (e.g., not more than five years); proof of frequent preservation planning/policy updates.

---

My team at ICPSR supports this function, but is not directly responsible for preservation planning, so this post will be brief. 

My experience is that ICPSR has a very robust preservation planning function which engages regularly with a wide array of communities, and which therefore keeps a close eye on changes in what the community needs, or changes that are necessitated by technology.

I suppose the best evidence for this might be the major change we undertook a few years ago to move all digital content from a variety of legacy formats to "spinning disk."  This was a major project, of course, but allowed ICPSR to stop worrying about the availability of legacy media readers and media which could fail without quick detection.

February Deposits at ICPSR

February 2011 deposits (and their file formats) at ICPSR:

# of files	# of deposits	File format
1	1	application/msoffice
284	23	application/msword
11	4	application/octet-stream
104	25	application/pdf
7	5	application/vnd.ms-excel
24	10	application/x-sas
80	28	application/x-spss
26	4	application/x-stata
2	1	application/x-stuffit
2	2	application/x-zip
19	6	message/rfc8220117bit
15	3	text/html
9	2	text/plain; charset=iso-8859-1
2	2	text/plain; charset=unknown
122	21	text/plain; charset=us-ascii
9	5	text/rtf
1	1	text/x-c; charset=iso-8859-1
22	7	text/x-c; charset=us-ascii
1	1	text/xml

Nothing too exciting this month. 

[ I thought I had posted this weeks ago, but clearly not. ]

ICPSR Machine Room maintenance

I don't see an announcement about this on the main web site, so I'll drop a short note here....

The University of Michigan Plant Operations department is performing some electrical work in our machine room starting at 8am this morning.  The work will last some unknown number of hours; my guess is 4-6.

We have just diverted traffic from our production web site over to the replica in the cloud.  Our most common services such as search, download, and on-line analysis will be available.  But services which require our web site to "remember" something -- such as data deposit, Summer Program registration -- are not enabled in the replica.

I'll post a follow-up once the work has completed, and will check back here periodically for comments.  If you see something that isn't working as you think it should, please comment here, and we'll take a look.

----

10:22am EDT : We're aware that study search is not working.  (Variable search and bibliography search are working.)  ICPSR recently updated the search to include the full-text of many types of documents (codebooks, survey instruments, user guides, etc), and the index location changed too.  This change was not reflected on the cloud replica, and the index there is now building.  Our apologies for the inconvenience this may cause.

ETA for conclusion of the electrical work is between 11am and noon EDT.

-----

11:18am EDT: The power work has been completed.  We've restored service to the main site.