ICPSR and the cloud

How is ICPSR using "the cloud?"

I've been getting this question a lot lately, and it feels like it's time to put together a blog post on this question.

From a functional standpoint ICPSR is using the cloud for identity and authentication, content delivery, archival storage, and data producer relationship management.  And if I include services based at the University of Michigan, I might also include data curation, and customer relationship management.

From a vendor standpoint here's a roster of some of the organizations with whom we're doing business, and how their piece of the cloud helps us run our business.

A typical transaction on the ICPSR web site looks like this:  Search.  Select content for download.  Create an ICPSR-specific identity.  Authenticate using that identity.  Download the content.  Do not return to ICPSR for at least a year.

Given that the ICPSR-specific identities are weak (i.e., web site visitors create them by entering an arbitrary email address and password) and given that they identity is often used only once, it seemed like a good idea to eliminate the need to create such an identity.  We don't need strong identities, but we do need identities that would be available to anyone.  Technologies like OpenID, Facebook Connect, and the like seemed promising, but who wants to build infrastructure which talks to all of them?

Janrain does.

We use Janrain Engage as one part of our identity and authentication strategy.  Janrain acts as a third party between the content provider (ICPSR) and the identity providers.  And so when someone needs to log in to ICPSR's portal, they see a screen that looks something like this:

So there's no need to create an account and password at ICPSR.  And if someone does return later, they don't have to log in to our site if they've already logged in to their identity provider's site.  (This is Single Sign-On or SSO.)

We're hosting several web portals in Amazon's cloud.  We're using Amazon's Infrastructure as a Service (IaaS) to stand-up Linux systems in the Amazon Elastic Computing Cloud (EC2) that are identical to the images we host locally.  We back the instances with Elastic Block Storage (EBS) volumes so that the content persists when we need to terminate and restart a computing instance. 

We also host a replica of our on-site delivery system in Amazon's cloud for disaster recovery (DR) purposes.  We find that we have the opportunity to "test" this replica at least once per year when ICPSR's headquarters loses power for several hours due to high winds, ice storms, or other acts of nature.

The Amazon service has been very reliable overall (despite a few highly publicized events), and certainly more reliable than our own on-site facilities.  We also like that we can scale resources up and down very quickly, and that we have clear costs associated with the infrastructure.  (Anyone at an institution of higher learning who has tried to calculate the actual cost of electricity used knows what I mean.)

I've posted many times about our relationship with DuraCloud, and how we're using it as a mechanism for storing archival copies in the cloud.  In many ways DuraCloud fulfills a role similar to that of Janrain Engage by providing a layer of abstraction between ICPSR's technical infrastructure and that of multiple service providers.  In this case we manage one vendor and one set of bills, but have the ability to store content in the cloud storage service of multiple providers (Amazon, Rackspace, Microsoft).

The acquisitions team at ICPSR keeps an eye on grants funded by places like the National Science Foundation and the National Institutes of Health.  If a grant looks like it may be producing data the team makes a note to contact the primary investigator (PI).  The goal is to have a conversation with the PI to see if there will indeed be data produced, and to see if it might be a good fit for ICPSR's holdings.  If so, we then try to convince the PI that depositing the content with ICPSR would be good for everyone (more data citations for the data producer; more re-use of the data by other researchers; etc.).

We had been using a home-built application to manage this content, but we found it to be a losing battle.  There was never enough money or time to build the types of relationship management reporting systems that the acquisition team wanted.  And so rather than trying to build a better mousetrap, we decided to rent a better mousetrap by moving the content into a professional contact/customer relationship management (CRM) system.  Like Salesforce.

 The University of Michigan central IT organization (ITS)  also delivers a handful of services that I would consider "the cloud" even though they do not package and market them that way.  File storage, trouble ticketing, and Drupal-hosting are all available from ITS, and they all look like cloud services to us because we pay for only what we use, we can scale them up and down reasonably quickly, and we do not have to deploy any local hardware or software to use them.

Hi Ho, a Googling we will go!

The University of Michigan announced (on Halloween! - I hope this is not a trick!) that it will be adopting Google as its collaboration platform.  The roll-out will happen over the course of the next year, and includes tools such as Gmail, Sites, Docs, Calendar, Blogger, and more.

I am delighted.

I've been using Google's Blogger technology (obviously) for some time to publish the Tech@ICPSR blog, and use Google Docs for almost any project where I would have used Microsoft Office in the past.  (I do still use PowerPoint from time-to-time if I need something fancy-schmancy, and don't have the time to conceptualize it as a Prezi instead.)

The biggest win for ICPSR, however, is with Gmail and Calendar.

When I arrived at ICPSR in 2002 we were running our own IMAP-based service with Eudora as the supported client.  And by supported I mean that we installed the free "hey look at these ads" version on each person's machine.  Off-site access was the responsibility of the individual, although we did hook it up to a campus webmail front-end eventually.  We were running MeetingMaker as our supported calendar client.  And by supported I mean that we installed the client on everyone's machine, but no one used it.

Sometime in 2005 or so we realized that it wasn't much fun running email and calendar services, and we also noted that we were already paying for an enterprise mail/calendar system that our parent organization, the Institute for Social Research (ISR), operated on the Exchange platform.  And so we dumped Eudroa and MeetingMaker and started using the Microsoft stack instead.

I was delighted.

However......

I soon experienced the harsh realities of life in the Microsoft stack.  Small mailbox quotas.  Feature-poor webmail experience.  Mailbox "archives" living in one-off files on my PC or file server.  And have you ever tried to find the full email headers in a piece of email stored on an Exchange server?  And like our days of running Eudora and MeetingMaker we continued to be isolated from the rest of campus since our Exchange system was local to ISR and not part of a campus-wide solution.

The honeymoon had ended.

I solved the problem for myself (sort of) by maintaining my "internal to ISR" meetings and email on the ISR Exchange server, but moving my "external" meetings and email to Google.  That is, I changed the U-M address book so that my bryan (at) umich.edu email address was routed to Gmail rather than the Exchange server.  And so when it comes to communicating with the world outside of the ISR, I have a rich email experience that works well in any web browser, superb mail searching, and despite not deleting a single piece of non-spam email in nearly three years, I have used less than 20% of my mail quota.  At this rate, I will need to delete my first email in 2024.  Nice.  Of course, the problem is that I now check email and calendars in two places:  MS Exchange (for my ISR world) and Google (for everything else).

And so I am looking forward to the day in 2012 when it all dovetails back together and there is just one place to check my mailbox and calendar again.

TRAC: A3.2: Written policies and procedures

A3.2 Repository has procedures and policies in place, and mechanisms for their review, update, and development as the repository grows and as technology and community practice evolve.

The policies and procedures of the repository must be complete, written or available in a tangible form, remain current, and must evolve to reflect changes in requirements and practice. The repository must demonstrate that a policy and procedure audit and maintenance is in place and regularly applied. Policies and procedures should address core areas, including, for example, transfer requirements, submission, quality control, storage management, disaster planning, metadata management, access, rights management, preservation strategies, staffing, and security. High-level documents should make organizational commitments and intents clear. Lower-level documents should make day-to-day practice and procedure clear. Versions of these documents must be well managed by the repository (e.g., outdated versions are clearly identified or maintained offline) and qualified staff and peers must be involved in reviewing, updating, and extending these documents. The repository should be able to document the results of monitoring for relevant developments; responsiveness to prevailing standards and practice, emerging requirements, and standards that are specific to the domain, if appropriate; and similar developments. The repository should be able to demonstrate that it has defined "comprehensive documentation" for the repository. See Appendix 3: Minimum Required Documents for more information.


Evidence: Written documentation in the form of policies, procedures, protocols, rules, manuals, handbooks, and workflows; specification of review cycle for documentation; documentation detailing review, update, and development mechanisms. If documentation is embedded in system logic, functionality should demonstrate the implementation of policies and procedures. 

---

A deliverable for a recent contract was something the client called an information system security plan.  Our understanding was that in past contracts this was always understood to be a short document (2-3 pages) that summarized ICPSR's IT systems, and described the measures taken by ICPSR to protect them from unauthorized use.  No big deal, right?

However, .....

In this most recent contract the security plan implementation details changed; rather than a brief summary document, the requirement was now two-fold.

The first deliverable consisted of a document showing the Federal Information Processing Standards (FIPS) categorization of the risks associated with our IT systems.  This document was based on a standard known as FIPS Publication 199.  It turns out that this methodology and level of documentation is relatively lightweight.

In brief, one asserts one of three levels (Low, Medium, High) of risk.  There was never a question of asserting High risk, and so the choice as to select either Low or Medium.  We worked with the University of Michigan's central IT security office, and based on the type of data preserved at ICPSR, they recommended that we select Low.

The second part required us to document the security controls defined by the National Institute of Standards and Technology related to a FIPS-199 categorization of Low risk.  This standard is described in NIST Special Publication 800-53, and requires a very high level of documentation.  (The standard is very heavy on policy and documentation, but very light on measurement and audit, and therefore some critics believe that this is a major flaw in the approach.)

Our NIST 800-53 security control documentation ran nearly 200 pages(!), and this page-count does not include documents which are required, but external, to 800-53.  For example, if 800-53 requires one to assert that there is a policy on topic X, one does not need to include the policy within the 800-53 security controls documentation, but it does require one to write the policy on topic X (if it does not already exist).  And so between the 800-53 controls and the external documents, our guess is that this ran well over 250 pages.

And so we are very well supplied with policies and procedures, and we even have the documentation to prove it now.

InfoWorld Geek IQ Test - 2011

I took the 2011 InfoWorld geek IQ test.  I knew the answers to some of the more techie questions, especially when they were related to networking (CIDR, DNS), but didn't do so well on the pop culture items.  Got a 65 which places me between Geek dilettante and Marketing Executive.

I haven't decided yet whether I'm happy or ashamed.

Firing clients

Seth Godin has published another gem:  The Unreasonable Customer.

In this post he argues that while there are certain circumstances where maintaining a relationship with an unreasonable customer is justified, in many cases it makes no sense.  This is spot-on advice.

Some clients are demanding, of course, but some are demanding in very constructive, very actionable ways.  The client who pushes ICPSR, say, to deliver content in more interesting, more innovative ways may be difficult, but ultimately makes ICPSR a stronger organization with better services.

But the client who makes demands which are unreasonable, and which take resources away from better serving the other clients only weakens the organization.  Instead of making services better, the organization struggles hopelessly to appease the unreasonable client.  Resources and time are lost.  Staff become exhausted and disillusioned.  Morale sinks.

In the olden days of working in the telecom industry in the mid 1990s I remember a case where a train had derailed and it had torn up a bunch of fiber near the Washington, DC area.  A handful of our clients had consequently lost their network connections.  Our company was doing the right things:  We informed the clients about the problem, and we were keeping close watch on the fiber restoration project, pushing the supplier (I think it might have been MCI) to give our circuits the top priority.  While no one was delighted to be without their Internet connection, they understood that the cause was beyond our control, and that they had made the decision to purchase only a single Internet connection from a single company.  (Clients who needed very, very high availability would routinely purchase multiple Internet connections from multiple providers.)

One client, however, refused to let the team work through the problem.  This client wasn't interested in service restoration; this client wanted to take out all of his frustration on the team.  "You're incompetent!"  "You should all be fired!"  "This is unacceptable!"

I tried to calm the client.  Maybe we could set up something short-term over a dial-up line?  And maybe long-term the right solution is to have more than one Internet connection so that if another train derails (this seemed to happen way more than one would expect) or there is a natural disaster, you'll still have your Internet connectivity?

Nothing worked.  It was clear that this one client didn't want help; he wanted a punching bag.

So we fired him.

"You're right.  It sounds like we're just not the right provider for you.  We can't meet your expectations.  We won't waste any more of your time trying to restore your service.  We'll need you to send back the router, or we will have to bill you for it.  Best wishes, and good luck with your next provider."

That did more for morale than the last ten company picnics and holiday parties combined.

I honestly don't remember if we did end up firing the client, or if just the threat ended his hysterics.  But it definitely changed the relationship, and it proved to the team that we wouldn't let unreasonable people stop them from doing good work.

TRAC: A3.1: Designated community

A3.1 Repository has defined its designated community(ies) and associated knowledge base(s) and has publicly accessible definitions and policies in place to dictate how its preservation service requirements will be met. 

The definition of the designated community(ies) (producer and user community) is arrived at through the planning processes used to create the repository and define its services. The definition will be drawn from various sources ranging from market research to service-level agreements for producers to the mission or scope of the institution within which the repository is embedded.

Meeting the needs of the designated community—the expected understandability of the information, not just access to it—will affect the digital object management, as well as the technical infrastructure of the overall repository. For appropriate long-term planning, the repository or organization must understand and institute policies to support these needs.

For a given submission of information, the repository must make clear the operational definition of understandability that is associated with the corresponding designated community(ies). The designated community(ies) may vary from one submission to another, as may the definition of understandability that establishes the repository’s responsibility in this area. This may range from no responsibility, if bits are only to be preserved, to the maintenance of a particular level of use, if understanding by the members of the designated community(ies) is determined outside the repository, to a responsibility for ensuring a given level of designated community(ies) human understanding, requiring appropriate Representation Information.

The documentation of understandability will typically include a definition of the applications the designated community(ies) will use with the information, possibly after transformation by repository services. For example, if a designated community is defined as readers of English with access to widely available document rendering tools, and if this definition is clearly associated with a given set of Content Information and Preservation Description Information, then the requirement is met.

Examples of designated community definitions include:

• General English-reading public educated to high school and above, with access to a Web Browser (HTML 4.0 capable).
• For GIS data: GIS researchers—undergraduates and above—having an understanding of the concepts of Geographic data and having access to current (2005, USA) GIS tools/computer software, e.g., ArcInfo (2005).
• Astronomer (undergraduate and above) with access to FITS software such as FITSIO, familiar with astronomical spectrographic instruments.
• Student of Middle English with an understanding of TEI encoding and access to an XML rendering environment.
• Variant 1: Cannot understand TEI
• Variant 2: Cannot understand TEI and no access to XML rendering environment
• Variant 3: No understanding of Middle English but does understand TEI and XML
• Two groups: the publishers of scholarly journals and their readers, each of whom have different rights to access material and different services offered to them.

Evidence: Mission statement; written definitions of the designated community(ies); documented policies; service-level agreements. 

---

Documentation for this TRAC requirement can be found in ICPSR's mission statement (published on our web portal) and in our deposit agreements.

DuraCloud Archiving and Preservation Webinar

Shameless self-promotion alert...

The nice folks at DuraSpace have published the audio and video from the recent webinar that Michele Kimpton (CEO, DuraSpace) and I gave on DuraCloud.

DuraCloud Archiving & Preservation Webinar

View more webinars from DuraSpace

Michele spends the first 5-10 minutes talking about the business case behind DuraCloud, and then I spend about 30 minutes talking about ICPSR and how we came to use DuraCloud to store a copy of our archival holdings.