Amazon Web Services makes Tech@ICPSR weep

June 2012 was looking to be a great, great month for uptime.  We were on track to have our best month since November 2011 this fiscal year - just 60 minutes of downtime across all services and all applications.  It was going to be beautiful.

And then Amazon Web Services had another power failure.

And then we wept.

The power failure took the TeachingWithData portal out of action.  (To be fair, it was already having significant problems due to its creaky technology platform, but this took it all the way out of action.)  The failure also took our delivery replica out of action, and gave Tech@ICPSR the joy of rebuilding it over the weekend.

But the real trouble was with a company called Janrain.

Janrain sells a service called Engage.  Engage is what allows content providers (like ICPSR) to use identity providers (like Google, Facebook, Yahoo, and many more) so that their clients (like you) do not need to create yet another account and password.  Engage is a hosted solution that we use for our single sign-on service using existing IDs, and it works 99.9% of the time.

However, this hosted solution lives in the cloud.  We just point the name signin.icpsr.umich.edu at an IP address we get from Janrain, plug in calls to their API, and then magic happens.

Except when the cloud breaks.

Amazon took Engage off-line for nearly four hours.  And then once it came back up, it was thoroughly confused for another three hours.  Ick.

So, counting all of that time as "downtime" our fabulous June 2012 numbers suddenly became our awful June 2012 numbers.  Here they are:

If you click on the image above, Blogger will make it bigger.

Of course, during a lot of that downtime, all of the features on the web site except for third-party login worked fine.  And most of the problem happened late on a Friday night and Saturday morning during the summer, so that's a good time for something bad to happen, if it has to happen at all.

ICPSR and the cloud

How is ICPSR using "the cloud?"

I've been getting this question a lot lately, and it feels like it's time to put together a blog post on this question.

From a functional standpoint ICPSR is using the cloud for identity and authentication, content delivery, archival storage, and data producer relationship management.  And if I include services based at the University of Michigan, I might also include data curation, and customer relationship management.

From a vendor standpoint here's a roster of some of the organizations with whom we're doing business, and how their piece of the cloud helps us run our business.

A typical transaction on the ICPSR web site looks like this:  Search.  Select content for download.  Create an ICPSR-specific identity.  Authenticate using that identity.  Download the content.  Do not return to ICPSR for at least a year.

Given that the ICPSR-specific identities are weak (i.e., web site visitors create them by entering an arbitrary email address and password) and given that they identity is often used only once, it seemed like a good idea to eliminate the need to create such an identity.  We don't need strong identities, but we do need identities that would be available to anyone.  Technologies like OpenID, Facebook Connect, and the like seemed promising, but who wants to build infrastructure which talks to all of them?

Janrain does.

We use Janrain Engage as one part of our identity and authentication strategy.  Janrain acts as a third party between the content provider (ICPSR) and the identity providers.  And so when someone needs to log in to ICPSR's portal, they see a screen that looks something like this:

So there's no need to create an account and password at ICPSR.  And if someone does return later, they don't have to log in to our site if they've already logged in to their identity provider's site.  (This is Single Sign-On or SSO.)

We're hosting several web portals in Amazon's cloud.  We're using Amazon's Infrastructure as a Service (IaaS) to stand-up Linux systems in the Amazon Elastic Computing Cloud (EC2) that are identical to the images we host locally.  We back the instances with Elastic Block Storage (EBS) volumes so that the content persists when we need to terminate and restart a computing instance. 

We also host a replica of our on-site delivery system in Amazon's cloud for disaster recovery (DR) purposes.  We find that we have the opportunity to "test" this replica at least once per year when ICPSR's headquarters loses power for several hours due to high winds, ice storms, or other acts of nature.

The Amazon service has been very reliable overall (despite a few highly publicized events), and certainly more reliable than our own on-site facilities.  We also like that we can scale resources up and down very quickly, and that we have clear costs associated with the infrastructure.  (Anyone at an institution of higher learning who has tried to calculate the actual cost of electricity used knows what I mean.)

I've posted many times about our relationship with DuraCloud, and how we're using it as a mechanism for storing archival copies in the cloud.  In many ways DuraCloud fulfills a role similar to that of Janrain Engage by providing a layer of abstraction between ICPSR's technical infrastructure and that of multiple service providers.  In this case we manage one vendor and one set of bills, but have the ability to store content in the cloud storage service of multiple providers (Amazon, Rackspace, Microsoft).

The acquisitions team at ICPSR keeps an eye on grants funded by places like the National Science Foundation and the National Institutes of Health.  If a grant looks like it may be producing data the team makes a note to contact the primary investigator (PI).  The goal is to have a conversation with the PI to see if there will indeed be data produced, and to see if it might be a good fit for ICPSR's holdings.  If so, we then try to convince the PI that depositing the content with ICPSR would be good for everyone (more data citations for the data producer; more re-use of the data by other researchers; etc.).

We had been using a home-built application to manage this content, but we found it to be a losing battle.  There was never enough money or time to build the types of relationship management reporting systems that the acquisition team wanted.  And so rather than trying to build a better mousetrap, we decided to rent a better mousetrap by moving the content into a professional contact/customer relationship management (CRM) system.  Like Salesforce.

 The University of Michigan central IT organization (ITS)  also delivers a handful of services that I would consider "the cloud" even though they do not package and market them that way.  File storage, trouble ticketing, and Drupal-hosting are all available from ITS, and they all look like cloud services to us because we pay for only what we use, we can scale them up and down reasonably quickly, and we do not have to deploy any local hardware or software to use them.

One week later - Facebook + Google v. MyData

ICPSR rolled out a new option for logging into our web site on May 18.  Instead of requiring web site visitors to create a MyData account (using an email address) and a password, we deployed a new service whereby web site visitors can use their existing Google ID or Facebook ID to login.  And, if they have already logged into Google (or Facebook), then there is no need to login again to ICPSR.  (This type of feature is often called SSO for Single Sign-On.)

I was curious to see how many people were taking advantage of this new service, and so I pulled a week's worth of information about newly created profiles.  (We always create a profile for a new ICPSR web site user, regardless if s/he uses Facebook, Google, or MyData.)  The specific time period covered was May 19, 2011 through mid-morning May 26, so just a bit over a week.

During that time there were nearly six hundred new profiles created, and a bit over two-thirds took advantage of the new SSO service.  Here are the numbers:

So, 399 new profiles created that used either Google or Facebook to log in to ICPSR's web site, and 171 new profiles where the person created a MyData account.

I wasn't surprised to see the non-MyData profiles to be higher than the MyData profiles, but I was surprised that the difference wasn't even more pronounced.  My guess would have been that most people have either a Facebook or Google ID (or both), and that most people would rather use that ID for logging into a system rather than taking the time to set-up a MyData account, especially since we've seen that most web site visitors don't come back very often (or at all).

Maybe, though, this very transient usage is what leads some people to create the MyData account.  Perhaps they view the account is one which is completely disposable, and they don't care if they don't remember the account or password?  And perhaps they would prefer to keep their reusable identity (from Facebook or Google) separate?

Google and Facebook join MyData as one of the ways to login to ICPSR

After an unexpected delay we finally rolled out two new ways to authenticate to the ICPSR web site.  In addition to using the original ICPSR MyData account, web site visitors may now use either their Facebook account or Google ID to login to the ICPSR web site.

We wanted to make this type of service available because it is clear that most of our web site visitors who download data use it just once, or only infrequently.  And so for every researcher or scholar that uses ICPSR on a regular basis, there are two casual users who just want to download a dataset as quickly and easily as possible.

For this latter audience MyData doesn't make sense.  Why create a new account with a new password for something that may never be used again?

Now, people who would like to use their Facebook account or Google ID to identify themselves may do so.  And, better still, if the person is already logged into either service, they do not have to login again to ICPSR.  (This is often called SSO for Single Sign-On.)

We also looked at supporting the OpenID service, but our experience was that OpenIDs were a little too confusing for people.  The idea that one's identity is a URL rather than an email address is less common, and it felt too foreign to most people.

We hope the new login options are useful, and make it even easier to use ICPSR.

Teaching With Data portal

We've launched a new version of our Teaching With Data portal.

TwD is one of the National Science Digital Library (NSDL) pathways. Each pathway has a domain-specific focus or interest, and ours is using quantitative social science research data as an element of teaching.

Jane Wang is the lead software developer on TwD, and she has been building the site using a Fedora stack called Muradora. One of the design requirements from the research team was that the site had to be based on Fedora, and rather than build a complete portal from scratch, Jane selected the stack that would be the best fit for our development environment.

Our TwD site is also interesting in that it is the first portal that we've deployed completely in the cloud; there is no local hardware at ICPSR supporting the project (except for Jane's laptop). All of the content and software resides in Amazon's cloud.

Another interesting aspect of this release is that we're using OpenID for authentication. This means that instead of needing to create yet another login and password to remember, visitors may instead use an existing login and password from one of many OpenID-compliant sources, such as Google, AOL and Yahoo. We're using a third-party service called RPX from JanRain to enable this feature. I'm expecting that we'll enable OpenID and other authentication systems (such as Facebook Connect) on the main ICPSR web site in the near future.

Of course, the site has other new features as well, and I'd like to encourage you to have a look by clicking the image above.

OR Meeting - Live Chat Session - Technology & Preservation at ICPSR

Next Wednesday I'll be participating in a "live chat" session with Nancy McGovern. I thought I might seed the technology portion of the conversation with these topics:

1. Fedora Repository software
   
2. OpenID authentication for the ICPSR Web site
   
3. Cloud services from Amazon Web Services

Since we only have 30 minutes, and since the agenda is for both technology and preservation, I don't want to include too many topics.

Anything important I may have left out?

The future of MyData

When I arrived at ICPSR back in late 2002, we didn't have any sort of system to issue identities or authenticate visitors to the web site. At that time we asked users to type in an email address, performed some basic sanity-checking (like does it contain an '@'?), and then showed them our terms of use if no one had ever entered that email address before. Simple. But not terribly useful.

We launched our identity and authentication service, MyData, in 2004, and it has been busy collecting identities since then. I just ran a query, and at this moment, we have 168,524 identities. We ask new account holders to give us just a little personal information, such as how they fit into the world of social science (student, faculty, policy-maker, ...? economics, political science, sociology, ...? favorite stat package?). It's great.

But I always knew that having our own local identities with our own proprietary software was not going to be the best solution in the long run. For a very long time I've been waiting for Internet2's Shibboleth project to really hit the tipping point. And while it does seem to be gaining acceptance in higher ed, it still feels like it is moving pretty slow, and it isn't at all clear that it will hit the tipping point this year. Or next year. It's hard to tell.

Then, a year or two I came across OpenID. Unlike Shib where the institution is the identity provider, OpenID is an open, decentralized way for individuals to create and manage identities, and to use that identity across multiple web sites (i.e., Single Sign-On). I created an OpenID back then, but it didn't seem like many sites were supporting OpenID for authentication.

Now it's 2009. Google, Yahoo, AOL, Facebook, etc., etc. are all supporting OpenID authentication. Lots of ICPSR's newest users are graduate students and recent graduates, and I'll bet many of them already have a Facebook account. Or Google account. Or Yahoo account. And I'll bet many of them would be just as happy to use that account to identity themselves to the ICPSR web site rather than creating yet another identity on our site.

We're refreshing the web site in August, and a change in authentication and identity services isn't on the short list. And so for better or worse everyone will be able to use a MyData account to login to ICPSR in September 2009. But I'll be really surprised if people will be using a MyData account to login when we roll our our next web site refresh in 2011.