ICPSR Web Availability - 2012-2013

Here are the final numbers for ICPSR's web site availability over our last fiscal year:

Click to embiggen

The year did not start off so well, and we reached the nadir quickly.  August 2012 was our worst period of availability in a very bad year for us overall.  January, March, and June 2012 also had very poor numbers.

The main antagonist we faced was a new and unusual problem with our Oracle database server.  For many years we would export the content for backup purposes each evening, and it worked well for a decade.  However, suddenly in 2012 we began to experience an outage just AFTER each export.  Despite intensive analysis by ourselves and local Oracle exports, we never could isolate the root cause of failure.

We eventually "solved" the problem by exporting our database only once per week v. once per day.  That left us more exposed to loss, of course, but it seemed to limit the outages to once per week v. once per day.

We then replaced the hardware with a new machine with a bit more processor and memory, but with blindingly fast solid-state drives. With the new machine deployed we returned to our daily export schedule, and the machine -- and our web availability -- have been in pretty good shape ever since. The machine went into service in April 2012, and the chart above makes it clear that life has been a little less hectic for our on-call engineer since then.

Introducing ICPSR's Virtual Data Enclave (VDE)

The ICPSR Virtual Data Enclave (VDE) is a secure, virtual environment in which a researcher can analyze sensitive data, create research products, and then take possession of those products and analysis.  And while he VDE is not a substitute for a physical enclave and the types of security protocols it facilitates, the VDE is very much a potential substitute for the traditional practice of distributing confidential data via removable media, such as CD-ROMs.

The VDE uses much of the same technology that ICPSR uses internally for its Secure Data management Environment (SDE) which we have described a few times.  In brief, we use a virtual desktop environment that is operated by the University of Michigan's central IT shop and connect it to what we call our Private Network Attached Storage (NAS) appliance.  Both the virtual desktop and NAS are behind a firewall, and we use the firewall and Windows group policies to restrict what actions one pay perform.  Download?  Nope.  Cut-and-paste between the virtual desktop and the real desktop? Uh uh.  Capture screenshots by taking a picture of your monitor?  Well, ......

The virtual environment keeps sensitive datasets under lock and key at ICPSR, but makes it available to researchers.  The environment contains the usual array of applications used in the social sciences (but no email!), exactly the same sort of stuff we might set up for a visiting scholar or OR.

The researcher accesses the environment through a small, easy-to-download and -install client based on VMware View Client.  Authentication takes place using standard University of Michigan credentials which we (ICPSR) and others at UMich can issue to "friends."  Access between the real desktop and the virtual desktop is encrypted, and we are in the process of adding IPSEC encryption between the virtual desktop and the NAS.  (This latter traffic passes over UMich's data backbone, and access to those routers is limited to UMich central IT network engineers.)

The virtual machine is completely ephemeral and can be wiped after each use.  Any intermediate research or results are stored on the ICPSR NAS.  Our NAS is backed up weekly, and tapes are cycled off-site quarterly.  Once the research has been completed ICPSR retains a "just in case you need it" snapshot for up to three years.

ICPSR web maintenance

We're updating a few pieces of core technology on our web server this afternoon:  httpd, mod_perl, Perl, and a few others.  Normally we like to perform maintenance like this during off-hours, but we're doing it at 12:30pm EDT today so that we have "all hands on-deck" to troubleshoot and solve problems.

We've already performed this maintenance on our staging server, and that went smoothly.  Our expectation is that this maintenance will last 15-30 minutes.

May 2012 web availability

Web availability was good, but not great, in May 2012:

Click to enlarge

Five main episodes of 25 to 59 minutes account for almost all of the 242 minutes of unavailability in May 2012.

On May 13 the production web server seemingly lost power, and it required an additional reboot and some TLC to bring the system back on-line fully (33 minutes).

On May 18 the search index on our CCEERC web portal became corrupted, and that disabled much of the usefulness of the site for nearly an hour (49 minutes).

On May 22 we saw the first of two episodes where the proxy (AJP) between Apache httpd and Apache tomcat faulted.  This did not recover on its own and required some help from the technology team.  This resulted in a medium-duration outage (27 minutes), and another similar fault occurred on the evening of May 31 (24 minutes).

On May 31 our production database server faulted, requiring a manual power-cycle, and also requiring the production web server to be rebooted (59 minutes).

My sense is that while we're in much better shape with regard to the instability caused by khugepaged, we are starting to see something a little amiss with the Apache proxy system.  It isn't clear to us at this time if the issue is faulty software or sub-optimized configuration on our part.

ICPSR system outage - 5/31/2012

ICPSR's content delivery systems faulted at approximately 8:30pm EDT on Thursday, May 31, 2012.  The oncall engineer discovered that the production Oracle database server had become unresponsive, and this disabled most features of most of our web portals.

After arriving on-site she rebooted the database server, but by then the production web server had become hopelessly confused.  She then rebooted that system as well, and all systems were back in service a bit before 9:30pm EDT.

The ICPSR technology team is reviewing system logs and access records to see if any further corrective action is required.

Our apologies for the inconvenience this no doubt caused to many of you.

Disaster Recovery at ICPSR : Part 4

Part 3 described how we activate the replica, and how it works.

Using the replica

We've used our replica system several times over the past three years.  Our usage falls into a couple of categories:

Scheduled maintenance.  There have been a couple of times where we've had scheduled maintenance, and we've pressed the replica into service.

The most recent instance was on February 12, 2012 when the campus networking guys upgraded the gear that connects ICPSR's home in the Perry Building to the backbone.  We executed the failover early on a Sunday, and then moved traffic back once we got the "all clear" signal.  This scenario tends to produce very good outcomes since we can plan for the transfer, and we aren't simultaneously trying to recover from some other problem.

Emergency failover.  The most common instances in this category are when the Perry Building loses power unexpectedly, and we need to move traffic over to the cloud replica as soon as possible.

This scenario also tends to have good outcomes since we can focus solely on the transfer, and there is relatively little we can do except wait for the power to be restored.  One complication can occur if the on-call engineer is not near a computer, and so there is a delay as s/he gets to the closest one.  Or, if the outage happens during the business day, we may need to execute the failover very quickly, before our UPS systems become drained.

Emergency non-failover.  This is the category that corresponds to those times when we actually do NOT press the replica into service, but should have in retrospect.

A common scenario is that we see an alert for a single service (say our Solr search engine), and we begin to troubleshoot the problem.  Initially we may not know whether the problem will be fixed in just a few minutes, or if it will turn into a multi-hour process.  My usual rule of thumb is to press the replica into service in 30 minutes have elapsed, and if it feels like we're not very close to solving the problem.

This can go very wrong, of course, if my "feeling" is wrong, and can go very, very wrong if my "feeling" is wrong and we are short-handed, and I'm the one who is knee-deep in troubleshooting.  It can be very easy to look up 90 minutes later and say, "Oops."

Managing the replica

In general managing the replica is very inexpensive and requires little monitoring (by humans).  We have found that the main effort occurs when we are making a major upgrade in a core piece of technology such as the hardware platform (32-bit to 64-bit), the operating system (RHEL 5 to RHEL 6), or the web server itself.  In practice it means that in addition to upgrading the staging and development environments at ICPSR, we also need to upgrade the replica environment as well, and so adds more of the same type of work, not a new type of work.

Disaster Recovery at ICPSR : Part 3

Part 2 described the virtual infrastructure we built in Amazon Web Services to deploy a replica of ICPSR's content delivery system.

Monitoring the replica system

This turns out to be pretty tricky.

The University of Michigan Network Operations Center (NOC) monitors both our physical servers located in the ICPSR machine room and our virtual servers in Amazon Web Services (AWS).  Monitoring the physical servers is very straight-forward, but monitoring the virtual servers is not.

For the virtual machines we need to pick a URL which does not require authentication or a cookie, and which will not be redirected.  We also need a URL that points to a simple page so that the monitoring system does not grab page elements from the production web server rather than the replica.  In practice we have found the barriers to be so plentiful and so daunting we have, in fact, done a pretty crummy job of keeping an eye on the health of the replica.

Until recently.

We now run an additional instance in AWS which has one sole purpose:  monitor the replica system.  And to make this fool-proof, we add the same little "lie" to /etc/hosts on the monitoring machine and point names like www.icpsr.umich.edu and www.cceerc.org to the cloud replicas rather than the production systems.  This has worked very well for us so far in 2012.

Initiating failover

Imagine that the ICPSR "on-call" has just received a series of SMS messages on the on-call cell phone.  Everything at ICPSR is down, and the campus alert system reports that the Perry Building (ICPSR's home) has lost power.  There is no estimate time for repair.  The world looks like this:

We initiate the failover procedure by changing the DNS CNAME records for www.icpsr.umich.edu and www.childcareresearch.org.  Instead of "pointing" to the physical machines in the Perry Building, we point them to the cloud replicas.  If the failure did not include the production DNS server, we would make the change there.  However, in this scenario, the entire building has lost power, and so we need to make the change on the stealth slave server in AWS.

Now, as it turns out, the stealth slave server is recognized as a master server by the other slave servers for ICPSR's domain:  one at University of Michigan central IT and one at the San Diego Supercomputer Center.  Once we make a change to our server here (or in the cloud) those slave servers will pick it up within a few minutes.  And once they do, web requests start hitting our replica system rather than the production system.  And so the world changes from this:

to this:

in just a few minutes.

We can reverse the failover by making the same simple DNS record change, but in reverse.  We change the pointer from the cloud back to the physical systems in the ICPSR machine room.

Next: Part 4: Our experience with the replica over the past three years

Disaster Recovery at ICPSR - Part 2

Part 1 ended with ICPSR embarking on a project to build an off-site replica of its delivery system.

Amazon Web Services

I had been exploring Amazon Web Services (AWS) a little bit in late 2008, and had found it to be a very quick and easy way to stand-up technical infrastructure.  In contrast to the process we had been using to try to locate equipment at a University of Michigan data center, locating (virtual) equipment in AWS was astonishingly easy. I needed only a credit card and a Firefox plug-in to get started, and by using the excellent AWS-supplied tutorials I had soon deployed a stealth, slave DNS server for icpsr.umich.edu in AWS.  (A stealth server does not appear in the NS records for a domain.)

Also, AWS made it easy to grow into the cloud a little bit at a time.  Is a "small" virtual server under-powered for a replica of our production web server?  No problem, just terminate that virtual machine and relaunch the image on a "medium" virtual server.  Likewise we could add storage space when we needed it v. investing in a storage array which would be obsolete within two years.

We soon built enough infrastructure in AWS to serve as a replica, and it looks like this:

Click to enlarge.

Touring the replica

In addition to the slave DNS server we also stood up three additional servers in AWS.

One, a replica Oracle database server.  This is what AWS calls a c1.medium-sized instance, and mirrors the content we store in our production database.  We export content from the production database each morning, copy it to AWS, and then import it into the replica.

Two, a replica of our Child Care and Early Education Research Connections (CCEERC) web portal.  This portal runs on a virtual interface on the production web server, but it isn't so easy to add virtual interfaces to AWS instances.  This is what AWS calls an m1.small-sized instance, and provides the same basic content and functionality as www.cceerc.org.  We use rsync over ssh twice each day to keep content and web applications up to date.

Three, a replica of our main web portal.  This runs on what AWS calls a m1.large-sized instance since it bears the largest burden of any component.  Like with the CCEERC replica we synchronize content here on a twice daily basis.  We also disable certain web applications, like the Deposit System, so that we do not introduce potentially sensitive content to the cloud.  However, common services like search, browse, download, and analyze online are all available.

Each replica has a list of little white lies inside /etc/hosts that lead each machine to believe that www.icpsr.umich.edu and db.icpsr.umich.edu really do reside in AWS.  This trick allows us to run the same apps in the cloud without resorting to fragile, high maintenance software modifications that try to distinguish between systems in the cloud and systems in ICPSR's machine room.

Next up: Part 3: Using the replica

Disaster Recovery at ICPSR - Part 1

I'll be running a series on disaster recovery planning (DRP) and execution at ICPSR.  I'm responsible for ensuring that we have a working disaster recovery plan for two key areas of ICPSR:  delivery of content via our web site, and preservation of content via archival storage.  The requirements and solutions of the two areas are quite different, and I'll address each one separately.


This first post will focus on disaster recovery for our web-based delivery system.

Background

After a particularly long outage (3-4 days) in late 2008 due to a major ice storm that knocked out the power to our building, ICPSR made the decision to invest in a disaster recovery plan for our web-based delivery system.  The idea was to create a plan which would allow my team to have the process and infrastructure in place so that we could recover from a disaster befalling our delivery system.  We defined "disaster" to be an outage which could conceivably last for many hours or even days.  And the goal was to be able to recover from a disaster within one hour.

It is important to note that we were not intending to build a "high availability" delivery system.  The goal of that type of system would be to move ICPSR into the so-called "five nines" level of availability, meaning that our infrastructure would be available at least 99.999% of the time.  Converting ICPSR's plethora of legacy systems and infrastructure into such a high availability system would be a major project requiring a significant investment over several years.

Instead we set the bar lower, but not too low.  What if ICPSR had a goal of 99% availability each month? In that scenario we do not need the level of investment and infrastructure to avoid almost all down-time; we only need to be able to recover from down-time quickly, and to prevent any long outages.  The investment to reach that goal would be much smaller, and it would serve our community well.

The Starting Point

At this point in time we already had reasonably robust local systems - powerful servers for web and database services, an enterprise-class storage system, and UPS backup for all of our systems.  In addition, the University of Michigan Network Operations Center (NOC) was monitoring our systems 24 x 7.  The NOC's network monitoring system (NMS) sent automated emails to us whenever a component faulted.

However, we did not have any sort of on-call rotation ensuring that a fault would be caught and corrected quickly, and we also did not have any backup or replica system which could be pressed into service if, say, our building lost power for several hours (or days).  So we were exposed to short outages becoming unnecessarily long, and to long outages where we had no control over the recovery time.

We were able to address the first issue quickly and effectively by establishing an on-call rotation, where the "on-call" served one week at a time and carried a cell phone which received SMS alerts from the NOC's NMS.  This meant that faults would now be picked up and acted upon immediately by someone on the ICPSR IT team.  This alone would eliminate one class of long-lived outages, for example, where a fault would occur late on a weekend night, but not be picked up for repair until Monday morning.

The next step was to design, build, deploy, and maintain a replica of our delivery system.  But where?

Next up:  Part 2:  Building the replica

Great FLAMEing file identification service

Some parts of the FLAME project will lend themselves to a microservices approach.  Microservices, like cloud computing, is a trendy, useful concept, but without a crystal clear definition.  But my take is that a microservice is something that performs one small, but useful bit of work, and which can be swapped in and out of an overall architecture at a component level.  It needs to have very clear inputs and outputs, and cannot contain any "secret sauce" that isn't part of its functional role.

Do not try this street magic at home.

One common activity at ICPSR is automated file identification.  Historically we've done this with the venerable UNIX utility file, but where we modify the magic database heavily, particularly for the formats we see most often.  We also post-process the output from file where we need additional handling above and beyond the capabilities of the magic database (e.g., making decisions based on the name or extension of the file).

Managing the magic database is not for the faint of heart.  (Try updating the Vorbis section.)  And this management has gotten both harder -- RHEL 6 uses a new format for its magic database which is incompatible with RHEL 5 -- and easier -- the new format eliminates the pesky magic.mime database.  However, we've gotten reasonably competent at managing magic and have come to rely on it for file format identification.

In support of the FLAME project we even created a little web service that takes a file's content and its name as input, and delivers a little snippet of XML as the output.  The XML contains the "human readable" answer from our magic database and the "MIME type" too.  This is our first FLAME-inspired web service.

If you'd like to try it, you can use your favorite form-capable URL transfer utility to do so.  Here's an example where I have run curl on one of our RHEL machines:

dhcp-bryan:; curl -F "file=@uuid-comparison.xlsx;filename=uuid-comparison.xlsx" www.icpsr.umich.edu/cgi-bin/wsifile<?xml version="1.0" encoding="utf-8"?><wsifile><ifile>Microsoft Excel</ifile><ifilemime>application/zip; charset=binary</ifilemime><uploadInfo>application/octet-stream</uploadInfo></wsifile>

feeding in an Excel file as the input, and another with a plain text file:

dhcp-bryan:; curl -F "file=@/etc/resolv.conf;filename=resolv.conf" www.icpsr.umich.edu/cgi-bin/wsifile<?xml version="1.0" encoding="utf-8"?><wsifile><ifile>ASCII text</ifile><ifilemime>text/plain; charset=us-ascii</ifilemime><uploadInfo>application/octet-stream</uploadInfo></wsifile>

and an interesting MS Word file:

dhcp-bryan:; curl -F "file=@2011-03CouncilPandAminutes.doc;filename=2011-03CouncilPandAminutes.doc" www.icpsr.umich.edu/cgi-bin/wsifile<?xml version="1.0" encoding="utf-8"?><wsifile><ifile>CDF V2 Document, Little Endian, Os: Windows, Version 5.1, Code page: 1200, Number of Characters: 0, Name of Creating Application: Aspose.Words for Java 4.0.3.0, Number of Pages: 1, Revision Number: 1, Security: 0, Template: Normal.dot, Number of Words: 0</ifile><ifilemime>application/msword; charset=binary</ifilemime><uploadInfo>application/octet-stream</uploadInfo></wsifile>

Feel free to try it out, and to post reactions, suggestions here.

March 2012 Web availability

March 2012 was not kind to us.

Clicking the image will display a full-size chart.  But please don't.  It is too ugly.

The main culprit in March was a continuing problem with the reliability of the production web server.  The environment - cooling, electricity, humidity - was fine, and the individual web applications were also fine, but something is not quite right with the kernel.  I think.  (If you would like to join the team as our new Senior Systems Architect and help solve the problem, see my post from last week.)

In March we saw multiple outages, each lasting over an hour.  The script always went something like this:

1. Load average increases by 5000-10000%
2. One web application stops responding and logging
3. KERN.INFO error messages from khugepaged and jsvc appear in syslog
4. Attempt to restart web application
5. Fail
6. Attempt to restart all web apps and their containers
7. Fail
8. Attempt to reboot machine
9. Fail
10. Optional:  Drive into office if weekend or early morning
11. Attempt to cycle power on machine
12. Mix of foul language and prayer
13. Repeat step #12
14. Success - machine is working again

Because we use the cloud instead of local, physical servers for many services, and because we haven't had all that many times where the machine needed its power cycled to solve a problem, we don't have things set-up for remote power access.  We'd like to address that.  (If you would like to join the team as our new Senior Systems Architect and help solve the problem, see my post from last week.)

So here's the plan to have a better April:

1. Disable khugepaged, hoping this might stop the machine from seizing up
2. Drive faster to the Perry Building, hoping this might result in faster applications of turning the power off and on
3. Hire the Senior Systems Architect, hoping that having a third pair of eyes on the problem might reveal its true cause and solution
4. Mix of foul language and prayer, hoping it will ease the pain

And, more seriously, we have also updated a few apps (like Solr) to use local storage rather than NFS-mounted storage for their work, particularly if the app tends to do a lot of writing to the filesystem.  NFS seems to be part of the mystery too.

FLAME update

We have three different tracks running on FLAME.

One track is conducting an analysis of the business requirements ICPSR has for what we are calling our "self archived" collection.  This is a collection of material best represented today by our Publication Related Archive, a set of materials that receives very, very little scrutiny between time of deposit and time of release on the web site.  We are imagining a future world where the quantity of "self-archived" materials increases dramatically from today's volumes, driven by NIH and NSF requirements to share and manage data.

I see the following questions generating the most discussion on this track:  How much disclosure review is necessary before releasing the content publicly?  Should the depositor have "edit" access to the metadata?  If so, should it be moderated or completely open?  How much "touch" does ICPSR need to have on these materials?

Another track is working on a crisp, concrete definition of what it means to "normalize" a system file from SAS, SPSS, or Stata.  ICPSR has long said that our approach is to "normalize" such files, producing plain ASCII data and set-ups, but what does that really mean?  And is that really possible?

I see the following questions generating the most discussion on this track:  Is ASCII the right thing, or ought it be a Unicode character set?  Are set-ups the right documentation or should it be DDI XML?  If we choose the former, is it sufficient to produce set-ups compatible with the original content type (e.g., SAS setups for a SAS file)?  What about precision?  Length of variable names?  Question text?  Is it possible to normalize without loss, and if not, how much loss is acceptable?  Can a computer do this without human intervention 99% of the time?

And the last track is working on a matrix that maps a set of parameters (inputs) to a resulting preservation commitment and set of actions.  For example, if one has a file which contains "documentation" (the type of content) in XML format in the UTF-8 character set (the format of the file), then perhaps the preservation commitment is "full preservation."

The key questions here, I believe, will be around what the right list of parameters is.  And if any of the parameters uses a controlled vocabulary, what's in the CV?  And what exactly does it mean to have a "full preservation" commitment?  What's involved beyond just keeping the bits around, which is presumably all one does with "bit-level preservation?"

The spring of our discontent

This has been a rough spring for the availability of our delivery system.  We are not having fun.  We are not amused.

The drill goes something like this:  The load average on the production web server soars from something around 1 to 50 or 100 or higher, and one of the web applications stops doing any useful work.  It is shown as running in the process table, but it does not answer queries and does no logging.

Killing the web application processes - a pair of tomcat daemons - works OK, but the processes remain in a defunct state, holding onto resources, and preventing a new web application from starting.  The parent PID is 1, but init isn't reaping the dead process.

Worse, if one tries to reboot the machine, say via /sbin/reboot, the machine hangs.  One has to cycle power on the machine to get it to restart cleanly.  Thank goodness for fast boot times and journaled filesystems!

Often these drills take place early in the morning (4am, 5am) when we run big automated jobs, like updating the search index.  And we always see a little something like this in /var/log/messages:

Mar 29 04:43:35 top kernel: INFO: task khugepaged:503 blocked for more than 120 seconds.
Mar 29 04:43:35 top kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 04:43:35 top kernel: khugepaged    D 0000000000000000     0   503      2 0x00000000
Mar 29 04:43:35 top kernel: ffff8804747d9c90 0000000000000046 0000000000000000 ffff880000058d80
Mar 29 04:43:35 top kernel: ffff8804747d9c20 0000000000000002 0000000000000003 ffff880000069b00
Mar 29 04:43:35 top kernel: ffff8804747b26b8 ffff8804747d9fd8 000000000000f4e8 ffff8804747b26b8
Mar 29 04:43:35 top kernel: Call Trace:
Mar 29 04:43:35 top kernel: [<ffffffff8116a1c8>] ? __mem_cgroup_try_charge+0x78/0x420
Mar 29 04:43:35 top kernel: [<ffffffff814eef25>] rwsem_down_failed_common+0x95/0x1d0
Mar 29 04:43:35 top kernel: [<ffffffff814ef083>] rwsem_down_write_failed+0x23/0x30
Mar 29 04:43:35 top kernel: [<ffffffff81276d83>] call_rwsem_down_write_failed+0x13/0x20
Mar 29 04:43:35 top kernel: [<ffffffff814ee582>] ? down_write+0x32/0x40
Mar 29 04:43:35 top kernel: [<ffffffff8116f140>] khugepaged+0x790/0x12c0
Mar 29 04:43:35 top kernel: [<ffffffff81090a90>] ? autoremove_wake_function+0x0/0x40
Mar 29 04:43:35 top kernel: [<ffffffff8116e9b0>] ? khugepaged+0x0/0x12c0
Mar 29 04:43:35 top kernel: [<ffffffff81090726>] kthread+0x96/0xa0
Mar 29 04:43:35 top kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20
Mar 29 04:43:35 top kernel: [<ffffffff81090690>] ? kthread+0x0/0xa0
Mar 29 04:43:35 top kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20

This can't be good.  It is never good when the kernel logs messages.  A happy kernel is a silent kernel.

Our experience has been that heavy use of NFS - especially by writers - tickles this bug more often, and so we've taken to off-loading some content from our EMC NAS to the local filesystem.  This also helps performance sometimes, but makes it more problematic to back up content given how we've architected things.

We've also done some digging into the deepest Red Hat forums and IRC channels looking for answers.  Maybe khugepaged is just the victim here?  Maybe the real culprint is the memory management in the kernel?  Or Colonel Mustard in the library?

This entry got us to try one more thing.

So we'll see how it goes next week.

If only we had Richard III to help.  Or Steinbeck.

Zynga says "buh bye" to Amazon

So one big bit of news is how Zynga, the company responsible for such Facebook games as Farmville and Words With Friends, is building out its own IT infrastructure (zCloud) to host its games rather than continuing to rely solely upon Amazon for this infrastructure.

The obvious question:

Why?


Building your own cloud is a big bet:  data centers, racks and racks of equipment, servers, network switches, cables.... lots of cables.  That's a very big investment to make for a platform that's just overhead to your main business: developing software (games).

However.....

If the platform is actually your business, then it makes a lot of sense.  In that world the product isn't software; the product is the platform.  In that world you want to own the platform.  You want the control (costs, performance, etc).

Zynga says that it will still use AWS for spikes in service that it cannot service within its own zCloud platform, and so it isn't the complete end of the relationship for the two companies.  But it is still a very big change for Zynga.

Sixteen products or one?

A recent conversation with Nathan Adams, ICPSR's Assistant IT Director for Software Development got me thinking about this....

It's no secret that ICPSR uses a package called Survey Documentation and Analysis (SDA) from UC Berkeley as our on-line analysis system.  But people may be surprised to learn that this one product forms the underpinnings of more than a dozen closely related ICPSR on-line analysis products.

One, Anonymous Analysis : This is where we make a dataset available via SDA and there is no authentication allowed.

Two, Authenticated Analysis : One must authenticate using MyData, Google, or Facebook.

Three, Member Analysis :  One must authenticate and also be using a computer located on the campus (even virtually) of a member institution.

Four, Private Analysis : One must authenticate and the identity used must be a member of a previously created group of identities.

Five through eight, Secure Analysis : Like any of the options above, but where the raw, proprietary, binary data files reside on a separate server, and where the ICPSR web server accesses the content via HTTPS rather than through the filesystem.

Nine through Sixteen, Non-disclosed Analysis : Like any of the eight options above, but where SDA's disclosure.txt controls have been used to attempt to prevent unintentional disclosure.

So sixteen different combinations!  And it is easy to imagine even more cropping up in the months ahead.

My experience is that one ends up with sixteen different online analysis "products" when things grow organically over time.  When things evolve due to a small tweaks in response to requests like, "Hey, could we use SDA for this, but with just one small change ..... ?"

It is easy to see how it happens.  But when things grow over time like this, they end up suffering from a profound lack of design, and end up costing more to maintain.  They are fragile.  They break when you change things, like the hardware.  Or the OS.  Or the NAS.  Or the authentication scheme.  Or the oil in your car.

So probably time to pull back a bit, pull together a team of content owners, and start asking some questions.

If we were going to start fresh today with an on-line analysis system, what should we build?

What sort of access controls are needed to prevent bad guys from using it?

What sort of disclosure mitigation capabilities are required to prevent accidents from happening?

To which populations might we need to restrict access?

What does the user experience look like?  Is this geared for the novice or for expert-in-a-hurry?  Or do we have multiple audiences and so need to build more than one experience?

Time to design.

Network maintenance - Sunday morning (EST) Feb 12, 2012

The University of Michigan central IT organization, ITS, will be upgrading the network gear that connects ICPSR's building to the campus data network.  The work is scheduled to start at 6am (EST) on the morning of Feb 12, 2012 and should take between one and two hours.

The ICPSR IT team will redirect traffic from the production system to our cloud replica during the maintenance period.  The replica runs in Amazon's cloud and features services such as search, analyze, and download, but purposefully does not enable features such as deposit.

Between the cut-over to the replica, the network maintenance, and the fallback to the production system, access will likely be a little rocky next Sunday morning.  Like a freeway during construction, if it is possible to take a detour around ICPSR's web site on Sunday morning, that's the safest route.  But if you find that you need to download some data or use the site, the replica will be available.

Tech@ICPSR talks about the cloud @ LA2M

Tech@ICPSR will be giving a talk on cloud computing at the February 1, 2011 LA2M meeting.  We'll be talking about the cloud; kind of a high-level overview of what different folks say the cloud is, and some of the consumer- and business-oriented services and systems that live in it.

I'll add a link to the materials shortly after the talk, and, if LA2M adds the video of the talk to their archive, I'll add a link to that as well.

Disaster Recovery v. High Availability

A question I often receive from customers and colleagues is:  If ICPSR has a replica of its production delivery system in Amazon's cloud, why is it that the web site is sometimes down due to scheduled maintenance or unplanned outages?

The short answer is:  ICPSR's cloud replica serves a disaster recovery (DR) purpose, but not a high availability (HA) purpose.  Of course, more often than not, this generates a look that falls somewhere between Bah! and This sounds like some made-up IT nonsense!  However, it really is the answer.  But that begs the question:  What's the difference between DR and HA?  But first a trip back in time....

As some long-time ICPSR clients may recall, the ICPSR delivery system was off-line for nearly a week during the holiday break between 2008 and 2009.  The root cause was a long power outage due to a major ice storm in the Midwest which knocked out power to many homes and businesses, including many in Ann Arbor.  And because ICPSR resides in a building just a little bit off the University of Michigan's central campus, we're just like any other home or business that waits for DTE Energy to restore power.

As one might expect both myself and the ICPSR Director at the time, Myron Gutmann, were quite anxious for the power to be restored.  The storm had caused so much damage that it wasn't at all clear when the building's power would be restored.  And, after the first few days without power - and heat - the building's pipes were in danger of bursting.  Things were looking pretty bad.

However, as it turned out we had been experimenting with Amazon's new computing and storage cloud just prior to the storm.  It would be pretty easy to stand up a minimal web server in Amazon's cloud, something that would basically say Yes, we know our delivery system is down, and we're sorry about that.  And here's the best guess from the local power company about when power will be restored.  We then worked with some of our colleagues at the University of Michigan and the San Diego Computing Center to update the system that maps names (like www.icpsr.umich.edu) to network addresses so that ICPSR's URLs for its web site would point to this new, minimal web server in Amazon's cloud.  That didn't fix the problem, of course, but it let people know that ICPSR knew there was a problem, and shared the best information we had about the problem.

Once power was restored and the main delivery system came back on-line, I had a long conversation with Myron about how we wanted to position ICPSR for any future problem like this.  What if the building lost power again for an extended period?  What if a tornado knocked down the whole building?  What if the U-M suffered some catastrophic problem with its network?

One option was to change the architecture of ICPSR's delivery systems.  Rather than having a complex series of simple web applications, we could redesign and rebuild the whole system so that it would also contain a middle layer of technology that would catch and route incoming requests to one of many delivery system components.  And rather than having a single production system at the University of Michigan, we would build a multi-site production system spread across multiple network providers and service providers so that no single problem would disrupt services.  This is essentially the high availability (HA) version of ICPSR's delivery system.  It would have the virtue of providing true 99.99%+ reliability, but would cost plenty of money to design, build, and operate.  If you are running IT systems for a bank or a hospital or an aircraft carrier, you build them with HA.  But what about a data archive?

Another option was to keep the ICPSR delivery architecture the same, but replicate it somewhere off-site.  Automated jobs could keep the web content, data content, and web applications synchronized.  And an easy - but manual - process could be used to redirect traffic to the replica when needed.  In this world there would still be plenty of times where a component of ICPSR's delivery system might be off-line due to maintenance or a fault, but if the maintenance or fault was long-lived, then the replica could be pressed into service.  This type of solution would be inexpensive to design, deploy, and operate, and would deliver a credible disaster recovery (DR) story, but would probably only give us uptime somewhere between 99.0% and 99.9%.  Would that be good enough?

In the end, of course, we decided that the best use of resources would be to build a system that would still have some outages from time to time, but which would never again be off-line for an entire week.  We set an availability goal of 99.5% for each month across all components.  That is, every time a single component faults - search, download, online analysis, and so on - it counts against the uptime of the WHOLE system.  And we would leave it up to the judgement of the on-call engineer to decide when a problem was likely to be long-lived enough to warrant a switch to the replica.

So we chose DR instead of HA.

Looking back, my sense is that we made the right decision.  In practice we seem to hit our 99.5% availability goal most months, and because we did not tie up our software and systems development resources on rebuilding the delivery system to guarantee HA, we were able to design and build systems like our Restricted Contract System, Secure Data Environment, and Virtual Data Enclave.  Of course, when we need to perform a major bit of maintenance like last weekend where it is important that we continue to point www.icpsr.umich.edu at the production system rather than the replica, it always makes me wonder about the HA alternative.

TRAC: A3.9: Self-assessment and certification

A3.9 Repository commits to a regular schedule of self-assessment and certification and, if certified, commits to notifying certifying bodies of operational changes that will change or nullify its certification status.

A repository cannot self-certify because an objective, external measurement using a consistent and repeatable certification process is needed to ensure and demonstrate that the repository meets and will likely continue to meet preservation requirements. Therefore, certification is the best indicator that the repository meets its requirements, fulfills its role, and adheres to appropriate standards. The repository must demonstrate that it integrates certification preparation and response into its operations and planning.

Evidence: Completed, dated audit checklists from self-assessment or objective audit; certificates awarded for certification; presence in a certification register (when available); timetable or budget allocation for future certification. 

---

Like a few of the other A-group TRAC requirements, this one really operates at the uppermost level of the organization, and so it is difficult to address it from the IT perspective.

HOWEVA..... One barrier to implementing a regular certification cycle are some fundamental questions:

Where do I find a list of consultants or analysts that can grant "TRAC certification" to my repository? 

Which organization sanctions those consultants and analysts? 

What does it mean - precisely - to be "TRAC certified?" 

Are there different levels of TRAC certification, much like FISMA levels? 

If I'm already FISMA certified, does that automatically grant TRAC certification for certain items (especially in section C)?

And so on.

It seems like there is a business opportunity here.

For instance, if ICPSR asserted that it was now in the business of reviewing TRAC requirements for organizations, and a team of ICPSR analysts would either certify your data archive as TRAC compliant or would identify clear action items required to become compliant, would that be a useful thing?  Or would other organizations rise up to say, "Hey, who are you, ICPSR, to be granting certifications?"

How should this work?

ICPSR web availability through December 2012

Web availability in December was looking very, very good through most of the month.  We had seen only a single noteworthy event the entire month, and that resulted only in a few minutes of downtime.  (As happens from time to time, a member site was scraping our web pages, presumably to collect the metadata we publish.  And while professional scrapers like Google, Yahoo, and the other search engines scrape gently and non-intrusively, this is not often the case with less experienced scrapers.)

Of course, December is always a tricky month here at ICPSR.  Snow storms.  Ice storms.  Power outages.  I can't remember the last time that my entire team was able to take off the entire week between Christmas and New Years (like the rest of the U-M) without having to come into the office to troubleshoot a problem.

And this year was no different.

We started to see sporadic up/down alerts from the U-M network monitoring system on the morning of December 30.  It looked like our production web server was working OK overall, but having some problems. When we tried to load the home page from home, the page wouldn't load.  And when we tried to login (via ssh) from home, the connection timed out.  It looked as if everything was down even though the monitoring system said it was OK.

We found we could log into other systems on campus, and then use those as a launch pad to get to ICPSR.  All of our systems were up, but none seemed reachable from systems off campus.  This explained why the U-M monitoring system didn't through more alarms earlier.

Then we noticed this:

http://status.its.umich.edu/outage.php?id=73300

(I think this link works even from off-campus.)

We then worked with the campus network engineers to draw their attention to the problem that was affecting us.  Unfortunately it was kind of helpful to have the ICPSR web site be unavailable from off-campus as a test case; we would know the network was fixed when the web site was available again.

All in all not a horrible month for availability, but we moved from 99.9% on Dec 29 to 99.5% by the end of Dec 30.