A tiny wishlist for Amazon Web Services' Route 53

We've been using the DNS hosting service, Route 53, from Amazon Web Services (AWS). The default port for a DNS server is UDP (and TCP) 53, and I've always presumed that this was the answer to the question:  Why did Amazon name its DNS service Route 53?

In general I like the Route 53 service pretty well.  It's smart how the DNS servers listed for a Hosted Zone (the term AWS uses for a domain hosted in Route 53) reside in different top-level domains, like ORG, NET, COM, and even CO.UK. The UI in the AWS Management Console is fine for managing small zones that contain just a handful of records.

There's one feature that I wish Route 53 had, though, and it would be particularly useful, I think, to research organizations in higher education.

In our grants and contracts there is often a commitment to build, deploy, and operate some technology deliverable.  Often the technology is a web portal of some sort, and the investigator is keen to register a new domain.  This leads to an initial registration of something like:

WhizBangProject.org

The domain may have only the smallest number of records:  an SOA and NS records, of course, and then perhaps an MX record routing mail to a central server, and an A record pointing to the IPv4 address of the web portal.

Soon, though, the researcher may decide to register the same name in different top-level domains, and we have:

WhizBangProject.net

WhizBangProject.com

WhizBangProject.info

joining the mix.  These domains have EXACTLY the same records as the first one, and so if one is running his/her own DNS service, one can configure the DNS server to use the same zone file when loading all of the domains.  This is nice - one file with one set of records to manage for many different domains.

However, it is often the case that the investigator discovers that the original name is not satisfactory, and so we then register an alternate name in several domains:

CoolBeansResearch.org

CoolBeansResearch.net

CoolBeansResearch.com

CoolBeansResearch.info

and maybe a slight variant too:

Cool-Beans-Research.org

Cool-Beans-Research.net

Cool-Beans-Research.com

Cool-Beans-Research.info

In a world where one runs one's own DNS server, the additional domains are not much extra work.  Like the original solution where we pointed the new domains at the same zone file, we can just point these new domains at that same zone file.

I wish Route 53 would let me create a collection of what they call a Record Set, and then apply those same records to an arbitrary set of what they call Hosted Zones.  If the SOA and NS Record Sets were unique to each Hosted Zone, that would be OK; it is really the other records - the ones we add ourselves in Route 53 - that we would want to share across all of the Hosted Zones.

My Amazon Web Services wishlist for the U-M

I'm serving on a University of Michigan task force that is looking at ways in which we can make cloud computing easier for faculty, students, and staff to consume.  This presupposes, of course, that at least some of the university community have research or business or other needs that would be well served by a cloud-type solution.

For those of us that are already using the cloud to solve a few different problems -- off-site archival copies, disaster recovery solution for delivery systems, among others -- the problem isn't so much how to get us to use cloud computing, but how the U-M can help us get the most value for our dollar.

With this in mind I offer my Amazon Web Services (AWS) wishlist for the U-M:

• Build Amazon Machine Images (AMI) for 64-bit Red Hat Linux (and, optionally, 64-bit Windows Server). Put any security or system or software goodies into the image that would be available to the entire university community (IT directors, grad students, casual users). This saves us from needing to build and maintain our own AMI or, worse, using one from a third-party.
• Deploy an AWS Virtual Private Cloud (VPC) that connects our own little piece of AWS “cloud space” to the rest of campus over a secure link. Allow instances running within this VPC to access infrastructure such as Active Director. Treat this part of AWS as if it were just another network (or data center) on campus. This enables us to deploy services dependent upon campus infrastructure in AWS more easily.
• Deploy an AWS Direct Connect between the VPC and UMnet (or Merit [the State of Michigan research and education network] or Abilene [Internet2's national network]). This grants us a fast, secure, inexpensive pipe for moving content between campus and AWS. We could start to deploy I/O-intensive resources in AWS more readily if we don’t have to pay for the bits individually.
• Implement an agreement where AWS has one customer (the University of Michigan) rather than many. (ICPSR alone has four different identities within AWS, largely so that we can map expenses from one identity to a university account.) This one customer would have different sub-accounts, and the usage across ALL of the sub-accounts would roll up to set pricing. ICPSR stores over 1TB of content in AWS S3 for example, and so our GB/month rate is $0.11. Other uses at U-M who store content in AWS S3, but less than 1TB, are paying over $0.12/GB/month. That is only a small amount more than $0.01/GB, but it adds up over ALL accounts each month.
• Explore the feasibility of allowing one to use U-M credentials (via Shibboleth?) to access key web applications at AWS, such as the AWS Management Console. We currently have to provision a separate email address and local (to AWS) password.
• Explore the feasibility of using an AWS Storage Gateway as a means to deliver additional storage needs for bursty or short-lived storage needs. It would be fabulous if we could buy nearly unlimited space in the U-M storage cloud. This is more feasible if we can use AWS storage for short-lived "bursts" of temporary storage.

ICPSR web site maintenance

We ran into a few problems last Wednesday during our system update.  We rolled back the changes, and are giving it another go this evening at 10pm EDT.  We're going to move traffic to our replica in the cloud during the maintenance so that we have more time for troubleshooting.

Our cloud replica has many of the features of our main site (search, download, analyze), but does not include features that transfer materials to ICPSR, such as our online Deposit Form.

Amazon's loss is SDSC's gain

One of the recent Amazon Web Services (AWS) power outages has left some of my EBS volumes in an inconsistent state.  If these were simple volumes, each containing a filesystem, then the fix is easy:  just dismount the filesystem, run fsck to check it, and then remount the filesystem after it has been fixed.  We have done this on several of our EC2 instances that had inconsistent volumes.

Unfortunately, for these particular volumes we have bonded them together to form a virtual RAID.  And this RAID is used as a single multi-TB filesystem which is much bigger than fsck can handle.  So we are kind of stuck.

One option would be to newfs the big filesystem, and to move the several TBs of content back into AWS, but that would be very slow.  And if there is another power outage......

So instead we called up our pals at Duracloud and asked them if they could help us enable replication of our content to a second provider.  (The first provider is - ironically - AWS.  But their S3 service, not their EC2/EBS service.)  They said they'd be happy to help, and, in fact, they will starting to replicate our content later this same week.  (Now that's service!)

The new copy of our content will now be replicated in...... SDSC's storage cloud.  This really brings us full circle at ICPSR since our very first off-site archival copy was stored at SDSC. Back then (like in 2008) it was stored in their Storage Resource Broker (SRB) system, and we used a set of command-line utilities to sync content between ICPSR and SDSC.  

The SRB stuff was kind of clunky for us, especially given our large number of files, our sometimes large files (>2GB), and our sometimes poorly named files (e.g., control characters in file names).  Our content then moved into Chronopolis from SRB, and then at the end of the demonstration project, we asked SDSC to dispose of the copy they had.  But now it is coming back......

Tagging EC2 instances and EBS volumes

Adding this to my Amazon (Web Services) wishlist....

Optional Billing tag which can be set when an EBS volume is created or when an EC2 instance is launched

There is a lot of convenience is having a single AWS account.  It makes it easier to find running instances in the AWS Console.  It eliminates the need to share AMIs across accounts.  It obviates the need to remember (and record) multiple logins and passwords.

However, there is one big win in having multiple AWS accounts:  It makes it easier to tie the charges for one set of cloud technology (one account) to a revenue source.  And so we often have four or five different AWS accounts for the four or five different projects we have underway.

It would give me the best of both worlds if I could have my single AWS account, but then specify a special-purpose tag (say, Billing) when I provision a piece of cloud infrastructure.  This would be an optional tag that I could set I launch an instance or create a volume.  This tag would control the format and grouping of charges on my monthly AWS invoice for that account.

For example, say I launch a small instance and set the value of the Billing tag to U12345 (a made-up University of Michigan account number).  And then I launch a second one with a Billing tag of F56789.  And then in addition to the usual AWS invoice with a line item like this:

AWS Service Charges


Amazon Elastic Compute Cloud
US East (Northern Virginia) Region
    Amazon EC2 running Linux/UNIX
        $0.080 per Small Instance (m1.small) instance-hour  1440 hours   $115.20

I would see an additional section:

AWS Service Charges by tag
U12345

Amazon Elastic Compute Cloud
US East (Northern Virginia) Region
    Amazon EC2 running Linux/UNIX
        $0.080 per Small Instance (m1.small) instance-hour  720 hours   $57.60
F56789

Amazon Elastic Compute Cloud
US East (Northern Virginia) Region
    Amazon EC2 running Linux/UNIX
        $0.080 per Small Instance (m1.small) instance-hour  720 hours   $57.60

This would make it easy for me to take my single invoice from Amazon and "allocate" it (a term from Concur, the system we use for managing this sort of thing) to the right internal account.

Amazon Web Services makes Tech@ICPSR weep

June 2012 was looking to be a great, great month for uptime.  We were on track to have our best month since November 2011 this fiscal year - just 60 minutes of downtime across all services and all applications.  It was going to be beautiful.

And then Amazon Web Services had another power failure.

And then we wept.

The power failure took the TeachingWithData portal out of action.  (To be fair, it was already having significant problems due to its creaky technology platform, but this took it all the way out of action.)  The failure also took our delivery replica out of action, and gave Tech@ICPSR the joy of rebuilding it over the weekend.

But the real trouble was with a company called Janrain.

Janrain sells a service called Engage.  Engage is what allows content providers (like ICPSR) to use identity providers (like Google, Facebook, Yahoo, and many more) so that their clients (like you) do not need to create yet another account and password.  Engage is a hosted solution that we use for our single sign-on service using existing IDs, and it works 99.9% of the time.

However, this hosted solution lives in the cloud.  We just point the name signin.icpsr.umich.edu at an IP address we get from Janrain, plug in calls to their API, and then magic happens.

Except when the cloud breaks.

Amazon took Engage off-line for nearly four hours.  And then once it came back up, it was thoroughly confused for another three hours.  Ick.

So, counting all of that time as "downtime" our fabulous June 2012 numbers suddenly became our awful June 2012 numbers.  Here they are:

If you click on the image above, Blogger will make it bigger.

Of course, during a lot of that downtime, all of the features on the web site except for third-party login worked fine.  And most of the problem happened late on a Friday night and Saturday morning during the summer, so that's a good time for something bad to happen, if it has to happen at all.

AWS power outage aftermath

As it turns out, it doesn't take all that long to run fsck on a large filesystem comprised of multiple AWS Elastic Block Storage (EBS) volumes:

[root@cloudora ~]# df -h /dev/md0
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              4.9T  2.9T  1.7T  64% /arcstore 

[root@cloudora ~]# fsck -y /dev/md0
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
/dev/md0 is mounted.
WARNING!!!  Running e2fsck on a mounted filesystem may cause
SEVERE filesystem damage.
Do you really want to continue (y/n)? yes
/dev/md0 has gone 454 days without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Error allocating icount link information: Memory allocation failed
e2fsck: aborted

I saw a few posts about coaxing e2fsck to use the filesystem for scratch space rather than memory, but unfortunately the older version of the program available on this EC2 instance does not support it.

So I think that we may end up blowing away this copy of archival storage and replacing it with a fresh one.

Amazon power outage and Amazon

Amazon suffered a power outage in their northern Virginia data center last week.  Here is my abridged timeline of events from the Amazon Service Health Dashboard:

Jun 14, 8:50 PM PDT We are investigating degraded performance for some volumes in a single AZ in the us-east-1 region.
Jun 14, 10:29 PM PDT We can confirm a portion of a single Availability Zone in the US-EAST-1 Region lost power. We are actively restoring power to the effected EC2 instances and EBS volumes. We are continuing to see increased API errors. Customers might see increased errors trying to launch new instances in the Region.
Jun 15, 12:11 AM PDT As a result of the power outage tonight in the US-EAST-1 region, some EBS volumes may have inconsistent data. As we bring volumes back online, any affected volumes will have their status in the "Status Checks" column in the Volume list in the console listed as "Impaired." You can use the console to re-enable IO by clicking on "Enable Volume IO" in the volume detail section, after which we recommend you verify the consistency of your data by using a tool such as fsck or chkdsk. If your instance is stuck, depending on your operating system, resuming IO may return the instance to service. If not, we recommend rebooting your instance after resuming IO.
Jun 15, 3:26 AM PDT The service is now fully recovered and is operating normally. Customers with impaired volumes may still need to follow the instructions above to recover their individual EC2 and EBS resources. We will be following up here with the root cause of this event.

And, indeed, Amazon did follow-up on the root cause of the problem.  Based on the post-mortem that has been reported in several venues, the root cause was a fault in commercial power.  And a generator.  And an electrical panel.  One view is that Amazon got very unlucky with power problems; another view is that they did not test their fail-over thoroughly enough.  I lean more to the former view.

ICPSR didn't suffer any outages.  For example, our cloud-based replica was available to us the entire time.  We did receive notifications from Amazon that specific EBS volumes (basically a virtual block device that may be attached to a cloud-based machine) may have been corrupted, and should be inspected.  Amazon included the specific volume.  Here's an example notification:

Dear ICPSR Technology ,
Your volume may have experienced data inconsistency issues due to failures during the 6/14/2012 power failure in the US-EAST-1 region. To restore access to your data we have re-enabled IO but we recommend you validate consistency of your data with a took such as fsck or chkdsk. For more information about impaired volumes see:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/monitoring-volume-status.html
Sincerely,
EBS Support

So this did create a bit of unscheduled work for the technology team because we had four affected volumes.

One was not attached to anything, and was not in use.  

One was attached to a machine we had recently retired.

But two were attached to a machine that stores an encrypted copy of our archival holdings.  The volumes are each 1TB and part of a multi-TB virtual RAID.  This makes for a very, very long-running fsck to inspect for problems.

I'll have the conclusion on Friday.

Disaster Recovery at ICPSR : Part 4

Part 3 described how we activate the replica, and how it works.

Using the replica

We've used our replica system several times over the past three years.  Our usage falls into a couple of categories:

Scheduled maintenance.  There have been a couple of times where we've had scheduled maintenance, and we've pressed the replica into service.

The most recent instance was on February 12, 2012 when the campus networking guys upgraded the gear that connects ICPSR's home in the Perry Building to the backbone.  We executed the failover early on a Sunday, and then moved traffic back once we got the "all clear" signal.  This scenario tends to produce very good outcomes since we can plan for the transfer, and we aren't simultaneously trying to recover from some other problem.

Emergency failover.  The most common instances in this category are when the Perry Building loses power unexpectedly, and we need to move traffic over to the cloud replica as soon as possible.

This scenario also tends to have good outcomes since we can focus solely on the transfer, and there is relatively little we can do except wait for the power to be restored.  One complication can occur if the on-call engineer is not near a computer, and so there is a delay as s/he gets to the closest one.  Or, if the outage happens during the business day, we may need to execute the failover very quickly, before our UPS systems become drained.

Emergency non-failover.  This is the category that corresponds to those times when we actually do NOT press the replica into service, but should have in retrospect.

A common scenario is that we see an alert for a single service (say our Solr search engine), and we begin to troubleshoot the problem.  Initially we may not know whether the problem will be fixed in just a few minutes, or if it will turn into a multi-hour process.  My usual rule of thumb is to press the replica into service in 30 minutes have elapsed, and if it feels like we're not very close to solving the problem.

This can go very wrong, of course, if my "feeling" is wrong, and can go very, very wrong if my "feeling" is wrong and we are short-handed, and I'm the one who is knee-deep in troubleshooting.  It can be very easy to look up 90 minutes later and say, "Oops."

Managing the replica

In general managing the replica is very inexpensive and requires little monitoring (by humans).  We have found that the main effort occurs when we are making a major upgrade in a core piece of technology such as the hardware platform (32-bit to 64-bit), the operating system (RHEL 5 to RHEL 6), or the web server itself.  In practice it means that in addition to upgrading the staging and development environments at ICPSR, we also need to upgrade the replica environment as well, and so adds more of the same type of work, not a new type of work.

Disaster Recovery at ICPSR : Part 3

Part 2 described the virtual infrastructure we built in Amazon Web Services to deploy a replica of ICPSR's content delivery system.

Monitoring the replica system

This turns out to be pretty tricky.

The University of Michigan Network Operations Center (NOC) monitors both our physical servers located in the ICPSR machine room and our virtual servers in Amazon Web Services (AWS).  Monitoring the physical servers is very straight-forward, but monitoring the virtual servers is not.

For the virtual machines we need to pick a URL which does not require authentication or a cookie, and which will not be redirected.  We also need a URL that points to a simple page so that the monitoring system does not grab page elements from the production web server rather than the replica.  In practice we have found the barriers to be so plentiful and so daunting we have, in fact, done a pretty crummy job of keeping an eye on the health of the replica.

Until recently.

We now run an additional instance in AWS which has one sole purpose:  monitor the replica system.  And to make this fool-proof, we add the same little "lie" to /etc/hosts on the monitoring machine and point names like www.icpsr.umich.edu and www.cceerc.org to the cloud replicas rather than the production systems.  This has worked very well for us so far in 2012.

Initiating failover

Imagine that the ICPSR "on-call" has just received a series of SMS messages on the on-call cell phone.  Everything at ICPSR is down, and the campus alert system reports that the Perry Building (ICPSR's home) has lost power.  There is no estimate time for repair.  The world looks like this:

We initiate the failover procedure by changing the DNS CNAME records for www.icpsr.umich.edu and www.childcareresearch.org.  Instead of "pointing" to the physical machines in the Perry Building, we point them to the cloud replicas.  If the failure did not include the production DNS server, we would make the change there.  However, in this scenario, the entire building has lost power, and so we need to make the change on the stealth slave server in AWS.

Now, as it turns out, the stealth slave server is recognized as a master server by the other slave servers for ICPSR's domain:  one at University of Michigan central IT and one at the San Diego Supercomputer Center.  Once we make a change to our server here (or in the cloud) those slave servers will pick it up within a few minutes.  And once they do, web requests start hitting our replica system rather than the production system.  And so the world changes from this:

to this:

in just a few minutes.

We can reverse the failover by making the same simple DNS record change, but in reverse.  We change the pointer from the cloud back to the physical systems in the ICPSR machine room.

Next: Part 4: Our experience with the replica over the past three years

Disaster Recovery at ICPSR - Part 2

Part 1 ended with ICPSR embarking on a project to build an off-site replica of its delivery system.

Amazon Web Services

I had been exploring Amazon Web Services (AWS) a little bit in late 2008, and had found it to be a very quick and easy way to stand-up technical infrastructure.  In contrast to the process we had been using to try to locate equipment at a University of Michigan data center, locating (virtual) equipment in AWS was astonishingly easy. I needed only a credit card and a Firefox plug-in to get started, and by using the excellent AWS-supplied tutorials I had soon deployed a stealth, slave DNS server for icpsr.umich.edu in AWS.  (A stealth server does not appear in the NS records for a domain.)

Also, AWS made it easy to grow into the cloud a little bit at a time.  Is a "small" virtual server under-powered for a replica of our production web server?  No problem, just terminate that virtual machine and relaunch the image on a "medium" virtual server.  Likewise we could add storage space when we needed it v. investing in a storage array which would be obsolete within two years.

We soon built enough infrastructure in AWS to serve as a replica, and it looks like this:

Click to enlarge.

Touring the replica

In addition to the slave DNS server we also stood up three additional servers in AWS.

One, a replica Oracle database server.  This is what AWS calls a c1.medium-sized instance, and mirrors the content we store in our production database.  We export content from the production database each morning, copy it to AWS, and then import it into the replica.

Two, a replica of our Child Care and Early Education Research Connections (CCEERC) web portal.  This portal runs on a virtual interface on the production web server, but it isn't so easy to add virtual interfaces to AWS instances.  This is what AWS calls an m1.small-sized instance, and provides the same basic content and functionality as www.cceerc.org.  We use rsync over ssh twice each day to keep content and web applications up to date.

Three, a replica of our main web portal.  This runs on what AWS calls a m1.large-sized instance since it bears the largest burden of any component.  Like with the CCEERC replica we synchronize content here on a twice daily basis.  We also disable certain web applications, like the Deposit System, so that we do not introduce potentially sensitive content to the cloud.  However, common services like search, browse, download, and analyze online are all available.

Each replica has a list of little white lies inside /etc/hosts that lead each machine to believe that www.icpsr.umich.edu and db.icpsr.umich.edu really do reside in AWS.  This trick allows us to run the same apps in the cloud without resorting to fragile, high maintenance software modifications that try to distinguish between systems in the cloud and systems in ICPSR's machine room.

Next up: Part 3: Using the replica

Disaster Recovery at ICPSR - Part 1

I'll be running a series on disaster recovery planning (DRP) and execution at ICPSR.  I'm responsible for ensuring that we have a working disaster recovery plan for two key areas of ICPSR:  delivery of content via our web site, and preservation of content via archival storage.  The requirements and solutions of the two areas are quite different, and I'll address each one separately.


This first post will focus on disaster recovery for our web-based delivery system.

Background

After a particularly long outage (3-4 days) in late 2008 due to a major ice storm that knocked out the power to our building, ICPSR made the decision to invest in a disaster recovery plan for our web-based delivery system.  The idea was to create a plan which would allow my team to have the process and infrastructure in place so that we could recover from a disaster befalling our delivery system.  We defined "disaster" to be an outage which could conceivably last for many hours or even days.  And the goal was to be able to recover from a disaster within one hour.

It is important to note that we were not intending to build a "high availability" delivery system.  The goal of that type of system would be to move ICPSR into the so-called "five nines" level of availability, meaning that our infrastructure would be available at least 99.999% of the time.  Converting ICPSR's plethora of legacy systems and infrastructure into such a high availability system would be a major project requiring a significant investment over several years.

Instead we set the bar lower, but not too low.  What if ICPSR had a goal of 99% availability each month? In that scenario we do not need the level of investment and infrastructure to avoid almost all down-time; we only need to be able to recover from down-time quickly, and to prevent any long outages.  The investment to reach that goal would be much smaller, and it would serve our community well.

The Starting Point

At this point in time we already had reasonably robust local systems - powerful servers for web and database services, an enterprise-class storage system, and UPS backup for all of our systems.  In addition, the University of Michigan Network Operations Center (NOC) was monitoring our systems 24 x 7.  The NOC's network monitoring system (NMS) sent automated emails to us whenever a component faulted.

However, we did not have any sort of on-call rotation ensuring that a fault would be caught and corrected quickly, and we also did not have any backup or replica system which could be pressed into service if, say, our building lost power for several hours (or days).  So we were exposed to short outages becoming unnecessarily long, and to long outages where we had no control over the recovery time.

We were able to address the first issue quickly and effectively by establishing an on-call rotation, where the "on-call" served one week at a time and carried a cell phone which received SMS alerts from the NOC's NMS.  This meant that faults would now be picked up and acted upon immediately by someone on the ICPSR IT team.  This alone would eliminate one class of long-lived outages, for example, where a fault would occur late on a weekend night, but not be picked up for repair until Monday morning.

The next step was to design, build, deploy, and maintain a replica of our delivery system.  But where?

Next up:  Part 2:  Building the replica

Zynga says "buh bye" to Amazon

So one big bit of news is how Zynga, the company responsible for such Facebook games as Farmville and Words With Friends, is building out its own IT infrastructure (zCloud) to host its games rather than continuing to rely solely upon Amazon for this infrastructure.

The obvious question:

Why?


Building your own cloud is a big bet:  data centers, racks and racks of equipment, servers, network switches, cables.... lots of cables.  That's a very big investment to make for a platform that's just overhead to your main business: developing software (games).

However.....

If the platform is actually your business, then it makes a lot of sense.  In that world the product isn't software; the product is the platform.  In that world you want to own the platform.  You want the control (costs, performance, etc).

Zynga says that it will still use AWS for spikes in service that it cannot service within its own zCloud platform, and so it isn't the complete end of the relationship for the two companies.  But it is still a very big change for Zynga.

Systems Architect Senior job posting @ ICPSR

We've posted another job opening on my team at ICPSR.  I'll include the text from the job description below, but the short-lived link to the U-M job site is http://umjobs.org/job_detail/65035/systems_architect_senior.

In brief we're looking for someone with deep experience building production operational environments for web applications who can apply those skills to the ICPSR environment.  We currently have a mix of stuff running on real hardware at ICPSR and virtual hardware in Amazon's EC2, and we have a mix of legacy Perl CGI code and newer Java-based web applications.  A mix of myself and Steve Burling (who retires at the end of January) have been the main architects of the environment, but it has grown so much over the past 5-10 years that it has become a full-time job.

Here are the details:

Systems Architect Senior

Job Summary

The Inter-university Consortium for Political and Social Research (ICPSR), the world's largest archive of digital social science data, is now accepting applications for a Systems Architect Senior. ICPSR is a unit within the Institute for Social Research at the University of Michigan. ICPSR's data are the foundation for thousands of research articles, reports, and books. Findings from these data are put to use by scholars, policy analysts, policy makers, the media, and the public. This position reports to the Assistant Director, Technology Operations, Computer and Network Services.

Responsibilities*

Responsibilities
This position is responsible for the design, implementation, maintenance, and regular management of ICPSRs web systems development, staging, production, and disaster recovery operational environments. This consists of several distinct platforms, including local physical hardware and virtual systems hosted in Amazons Elastic Computing Cloud (EC2). The successful candidate will also work closely with the Assistant Director, Software Development, Computer and Network Services to define and implement functional requirements.

One, this position will select, install, and manage integrated development environment (IDE) software on developer workstations, and the underlying software repository. The incumbent systems are Eclipse and CVS, respectively.

Two, this position will select, install, manage, and maintain the testing, staging, and production platform environments used by ICPSR to deploy and test new web applications. The incumbent web application server is Apache Tomcat, sometimes run as a stand-alone web server and sometimes as a client to Apache Httpd. The incumbent server platform is a mix of local, physical servers and Elastic Computing Cloud (EC2) instances running within Amazon Web Services (AWS). ICPSR has interest in exploring a more complete. cloud-based web application platform such as AWS Elastic BeanStalk.

Three, this position will manage the AWS-hosted replica of ICPSRs production web environment. This includes building and maintaining tools that synchronize software, static content, and database content between the production environment and the replica environment.

Four, the over-arching responsibility of this position is to maintain and improve ICPSRs capacity for delivering high-availability, high-performance web-based services, managing the tension between the desire to have well-defined, documented, predictable deployments and business processes and the desire to have fast moving, fluid, and flexible deployments and business processes.

Required Qualifications*

BS in Computer Science, Computer Engineering, or at least eight years of experience with designing and managing complex web application hosting environments
Two or more years of experience with J2EE application servers (such as Tomcat)
Two or more years of experience with virtualization products or services, such as Amazon Web Services
In-depth expertise with RedHat Enterprise Linux 5 and 6
In-depth knowledge of networking principles and network support
In-depth knowledge of Web technologies (Apache, Tomcat)
Experience operating an RDBMS (Oracle, MySQL)
Experience working with monitoring tools, control software (CVS , Subversion, Perforce), and build tools (Make , Ant)
Strong understanding of web application architectures
Enthusiastic self-starter who works well with other team members
Excellent inter-personal skills with the ability to communicate clearly to peers, vendors, customers, and colleagues

Desired Qualifications*

MS in Computer Science or Computer Engineering
At least five years of experience with J2EE application servers (such as Tomcat)
Experience with Veritas products (Veritas Netbackup)
Storage experience (EMC)
Experience as an Infrastructure Engineer in a high availability environment
Expertise working with monitoring tools , version control software (CVS , Subversion, Perforce), and build tools (Make , Ant)

Underfill Statement

This position may be underfilled at a lower classification depending on the qualifications of the selected candidate.

U-M EEO/AA Statement

The University of Michigan is an equal opportunity/affirmative action employer.

Amazon's new AWS icons

Below is a completely unreadable schematic of ICPSR's replica of its web infrastructure in Amazon's cloud. This is just one way that we're using the cloud, and Amazon in particular.

Click the picture to enlarge.

Two weeks ago, Amazon released a nice set of icons for use in common drawing and presentation software.  The set contains an icon for all of the Amazon Web Services (AWS) services and types of infrastructure, and it also contains generic, gray icons for non-AWS elements.  I used the icons to create the nice schematic above.

The diagram is based on one of the examples Amazon includes in the PPTX-format set of icons.  I needed to delete a few services and servers that we don't use (e.g., Route 53 for DNS).  The diagram shows the ICPSR machine room on the left, and the three main systems that deliver our production web service:  a big web server, an even bigger database server, and an even bigger still EMC storage appliance.  We synchronize the content from these systems into corresponding systems in the AWS cloud.

We use EC2 instances in the US-East region to host our replica.  Unlike physical hardware where we sometimes host multiple IP addresses on a single machine, we maintain a one-to-one mapping between virtual machines and IP addresses in EC2.  And so one physical web server in ICPSR's machine room ends up as a pair of virtual servers in Amazon's cloud. 

We initiate a failover by changing the DNS A (address) record for www.icpsr.umich.edu and www.cceerc.org.  This change can take place on either a physical DNS server located at ICPSR or a virtual DNS server located in AWS.  The time-to-live (TTL) is very low, only 300 seconds, and so once we initiate the failover procedure, web browsers will start using the replica very soon.  (However, we have noticed that long-lived processes which do not regularly refresh name-to-address resolution for URLs, like crawlers, take much longer to failover.)

The replica supports most of the common services on the production ICPSR web site, such as search, download, analyze online, etc, but it does not support services where someone submits content to us, such as the Deposit System.

It is important to note that our replica is intended as a disaster recovery (DR) solution, not a high availability solution.  That is, the purpose of the replica is to allow ICPSR to recover quickly from failure, and to avoid a long (e.g., multi-day) period of unavailability.  The replica design is not at all a solution for a high-availability web site, one that would never be down even for a second.  It would take a significant investment to change the architecture of ICPSR's delivery platform to meet such a requirement.

ICPSR and the cloud

How is ICPSR using "the cloud?"

I've been getting this question a lot lately, and it feels like it's time to put together a blog post on this question.

From a functional standpoint ICPSR is using the cloud for identity and authentication, content delivery, archival storage, and data producer relationship management.  And if I include services based at the University of Michigan, I might also include data curation, and customer relationship management.

From a vendor standpoint here's a roster of some of the organizations with whom we're doing business, and how their piece of the cloud helps us run our business.

A typical transaction on the ICPSR web site looks like this:  Search.  Select content for download.  Create an ICPSR-specific identity.  Authenticate using that identity.  Download the content.  Do not return to ICPSR for at least a year.

Given that the ICPSR-specific identities are weak (i.e., web site visitors create them by entering an arbitrary email address and password) and given that they identity is often used only once, it seemed like a good idea to eliminate the need to create such an identity.  We don't need strong identities, but we do need identities that would be available to anyone.  Technologies like OpenID, Facebook Connect, and the like seemed promising, but who wants to build infrastructure which talks to all of them?

Janrain does.

We use Janrain Engage as one part of our identity and authentication strategy.  Janrain acts as a third party between the content provider (ICPSR) and the identity providers.  And so when someone needs to log in to ICPSR's portal, they see a screen that looks something like this:

So there's no need to create an account and password at ICPSR.  And if someone does return later, they don't have to log in to our site if they've already logged in to their identity provider's site.  (This is Single Sign-On or SSO.)

We're hosting several web portals in Amazon's cloud.  We're using Amazon's Infrastructure as a Service (IaaS) to stand-up Linux systems in the Amazon Elastic Computing Cloud (EC2) that are identical to the images we host locally.  We back the instances with Elastic Block Storage (EBS) volumes so that the content persists when we need to terminate and restart a computing instance. 

We also host a replica of our on-site delivery system in Amazon's cloud for disaster recovery (DR) purposes.  We find that we have the opportunity to "test" this replica at least once per year when ICPSR's headquarters loses power for several hours due to high winds, ice storms, or other acts of nature.

The Amazon service has been very reliable overall (despite a few highly publicized events), and certainly more reliable than our own on-site facilities.  We also like that we can scale resources up and down very quickly, and that we have clear costs associated with the infrastructure.  (Anyone at an institution of higher learning who has tried to calculate the actual cost of electricity used knows what I mean.)

I've posted many times about our relationship with DuraCloud, and how we're using it as a mechanism for storing archival copies in the cloud.  In many ways DuraCloud fulfills a role similar to that of Janrain Engage by providing a layer of abstraction between ICPSR's technical infrastructure and that of multiple service providers.  In this case we manage one vendor and one set of bills, but have the ability to store content in the cloud storage service of multiple providers (Amazon, Rackspace, Microsoft).

The acquisitions team at ICPSR keeps an eye on grants funded by places like the National Science Foundation and the National Institutes of Health.  If a grant looks like it may be producing data the team makes a note to contact the primary investigator (PI).  The goal is to have a conversation with the PI to see if there will indeed be data produced, and to see if it might be a good fit for ICPSR's holdings.  If so, we then try to convince the PI that depositing the content with ICPSR would be good for everyone (more data citations for the data producer; more re-use of the data by other researchers; etc.).

We had been using a home-built application to manage this content, but we found it to be a losing battle.  There was never enough money or time to build the types of relationship management reporting systems that the acquisition team wanted.  And so rather than trying to build a better mousetrap, we decided to rent a better mousetrap by moving the content into a professional contact/customer relationship management (CRM) system.  Like Salesforce.

 The University of Michigan central IT organization (ITS)  also delivers a handful of services that I would consider "the cloud" even though they do not package and market them that way.  File storage, trouble ticketing, and Drupal-hosting are all available from ITS, and they all look like cloud services to us because we pay for only what we use, we can scale them up and down reasonably quickly, and we do not have to deploy any local hardware or software to use them.

A dangerous combination

Do you like irony?

It turns out that the University of Michigan, like many other organizations, has decided to use the cloud for keeping track of its "Travel and Expense" software and reporting, and has therefore adopted Concur.

I think that the University has made a good decision to put this in the cloud, and to look to use a hosted solution (Software as a Service (SaaS)).  Using an existing service makes much more sense than building our own software.  How could the U-M build a better application than a company that makes its living doing exactly this sort of thing?

Now, this isn't to say that I am a huge fan of Concur (or at least how it has been implemented at U-M).  I don't find the workflow or interface to be all that intuitive, and there are a couple of things that really trip me up all the time.  For example, when entering the name of someone, sometimes I am supposed to enter their LAST name and sometimes I am supposed to enter their FIRST name, and I can never remember which to enter.  (Cue sad music.)

But the really challenging part about using this cloud service is when I use it to pay for cloud services.  (Cue ironic music.)

Each month I get a bill from Amazon.  And DuraCloud.  And another one from DuraCloud (because we use more space than our membership allows.)  And another one from Amazon.  (Two different projects with different credit cards and different pools of machines.)  And Salesforce.  And....

So each month I print the invoice to PDF.  And I fetch the receipt from my university credit card, and PDF that too.  And then I bundle them together in an expense report in Concur.  And that's when the trouble starts:  How do I classify the expense?

This is almost certainly not the fault of the Concur software, of course.  The problem is in the controlled vocabulary of "expense types" that the U-M has plugged into the system.  Not one is a good fit for paying cloud providers.  And so I pick one from the choices I do have.

Computer maintenance?

Computer rental?

Memberships (especially for the DuraSpace one, which is indeed a membership)?

Other?

My expense report is reviewed by at least four different people (two within ICPSR, at least one within our parent organization, the Institute for Social Research, at at least one at the U-M central Business and Finance unit).  If any one of them believes that I have selected the wrong expense type, the report returns to me, and I then must resubmit it.  The good news is that I don't have to reload the invoice or receipt, and so the process is relatively simple.

But for those of you about to implement Concur or another expense and travel reporting system, please add a new expense category for your IT managers:  Cloud computing services.

ICPSR Web availability through October 2011

Ick.

This is not a good trend.

Our overall availability (i.e., all components are working properly) sank below 99.5% again in October.  The main culprit was a nearly two hour period on October 12, 2011 when a series of common alerts turned out to have an uncommon cause.  The oncall systems engineer went through our usual series of steps to bring the service back online, and while the steps seemed to help at first, it was clear that the fix was just temporary, and more diagnostic work was necessary.  This series of events also happened at an inopportune time, just as many of us were in transit between the office and home (and then back to the office again).

We also had a problem with our search engine technology (Solr) late in the month, and that contributed another 46 minutes to our unavailability.  (Other components were working fine, but search was not.)

My apologies to those of you who were trying to get some work done on our site last month, and got bit by either of these problems.

E Amazon Unum - Out of Amazon, One

Photo from http://www.flickr.com/photos/polselli/1250189137/

Amazon announced a new "region" in the United States last week:  the GovCloud region.  The geographic location of the new region is on the US West Coast, but the logical location is Washington, DC.

Amazon says that ALL of its availability regions offer FISMA Moderate security controls, but this region offers one additional feature and demands one additional requirement so that it "supports the processing and storage of International Traffic in Arms (ITAR) controlled data and the hosting of ITAR controlled applications."  The post goes on to say that:

As you may know, ITAR stipulates that all controlled data must be stored in an environment where logical and physical access is limited to US Persons (US citizens and permanent residents). This Region (and all of the AWS Regions) also provides FISMA Moderate controls. This means that we have completed the implementation of a series of controls and have also passed an independent security test and evaluation. Needless to say, it also supports existing security controls and certifications such as PCI DSS Level 1, ISO 27001, and SAS 70.

This gets interesting for organizations like ICPSR that conduct a lot of business with the US Government.  Earlier this year we spent mounted a significant effort to categorize the security level for content stored in our archive, and then documented the relevant NIST security controls.  It is easy to imagine this this type of effort will repeat itself as we interact with more federal agencies, and as those agencies struggle to become compliant with FISMA.

However, if I can short-circuit the process by using Amazon Web Services as my "machine room," and relying on Amazon's existing certifications and controls, then I may be able to ease the burden of writing and maintaining (and possibly implementing!) our own controls.  I would not expect to eliminate the entire effort of documenting NIST security controls, but I may be able to point to Amazon's existing controls and documentation for, say, those controls related to the physical machine room.  And remote access.

Indeed, instead of an AWS-hosted instance creating a barrier to a project ("oh no, if we build this in the cloud, we'll need to re-do all of the relevant NIST controls!"), it would facilitate the project. 

Amazon outage bites ICPSR too

Amazon reported a problem with its US-EAST region yesterday evening (EDT time).  Their service health dashboard reported that instances (virtual machines) in that region were having problems connecting to the Internet.  That was definitely true of our stuff.

We first saw alerts for our (virtual) equipment in the US-EAST at 22:25 EDT.  At that time we lost connectivity to every single instance in the US-EAST, but could still reach a small number of instances we have in other regions.  This affected the cloud-based replica of our production web server, the Teaching With Data web portal, and our "social login" service.  This latter service runs on an Amazon US-EAST operated by a company called Janrain, and isn't part of the instances where ICPSR has direct control.

By 22:56 EDT all of our systems were again reachable from the Internet, and no further action on our part (restoring content, restarting the instance) was necessary.

I have not yet seen a post mortem from Amazon, but based on my time in the data networking biz, my guess is that someone (Amazon or one of its transit or peering partners) made a routing change which blackholed their US-EAST traffic.

All in all Amazon continues to do a very good job with their cloud infrastructure, but this is a reminder that one would need to replicate services across several regions if one was to build a service with a very high level of availability.