A tiny wishlist for Amazon Web Services' Route 53

We've been using the DNS hosting service, Route 53, from Amazon Web Services (AWS). The default port for a DNS server is UDP (and TCP) 53, and I've always presumed that this was the answer to the question:  Why did Amazon name its DNS service Route 53?

In general I like the Route 53 service pretty well.  It's smart how the DNS servers listed for a Hosted Zone (the term AWS uses for a domain hosted in Route 53) reside in different top-level domains, like ORG, NET, COM, and even CO.UK. The UI in the AWS Management Console is fine for managing small zones that contain just a handful of records.

There's one feature that I wish Route 53 had, though, and it would be particularly useful, I think, to research organizations in higher education.

In our grants and contracts there is often a commitment to build, deploy, and operate some technology deliverable.  Often the technology is a web portal of some sort, and the investigator is keen to register a new domain.  This leads to an initial registration of something like:

WhizBangProject.org

The domain may have only the smallest number of records:  an SOA and NS records, of course, and then perhaps an MX record routing mail to a central server, and an A record pointing to the IPv4 address of the web portal.

Soon, though, the researcher may decide to register the same name in different top-level domains, and we have:

WhizBangProject.net

WhizBangProject.com

WhizBangProject.info

joining the mix.  These domains have EXACTLY the same records as the first one, and so if one is running his/her own DNS service, one can configure the DNS server to use the same zone file when loading all of the domains.  This is nice - one file with one set of records to manage for many different domains.

However, it is often the case that the investigator discovers that the original name is not satisfactory, and so we then register an alternate name in several domains:

CoolBeansResearch.org

CoolBeansResearch.net

CoolBeansResearch.com

CoolBeansResearch.info

and maybe a slight variant too:

Cool-Beans-Research.org

Cool-Beans-Research.net

Cool-Beans-Research.com

Cool-Beans-Research.info

In a world where one runs one's own DNS server, the additional domains are not much extra work.  Like the original solution where we pointed the new domains at the same zone file, we can just point these new domains at that same zone file.

I wish Route 53 would let me create a collection of what they call a Record Set, and then apply those same records to an arbitrary set of what they call Hosted Zones.  If the SOA and NS Record Sets were unique to each Hosted Zone, that would be OK; it is really the other records - the ones we add ourselves in Route 53 - that we would want to share across all of the Hosted Zones.

DuraSpace announces SDSC as storage partner

DuraSpace announced recently their relationship with SDSC as a storage provider for DuraCloud.  As I posted a while ago, we have been using both DuraCloud and their SDSC storage partner for a while.  It's great to see DuraCloud continue to grow.

Introducing ICPSR's Virtual Data Enclave (VDE)

The ICPSR Virtual Data Enclave (VDE) is a secure, virtual environment in which a researcher can analyze sensitive data, create research products, and then take possession of those products and analysis.  And while he VDE is not a substitute for a physical enclave and the types of security protocols it facilitates, the VDE is very much a potential substitute for the traditional practice of distributing confidential data via removable media, such as CD-ROMs.

The VDE uses much of the same technology that ICPSR uses internally for its Secure Data management Environment (SDE) which we have described a few times.  In brief, we use a virtual desktop environment that is operated by the University of Michigan's central IT shop and connect it to what we call our Private Network Attached Storage (NAS) appliance.  Both the virtual desktop and NAS are behind a firewall, and we use the firewall and Windows group policies to restrict what actions one pay perform.  Download?  Nope.  Cut-and-paste between the virtual desktop and the real desktop? Uh uh.  Capture screenshots by taking a picture of your monitor?  Well, ......

The virtual environment keeps sensitive datasets under lock and key at ICPSR, but makes it available to researchers.  The environment contains the usual array of applications used in the social sciences (but no email!), exactly the same sort of stuff we might set up for a visiting scholar or OR.

The researcher accesses the environment through a small, easy-to-download and -install client based on VMware View Client.  Authentication takes place using standard University of Michigan credentials which we (ICPSR) and others at UMich can issue to "friends."  Access between the real desktop and the virtual desktop is encrypted, and we are in the process of adding IPSEC encryption between the virtual desktop and the NAS.  (This latter traffic passes over UMich's data backbone, and access to those routers is limited to UMich central IT network engineers.)

The virtual machine is completely ephemeral and can be wiped after each use.  Any intermediate research or results are stored on the ICPSR NAS.  Our NAS is backed up weekly, and tapes are cycled off-site quarterly.  Once the research has been completed ICPSR retains a "just in case you need it" snapshot for up to three years.

Amazon's loss is SDSC's gain

One of the recent Amazon Web Services (AWS) power outages has left some of my EBS volumes in an inconsistent state.  If these were simple volumes, each containing a filesystem, then the fix is easy:  just dismount the filesystem, run fsck to check it, and then remount the filesystem after it has been fixed.  We have done this on several of our EC2 instances that had inconsistent volumes.

Unfortunately, for these particular volumes we have bonded them together to form a virtual RAID.  And this RAID is used as a single multi-TB filesystem which is much bigger than fsck can handle.  So we are kind of stuck.

One option would be to newfs the big filesystem, and to move the several TBs of content back into AWS, but that would be very slow.  And if there is another power outage......

So instead we called up our pals at Duracloud and asked them if they could help us enable replication of our content to a second provider.  (The first provider is - ironically - AWS.  But their S3 service, not their EC2/EBS service.)  They said they'd be happy to help, and, in fact, they will starting to replicate our content later this same week.  (Now that's service!)

The new copy of our content will now be replicated in...... SDSC's storage cloud.  This really brings us full circle at ICPSR since our very first off-site archival copy was stored at SDSC. Back then (like in 2008) it was stored in their Storage Resource Broker (SRB) system, and we used a set of command-line utilities to sync content between ICPSR and SDSC.  

The SRB stuff was kind of clunky for us, especially given our large number of files, our sometimes large files (>2GB), and our sometimes poorly named files (e.g., control characters in file names).  Our content then moved into Chronopolis from SRB, and then at the end of the demonstration project, we asked SDSC to dispose of the copy they had.  But now it is coming back......

Surviving the move to Google

The University of Michigan is moving its business productivity systems (mail, calendar, among others) to Google this year.  Some university institutes and colleges have already made the move, and others will make the move later this year.  ICPSR and its parent, the Institute for Social Research, will move in August, although about 70 staff at ISR will make the move early.  These "Google Guides" will help others with the transition.

One concept that might be helpful for the move is to distinguish between an email address and email mailbox.

An email address is basically a pointer.  It can point to another email address, or it can point to an email mailbox.  People publish and share their email address with others, and this is the piece of information we use to target a piece of email.

An email mailbox is a place where email lands.  You log in to an email system (Gmail, Exchange, AOL, and many more) with an ID and password, and once there, you can search for messages, read messages, sort, filter, send, and the rest.

The big change at the U-M this year with email is with everyone's email mailbox.  It's changing from some legacy system at the U-M to Gmail.

Below is a typical example of how things work today at ICPSR.  The @icpsr.umich.edu email address is just a pointer to the U-M enterprise directory entry.  That entry - which ends with @umich.edu - is also just a pointer to another email address ending in @isr.umich.edu.  And that email address is the thing that actually points to the email mailbox, which in our case is the ISR Exchange server.  The first diagram below shows the relationship between email addresses and email mailboxes in the current system for my own email:

Contrast that with the image just above that shows the relationships after the move to Google.  There are two main changes.

The first is that the email mailbox now lives in Google Gmail rather than Exchange, and my email software is a web browser rather than Outlook.  This is a very big change.  Some will find it a pleasant change, and others will hate the new system.

The second is that the roles of the @isr.umich.edu and @umich.edu email address have reversed.  The @isr.umich.edu email address is now just a pointer to another email address, and the @umich.edu email address is the one that "points" directly to Gmail.  And, of course, just like before, one can publish or use any of the three addresses, and the mail goes to the same email mailbox.

Tagging EC2 instances and EBS volumes

Adding this to my Amazon (Web Services) wishlist....

Optional Billing tag which can be set when an EBS volume is created or when an EC2 instance is launched

There is a lot of convenience is having a single AWS account.  It makes it easier to find running instances in the AWS Console.  It eliminates the need to share AMIs across accounts.  It obviates the need to remember (and record) multiple logins and passwords.

However, there is one big win in having multiple AWS accounts:  It makes it easier to tie the charges for one set of cloud technology (one account) to a revenue source.  And so we often have four or five different AWS accounts for the four or five different projects we have underway.

It would give me the best of both worlds if I could have my single AWS account, but then specify a special-purpose tag (say, Billing) when I provision a piece of cloud infrastructure.  This would be an optional tag that I could set I launch an instance or create a volume.  This tag would control the format and grouping of charges on my monthly AWS invoice for that account.

For example, say I launch a small instance and set the value of the Billing tag to U12345 (a made-up University of Michigan account number).  And then I launch a second one with a Billing tag of F56789.  And then in addition to the usual AWS invoice with a line item like this:

AWS Service Charges


Amazon Elastic Compute Cloud
US East (Northern Virginia) Region
    Amazon EC2 running Linux/UNIX
        $0.080 per Small Instance (m1.small) instance-hour  1440 hours   $115.20

I would see an additional section:

AWS Service Charges by tag
U12345

Amazon Elastic Compute Cloud
US East (Northern Virginia) Region
    Amazon EC2 running Linux/UNIX
        $0.080 per Small Instance (m1.small) instance-hour  720 hours   $57.60
F56789

Amazon Elastic Compute Cloud
US East (Northern Virginia) Region
    Amazon EC2 running Linux/UNIX
        $0.080 per Small Instance (m1.small) instance-hour  720 hours   $57.60

This would make it easy for me to take my single invoice from Amazon and "allocate" it (a term from Concur, the system we use for managing this sort of thing) to the right internal account.

Amazon Web Services makes Tech@ICPSR weep

June 2012 was looking to be a great, great month for uptime.  We were on track to have our best month since November 2011 this fiscal year - just 60 minutes of downtime across all services and all applications.  It was going to be beautiful.

And then Amazon Web Services had another power failure.

And then we wept.

The power failure took the TeachingWithData portal out of action.  (To be fair, it was already having significant problems due to its creaky technology platform, but this took it all the way out of action.)  The failure also took our delivery replica out of action, and gave Tech@ICPSR the joy of rebuilding it over the weekend.

But the real trouble was with a company called Janrain.

Janrain sells a service called Engage.  Engage is what allows content providers (like ICPSR) to use identity providers (like Google, Facebook, Yahoo, and many more) so that their clients (like you) do not need to create yet another account and password.  Engage is a hosted solution that we use for our single sign-on service using existing IDs, and it works 99.9% of the time.

However, this hosted solution lives in the cloud.  We just point the name signin.icpsr.umich.edu at an IP address we get from Janrain, plug in calls to their API, and then magic happens.

Except when the cloud breaks.

Amazon took Engage off-line for nearly four hours.  And then once it came back up, it was thoroughly confused for another three hours.  Ick.

So, counting all of that time as "downtime" our fabulous June 2012 numbers suddenly became our awful June 2012 numbers.  Here they are:

If you click on the image above, Blogger will make it bigger.

Of course, during a lot of that downtime, all of the features on the web site except for third-party login worked fine.  And most of the problem happened late on a Friday night and Saturday morning during the summer, so that's a good time for something bad to happen, if it has to happen at all.

AWS power outage aftermath

As it turns out, it doesn't take all that long to run fsck on a large filesystem comprised of multiple AWS Elastic Block Storage (EBS) volumes:

[root@cloudora ~]# df -h /dev/md0
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              4.9T  2.9T  1.7T  64% /arcstore 

[root@cloudora ~]# fsck -y /dev/md0
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
/dev/md0 is mounted.
WARNING!!!  Running e2fsck on a mounted filesystem may cause
SEVERE filesystem damage.
Do you really want to continue (y/n)? yes
/dev/md0 has gone 454 days without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Error allocating icount link information: Memory allocation failed
e2fsck: aborted

I saw a few posts about coaxing e2fsck to use the filesystem for scratch space rather than memory, but unfortunately the older version of the program available on this EC2 instance does not support it.

So I think that we may end up blowing away this copy of archival storage and replacing it with a fresh one.

Amazon power outage and Amazon

Amazon suffered a power outage in their northern Virginia data center last week.  Here is my abridged timeline of events from the Amazon Service Health Dashboard:

Jun 14, 8:50 PM PDT We are investigating degraded performance for some volumes in a single AZ in the us-east-1 region.
Jun 14, 10:29 PM PDT We can confirm a portion of a single Availability Zone in the US-EAST-1 Region lost power. We are actively restoring power to the effected EC2 instances and EBS volumes. We are continuing to see increased API errors. Customers might see increased errors trying to launch new instances in the Region.
Jun 15, 12:11 AM PDT As a result of the power outage tonight in the US-EAST-1 region, some EBS volumes may have inconsistent data. As we bring volumes back online, any affected volumes will have their status in the "Status Checks" column in the Volume list in the console listed as "Impaired." You can use the console to re-enable IO by clicking on "Enable Volume IO" in the volume detail section, after which we recommend you verify the consistency of your data by using a tool such as fsck or chkdsk. If your instance is stuck, depending on your operating system, resuming IO may return the instance to service. If not, we recommend rebooting your instance after resuming IO.
Jun 15, 3:26 AM PDT The service is now fully recovered and is operating normally. Customers with impaired volumes may still need to follow the instructions above to recover their individual EC2 and EBS resources. We will be following up here with the root cause of this event.

And, indeed, Amazon did follow-up on the root cause of the problem.  Based on the post-mortem that has been reported in several venues, the root cause was a fault in commercial power.  And a generator.  And an electrical panel.  One view is that Amazon got very unlucky with power problems; another view is that they did not test their fail-over thoroughly enough.  I lean more to the former view.

ICPSR didn't suffer any outages.  For example, our cloud-based replica was available to us the entire time.  We did receive notifications from Amazon that specific EBS volumes (basically a virtual block device that may be attached to a cloud-based machine) may have been corrupted, and should be inspected.  Amazon included the specific volume.  Here's an example notification:

Dear ICPSR Technology ,
Your volume may have experienced data inconsistency issues due to failures during the 6/14/2012 power failure in the US-EAST-1 region. To restore access to your data we have re-enabled IO but we recommend you validate consistency of your data with a took such as fsck or chkdsk. For more information about impaired volumes see:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/monitoring-volume-status.html
Sincerely,
EBS Support

So this did create a bit of unscheduled work for the technology team because we had four affected volumes.

One was not attached to anything, and was not in use.  

One was attached to a machine we had recently retired.

But two were attached to a machine that stores an encrypted copy of our archival holdings.  The volumes are each 1TB and part of a multi-TB virtual RAID.  This makes for a very, very long-running fsck to inspect for problems.

I'll have the conclusion on Friday.

Job posting: Cloud Sourcing Manager (UMich)

The University of Michigan has posted a cloud-oriented position.  This is good news.  (The entire position is included in-line at the end of this post; the U-M job system does not retain job descriptions once the posting date has expired.)  However, based on my reading of the job description, it has a couple of serious flaws.

One, this position is funded for a single year.  Why?  Is this because the cloud is a passing fad?  Or because this position will be able to achieve complete success in a single year?  Why wouldn't this position be funded indefinitely?


[ Edit - I chatted with the hiring manager - Bill Wrobleski @ ITS - and he confirmed that the job actually has funding for at least two years.  This is very, very good news, and shows a much bigger commitment.  I would also highly recommend Bill as a colleague. ]

Two, the job position itself sends mixed messages.  For example, from the Job Summary:

This solution oriented, flexible and proactive manager will be responsible for working with key process stakeholders such as Procurement Services, Office of General Counsel and IT governance groups to establish effective, secure and sustainable cloud sourcing processes for IT-related activities.

OK, so this is going to be a pretty slow-moving effort, working with some of the most careful, cautious organizations.  Work will be in baby steps.  Right?  However, look at the Responsibilities:

Accelerate the adoption of cloud solutions by removing barriers, streamlining processes and supporting the move to cloud solutions.

Creative problem solving and strong strategic thinking skills.  

So work with lots of stakeholders AND accelerate the adoption of cloud solutions? How? (And this second thing looks more like a Required Qualification than a Responsibility.)

So in the spirit of offering constructive criticism, here is my unsolicited, unofficial, revised Job Summary:

Information and Technology Service (ITS) is seeking an experienced Cloud Sourcing Manager to promote a "cloud first" culture at the University of Michigan. This person will be responsible for making it easier to purchase and use enterprise-class cloud services (e.g., Amazon Web Services, Salesforce.com) by collaborating with U-M Procurement Services, Office of General Counsel and IT governance groups. This person will also be responsible for lowering barriers to using consumer-class cloud services (e.g., Dropbox, Lastpass) by conducting regular demonstrations and informational sessions (town halls, brown bags) to bring awareness of cloud resources to the entire U-M community. 

NOTE: this position is funded indefinitely.

So now the person has the opportunity to accelerate cloud use (where it makes sense) while s/he works through the bureaucracy of the U-M to make bigger, systemic changes.  I like this more already!

And now for my unsolicited, unofficial, revised Responsibilities.  First, I should note that I like all of these ones from the original posting:

• Maintain a broad understanding of cloud marketplace including Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS) offerings. 
• Manage relationships with cloud vendors including establishing service level standards and reporting. 
• Participate on and/or lead project teams that are leveraging new cloud-based products. 
• With appropriate stakeholders, support contract negotiation with cloud providers. 
• Track the use of cloud services and identify trends and investment opportunities. 
• Work cooperatively with higher education consortiums [sic] to provide and leverage expertise and combined purchasing power.

Some in the original posting are OK too, but strike me as details of the position (i.e., you need to work with other teams in ITS; and you should make sure that the cloud stuff can work with our existing IT infrastructure). I would not include them any more than I would include responsibilities like attending meetings, reading email, and brushing teeth.

And here are the items I would add.  Mine are really specific:

• Negotiate an invoicing mechanism with Amazon Web Services (AWS) such that the U-M receives preferential rates based on aggregate campus usage, not individual department usage
• Using AWS Direct Connect establish a private, secure, low-cost network connection between the U-M campus and the AWS cloud
• Using AWS Virtual Private Cloud establish a private, secure, virtual data center in the AWS cloud for use by researchers and IT staff at the U-M
• Organize a campus-wide "Cloud Day" event at U-M similar to events focused on IT security and cyberinfrastructure, where speakers would present information about real-life use-cases using cloud capabilities to solve real problems
• At least once per month conduct an informational session for some department, institute or similar organization on campus, demonstrating a consumer-grade cloud product, noting its pros and cons, and how researchers and staff may use it safely

I should note that the AWS-related items would also help me out a lot.

And as promised, the original posting:

Cloud Sourcing Manager

How to Apply

A cover letter and resume are required; the cover letter must be PAGE 1 of your resume. The letter should:

(1) specifically outline the reasons for your interest in the position;
(2) outline your particular skills and experience that directly relate to this position; and
(3) include your current or ending salary.

Starting salary may vary depending on qualifications and experience of the selected candidate.

Job Summary

Information and Technology Service (ITS) is seeking an experienced Cloud Sourcing Manager to oversee cloud computing activities at the University of Michigan. This solution oriented, flexible and proactive manager will be responsible for working with key process stakeholders such as Procurement Services, Office of General Counsel and IT governance groups to establish effective, secure and sustainable cloud sourcing processes for IT-related activities.

NOTE: this position is funded for 1-year.

Responsibilities*

*Maintain a broad understanding of cloud marketplace including Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS) offerings.
*Manage relationships with cloud vendors including establishing service level standards and reporting.
*Participate on and/or lead project teams that are leveraging new cloud-based products.
*With appropriate stakeholders, support contract negotiation with cloud providers.
*Serve as a single-point-of-contact for cloud-related issues for the campus community.
*Oversee the creation and maintenance of business processes that support the acquisition, operation and retirement of cloud services such as templates and instructional material.
*Work with Information and Infrastructure Assurance (IIA) to encourage and ensure the use of cloud services follow appropriate security, compliance and privacy standards.
*Accelerate the adoption of cloud solutions by removing barriers, streamlining processes and supporting the move to cloud solutions.
*Work with architecture and technical groups to establish technical integration support to allow cloud services to leverage standard directory, authentication and other interfaces.
*Track the use of cloud services and identify trends and investment opportunities.
*Work cooperatively with higher education consortiums to provide and leverage expertise and combined purchasing power.
*Demonstrate effective staff leadership and development.
*Creative problem solving and strong strategic thinking skills.

Required Qualifications*

*Bachelor's degree in Computer Science, business or related field or an equivalent combination of education and experience.
*Extensive experience with IaaS, PaaS and SaaS services and implementations.
*Broad knowledge of cloud computing marketplace.
*8 years of progressive IT managerial experience.
*Proven project management experience to meet customer expectations and mitigate risk.
*Superb verbal and written communication skills.
*Demonstrated supervisory experience to include: recruiting, mentoring, staff development, performance management, leadership, and/or team building.
*Ability to interact successfully with a wide range of people including faculty, executive leadership, technical staff and other campus groups.
*Strong organizational skills and the ability to successfully complete multiple tasks within established and changing deadlines.

Desired Qualifications*

*Masters Degree in business, technology or related field.
*Experience leading IT efforts in a higher education institution.
*Significant experience managing vendor relationships.
*Experience negotiating contracts and managing ongoing service levels.

Additional Information

The University of Michigan was featured as one of the "Great Colleges to Work For" in the 2011 Chronicle of Higher Education.

U-M EEO/AA Statement

The University of Michigan is an equal opportunity/affirmative action employer.

The cloud is not a hard drive

I recently read a piece in one of the IT trade publications about why buying infrastructure via a cloud provider (Amazon in particular) was a bad deal, and how one would be much better off buying storage in-house.  As is often the case, the opinion-piece compares apples to oranges, and the analysis is so flawed, one can't really make any use of the conclusion.  However, my experience is that the piece is hardly unique in its flaws, and that motivated today's post.

The usual analysis is to compare the cost of storage when purchased as a consumer-grade disk drive v. storage when purchased via a cloud-based infrastructure provider.  The numbers -- $100 for a 2TB drive from Best Buy v. $100/mo for 1TB of S3 space at Amazon -- are thrown out as directly comparable, and thus it is plain to see that it is better to spend that $100 just once to get storage rather than spending the same amount of money each month (and for less storage to boot).

The analysis then usually performs some hand-waving about how disk drives sometimes fail (really?) and how they don't last forever (shocking!), but these facts are just in the noise, and it really is clear how much better one is purchasing storage rather than "renting" it in the cloud.

However, this sort of comparison is like looking at the cost of one's grocery bill for a given meal v. the cost of having a meal at a restaurant.  The first is merely one part of the second, and they simply are not the same thing.

A disk drive from Best Buy can be a very useful thing.  I myself own one of these very same drives, and I use it every couple of weeks to back-up a PC we have at home.  (Clearly my disaster recovery process is not good.)  Because the storage only needs to be "lit" every so often, and only needs to be accessed from a single location, it works well.

However....

If instead I needed my storage to be available 24 x 7, and to have some credible DR plan, this wouldn't do.  Or if I wanted to be able to use a lot more storage some days (or weeks or months) and less storage on others, then buying storage for peak isn't so good.  Also, when it is "lit" my storage consumes some electricity to operate.  So maybe an analysis that includes things like mirrored copies, electricity, front-end access systems, the network, and .....  gets to a more apples-to-apples comparison.

The cloud may be the best solution to some problems, and a dreadful solution to other problems.  Maybe way overpriced.  But a lot of different stuff makes up "cloud storage" and the underlying media is only one of many; in fact, it may be the least expensive component.

Zynga says "buh bye" to Amazon

So one big bit of news is how Zynga, the company responsible for such Facebook games as Farmville and Words With Friends, is building out its own IT infrastructure (zCloud) to host its games rather than continuing to rely solely upon Amazon for this infrastructure.

The obvious question:

Why?


Building your own cloud is a big bet:  data centers, racks and racks of equipment, servers, network switches, cables.... lots of cables.  That's a very big investment to make for a platform that's just overhead to your main business: developing software (games).

However.....

If the platform is actually your business, then it makes a lot of sense.  In that world the product isn't software; the product is the platform.  In that world you want to own the platform.  You want the control (costs, performance, etc).

Zynga says that it will still use AWS for spikes in service that it cannot service within its own zCloud platform, and so it isn't the complete end of the relationship for the two companies.  But it is still a very big change for Zynga.

Tech@ICPSR talks about the cloud @ LA2M

Tech@ICPSR will be giving a talk on cloud computing at the February 1, 2011 LA2M meeting.  We'll be talking about the cloud; kind of a high-level overview of what different folks say the cloud is, and some of the consumer- and business-oriented services and systems that live in it.

I'll add a link to the materials shortly after the talk, and, if LA2M adds the video of the talk to their archive, I'll add a link to that as well.

Systems Architect Senior job posting @ ICPSR

We've posted another job opening on my team at ICPSR.  I'll include the text from the job description below, but the short-lived link to the U-M job site is http://umjobs.org/job_detail/65035/systems_architect_senior.

In brief we're looking for someone with deep experience building production operational environments for web applications who can apply those skills to the ICPSR environment.  We currently have a mix of stuff running on real hardware at ICPSR and virtual hardware in Amazon's EC2, and we have a mix of legacy Perl CGI code and newer Java-based web applications.  A mix of myself and Steve Burling (who retires at the end of January) have been the main architects of the environment, but it has grown so much over the past 5-10 years that it has become a full-time job.

Here are the details:

Systems Architect Senior

Job Summary

The Inter-university Consortium for Political and Social Research (ICPSR), the world's largest archive of digital social science data, is now accepting applications for a Systems Architect Senior. ICPSR is a unit within the Institute for Social Research at the University of Michigan. ICPSR's data are the foundation for thousands of research articles, reports, and books. Findings from these data are put to use by scholars, policy analysts, policy makers, the media, and the public. This position reports to the Assistant Director, Technology Operations, Computer and Network Services.

Responsibilities*

Responsibilities
This position is responsible for the design, implementation, maintenance, and regular management of ICPSRs web systems development, staging, production, and disaster recovery operational environments. This consists of several distinct platforms, including local physical hardware and virtual systems hosted in Amazons Elastic Computing Cloud (EC2). The successful candidate will also work closely with the Assistant Director, Software Development, Computer and Network Services to define and implement functional requirements.

One, this position will select, install, and manage integrated development environment (IDE) software on developer workstations, and the underlying software repository. The incumbent systems are Eclipse and CVS, respectively.

Two, this position will select, install, manage, and maintain the testing, staging, and production platform environments used by ICPSR to deploy and test new web applications. The incumbent web application server is Apache Tomcat, sometimes run as a stand-alone web server and sometimes as a client to Apache Httpd. The incumbent server platform is a mix of local, physical servers and Elastic Computing Cloud (EC2) instances running within Amazon Web Services (AWS). ICPSR has interest in exploring a more complete. cloud-based web application platform such as AWS Elastic BeanStalk.

Three, this position will manage the AWS-hosted replica of ICPSRs production web environment. This includes building and maintaining tools that synchronize software, static content, and database content between the production environment and the replica environment.

Four, the over-arching responsibility of this position is to maintain and improve ICPSRs capacity for delivering high-availability, high-performance web-based services, managing the tension between the desire to have well-defined, documented, predictable deployments and business processes and the desire to have fast moving, fluid, and flexible deployments and business processes.

Required Qualifications*

BS in Computer Science, Computer Engineering, or at least eight years of experience with designing and managing complex web application hosting environments
Two or more years of experience with J2EE application servers (such as Tomcat)
Two or more years of experience with virtualization products or services, such as Amazon Web Services
In-depth expertise with RedHat Enterprise Linux 5 and 6
In-depth knowledge of networking principles and network support
In-depth knowledge of Web technologies (Apache, Tomcat)
Experience operating an RDBMS (Oracle, MySQL)
Experience working with monitoring tools, control software (CVS , Subversion, Perforce), and build tools (Make , Ant)
Strong understanding of web application architectures
Enthusiastic self-starter who works well with other team members
Excellent inter-personal skills with the ability to communicate clearly to peers, vendors, customers, and colleagues

Desired Qualifications*

MS in Computer Science or Computer Engineering
At least five years of experience with J2EE application servers (such as Tomcat)
Experience with Veritas products (Veritas Netbackup)
Storage experience (EMC)
Experience as an Infrastructure Engineer in a high availability environment
Expertise working with monitoring tools , version control software (CVS , Subversion, Perforce), and build tools (Make , Ant)

Underfill Statement

This position may be underfilled at a lower classification depending on the qualifications of the selected candidate.

U-M EEO/AA Statement

The University of Michigan is an equal opportunity/affirmative action employer.

Amazon's new AWS icons

Below is a completely unreadable schematic of ICPSR's replica of its web infrastructure in Amazon's cloud. This is just one way that we're using the cloud, and Amazon in particular.

Click the picture to enlarge.

Two weeks ago, Amazon released a nice set of icons for use in common drawing and presentation software.  The set contains an icon for all of the Amazon Web Services (AWS) services and types of infrastructure, and it also contains generic, gray icons for non-AWS elements.  I used the icons to create the nice schematic above.

The diagram is based on one of the examples Amazon includes in the PPTX-format set of icons.  I needed to delete a few services and servers that we don't use (e.g., Route 53 for DNS).  The diagram shows the ICPSR machine room on the left, and the three main systems that deliver our production web service:  a big web server, an even bigger database server, and an even bigger still EMC storage appliance.  We synchronize the content from these systems into corresponding systems in the AWS cloud.

We use EC2 instances in the US-East region to host our replica.  Unlike physical hardware where we sometimes host multiple IP addresses on a single machine, we maintain a one-to-one mapping between virtual machines and IP addresses in EC2.  And so one physical web server in ICPSR's machine room ends up as a pair of virtual servers in Amazon's cloud. 

We initiate a failover by changing the DNS A (address) record for www.icpsr.umich.edu and www.cceerc.org.  This change can take place on either a physical DNS server located at ICPSR or a virtual DNS server located in AWS.  The time-to-live (TTL) is very low, only 300 seconds, and so once we initiate the failover procedure, web browsers will start using the replica very soon.  (However, we have noticed that long-lived processes which do not regularly refresh name-to-address resolution for URLs, like crawlers, take much longer to failover.)

The replica supports most of the common services on the production ICPSR web site, such as search, download, analyze online, etc, but it does not support services where someone submits content to us, such as the Deposit System.

It is important to note that our replica is intended as a disaster recovery (DR) solution, not a high availability solution.  That is, the purpose of the replica is to allow ICPSR to recover quickly from failure, and to avoid a long (e.g., multi-day) period of unavailability.  The replica design is not at all a solution for a high-availability web site, one that would never be down even for a second.  It would take a significant investment to change the architecture of ICPSR's delivery platform to meet such a requirement.

Google Music keeps the tunes playing

I started using the new Google Music production service.  I hadn't explored Google's previous offering, the Music Beta, all that much, but decided the time was right to dip a toe into the water.

The service has a lot of similarities to iTunes, of course, except one's library is in the cloud rather than on a PC (assuming one isn't using Apple's iCloud).  Google gives one free space to store 20k songs.  I'm using about 1% of that quota so far.

I like the idea of having a copy of our music in the cloud as an additional backup (or preservation copy), and it is also nice being able to use a standard browser window to manage and play the music.  One complaint I have about iTunes is that because it is conventional desktop software, one has to update it from time to time.  And this is somewhat more burdensome if one has to switch from a "standard" type of login on Windows to one with administrative rights, and then switch back again.

Google provides a tool which will copy music from one's existing storehouse (mine was an iTunes library).  The tool worked well for this purpose, and it did NOT require any administrative rights on my home WinXP (I know, I know) to download, install, and execute.  I started the copy one evening, and some 400 songs had been copied into Google Music by the morning.  One feature request:  It would be fabulous if the Music Manager tool would pull songs directly from a CD.

On the back-end I wonder if Google is using some form of de-duplication to minimize the amount of storage it needs to provision for this service?  It must be the case that there would be great overlap between music collections, particularly with the most popular songs, artists, albums, etc.  Google does such a good job of squeezing storage efficiency out of GMail; would expect them to do the same for their music service.

ICPSR and the cloud

How is ICPSR using "the cloud?"

I've been getting this question a lot lately, and it feels like it's time to put together a blog post on this question.

From a functional standpoint ICPSR is using the cloud for identity and authentication, content delivery, archival storage, and data producer relationship management.  And if I include services based at the University of Michigan, I might also include data curation, and customer relationship management.

From a vendor standpoint here's a roster of some of the organizations with whom we're doing business, and how their piece of the cloud helps us run our business.

A typical transaction on the ICPSR web site looks like this:  Search.  Select content for download.  Create an ICPSR-specific identity.  Authenticate using that identity.  Download the content.  Do not return to ICPSR for at least a year.

Given that the ICPSR-specific identities are weak (i.e., web site visitors create them by entering an arbitrary email address and password) and given that they identity is often used only once, it seemed like a good idea to eliminate the need to create such an identity.  We don't need strong identities, but we do need identities that would be available to anyone.  Technologies like OpenID, Facebook Connect, and the like seemed promising, but who wants to build infrastructure which talks to all of them?

Janrain does.

We use Janrain Engage as one part of our identity and authentication strategy.  Janrain acts as a third party between the content provider (ICPSR) and the identity providers.  And so when someone needs to log in to ICPSR's portal, they see a screen that looks something like this:

So there's no need to create an account and password at ICPSR.  And if someone does return later, they don't have to log in to our site if they've already logged in to their identity provider's site.  (This is Single Sign-On or SSO.)

We're hosting several web portals in Amazon's cloud.  We're using Amazon's Infrastructure as a Service (IaaS) to stand-up Linux systems in the Amazon Elastic Computing Cloud (EC2) that are identical to the images we host locally.  We back the instances with Elastic Block Storage (EBS) volumes so that the content persists when we need to terminate and restart a computing instance. 

We also host a replica of our on-site delivery system in Amazon's cloud for disaster recovery (DR) purposes.  We find that we have the opportunity to "test" this replica at least once per year when ICPSR's headquarters loses power for several hours due to high winds, ice storms, or other acts of nature.

The Amazon service has been very reliable overall (despite a few highly publicized events), and certainly more reliable than our own on-site facilities.  We also like that we can scale resources up and down very quickly, and that we have clear costs associated with the infrastructure.  (Anyone at an institution of higher learning who has tried to calculate the actual cost of electricity used knows what I mean.)

I've posted many times about our relationship with DuraCloud, and how we're using it as a mechanism for storing archival copies in the cloud.  In many ways DuraCloud fulfills a role similar to that of Janrain Engage by providing a layer of abstraction between ICPSR's technical infrastructure and that of multiple service providers.  In this case we manage one vendor and one set of bills, but have the ability to store content in the cloud storage service of multiple providers (Amazon, Rackspace, Microsoft).

The acquisitions team at ICPSR keeps an eye on grants funded by places like the National Science Foundation and the National Institutes of Health.  If a grant looks like it may be producing data the team makes a note to contact the primary investigator (PI).  The goal is to have a conversation with the PI to see if there will indeed be data produced, and to see if it might be a good fit for ICPSR's holdings.  If so, we then try to convince the PI that depositing the content with ICPSR would be good for everyone (more data citations for the data producer; more re-use of the data by other researchers; etc.).

We had been using a home-built application to manage this content, but we found it to be a losing battle.  There was never enough money or time to build the types of relationship management reporting systems that the acquisition team wanted.  And so rather than trying to build a better mousetrap, we decided to rent a better mousetrap by moving the content into a professional contact/customer relationship management (CRM) system.  Like Salesforce.

 The University of Michigan central IT organization (ITS)  also delivers a handful of services that I would consider "the cloud" even though they do not package and market them that way.  File storage, trouble ticketing, and Drupal-hosting are all available from ITS, and they all look like cloud services to us because we pay for only what we use, we can scale them up and down reasonably quickly, and we do not have to deploy any local hardware or software to use them.

Hi Ho, a Googling we will go!

The University of Michigan announced (on Halloween! - I hope this is not a trick!) that it will be adopting Google as its collaboration platform.  The roll-out will happen over the course of the next year, and includes tools such as Gmail, Sites, Docs, Calendar, Blogger, and more.

I am delighted.

I've been using Google's Blogger technology (obviously) for some time to publish the Tech@ICPSR blog, and use Google Docs for almost any project where I would have used Microsoft Office in the past.  (I do still use PowerPoint from time-to-time if I need something fancy-schmancy, and don't have the time to conceptualize it as a Prezi instead.)

The biggest win for ICPSR, however, is with Gmail and Calendar.

When I arrived at ICPSR in 2002 we were running our own IMAP-based service with Eudora as the supported client.  And by supported I mean that we installed the free "hey look at these ads" version on each person's machine.  Off-site access was the responsibility of the individual, although we did hook it up to a campus webmail front-end eventually.  We were running MeetingMaker as our supported calendar client.  And by supported I mean that we installed the client on everyone's machine, but no one used it.

Sometime in 2005 or so we realized that it wasn't much fun running email and calendar services, and we also noted that we were already paying for an enterprise mail/calendar system that our parent organization, the Institute for Social Research (ISR), operated on the Exchange platform.  And so we dumped Eudroa and MeetingMaker and started using the Microsoft stack instead.

I was delighted.

However......

I soon experienced the harsh realities of life in the Microsoft stack.  Small mailbox quotas.  Feature-poor webmail experience.  Mailbox "archives" living in one-off files on my PC or file server.  And have you ever tried to find the full email headers in a piece of email stored on an Exchange server?  And like our days of running Eudora and MeetingMaker we continued to be isolated from the rest of campus since our Exchange system was local to ISR and not part of a campus-wide solution.

The honeymoon had ended.

I solved the problem for myself (sort of) by maintaining my "internal to ISR" meetings and email on the ISR Exchange server, but moving my "external" meetings and email to Google.  That is, I changed the U-M address book so that my bryan (at) umich.edu email address was routed to Gmail rather than the Exchange server.  And so when it comes to communicating with the world outside of the ISR, I have a rich email experience that works well in any web browser, superb mail searching, and despite not deleting a single piece of non-spam email in nearly three years, I have used less than 20% of my mail quota.  At this rate, I will need to delete my first email in 2024.  Nice.  Of course, the problem is that I now check email and calendars in two places:  MS Exchange (for my ISR world) and Google (for everything else).

And so I am looking forward to the day in 2012 when it all dovetails back together and there is just one place to check my mailbox and calendar again.

DuraCloud Archiving and Preservation Webinar

Shameless self-promotion alert...

The nice folks at DuraSpace have published the audio and video from the recent webinar that Michele Kimpton (CEO, DuraSpace) and I gave on DuraCloud.

DuraCloud Archiving & Preservation Webinar

View more webinars from DuraSpace

Michele spends the first 5-10 minutes talking about the business case behind DuraCloud, and then I spend about 30 minutes talking about ICPSR and how we came to use DuraCloud to store a copy of our archival holdings.

A dangerous combination

Do you like irony?

It turns out that the University of Michigan, like many other organizations, has decided to use the cloud for keeping track of its "Travel and Expense" software and reporting, and has therefore adopted Concur.

I think that the University has made a good decision to put this in the cloud, and to look to use a hosted solution (Software as a Service (SaaS)).  Using an existing service makes much more sense than building our own software.  How could the U-M build a better application than a company that makes its living doing exactly this sort of thing?

Now, this isn't to say that I am a huge fan of Concur (or at least how it has been implemented at U-M).  I don't find the workflow or interface to be all that intuitive, and there are a couple of things that really trip me up all the time.  For example, when entering the name of someone, sometimes I am supposed to enter their LAST name and sometimes I am supposed to enter their FIRST name, and I can never remember which to enter.  (Cue sad music.)

But the really challenging part about using this cloud service is when I use it to pay for cloud services.  (Cue ironic music.)

Each month I get a bill from Amazon.  And DuraCloud.  And another one from DuraCloud (because we use more space than our membership allows.)  And another one from Amazon.  (Two different projects with different credit cards and different pools of machines.)  And Salesforce.  And....

So each month I print the invoice to PDF.  And I fetch the receipt from my university credit card, and PDF that too.  And then I bundle them together in an expense report in Concur.  And that's when the trouble starts:  How do I classify the expense?

This is almost certainly not the fault of the Concur software, of course.  The problem is in the controlled vocabulary of "expense types" that the U-M has plugged into the system.  Not one is a good fit for paying cloud providers.  And so I pick one from the choices I do have.

Computer maintenance?

Computer rental?

Memberships (especially for the DuraSpace one, which is indeed a membership)?

Other?

My expense report is reviewed by at least four different people (two within ICPSR, at least one within our parent organization, the Institute for Social Research, at at least one at the U-M central Business and Finance unit).  If any one of them believes that I have selected the wrong expense type, the report returns to me, and I then must resubmit it.  The good news is that I don't have to reload the invoice or receipt, and so the process is relatively simple.

But for those of you about to implement Concur or another expense and travel reporting system, please add a new expense category for your IT managers:  Cloud computing services.