Adding this to my Amazon (Web Services) wishlist....
Optional Billing tag which can be set when an EBS volume is created or when an EC2 instance is launched
There is a lot of convenience is having a single AWS account. It makes it easier to find running instances in the AWS Console. It eliminates the need to share AMIs across accounts. It obviates the need to remember (and record) multiple logins and passwords.
However, there is one big win in having multiple AWS accounts: It makes it easier to tie the charges for one set of cloud technology (one account) to a revenue source. And so we often have four or five different AWS accounts for the four or five different projects we have underway.
It would give me the best of both worlds if I could have my single AWS account, but then specify a special-purpose tag (say, Billing) when I provision a piece of cloud infrastructure. This would be an optional tag that I could set I launch an instance or create a volume. This tag would control the format and grouping of charges on my monthly AWS invoice for that account.
For example, say I launch a small instance and set the value of the Billing tag to U12345 (a made-up University of Michigan account number). And then I launch a second one with a Billing tag of F56789. And then in addition to the usual AWS invoice with a line item like this:
AWS Service Charges
Amazon Elastic Compute Cloud
US East (Northern Virginia) Region
Amazon EC2 running Linux/UNIX
$0.080 per Small Instance (m1.small) instance-hour 1440 hours $115.20
I would see an additional section:
AWS Service Charges by tag
U12345
Amazon Elastic Compute Cloud
US East (Northern Virginia) Region
Amazon EC2 running Linux/UNIX
$0.080 per Small Instance (m1.small) instance-hour 720 hours $57.60
F56789
Amazon Elastic Compute Cloud
US East (Northern Virginia) Region
Amazon EC2 running Linux/UNIX
$0.080 per Small Instance (m1.small) instance-hour 720 hours $57.60
This would make it easy for me to take my single invoice from Amazon and "allocate" it (a term from Concur, the system we use for managing this sort of thing) to the right internal account.
Showing posts with label ec2. Show all posts
Showing posts with label ec2. Show all posts
Wednesday, July 11, 2012
Friday, June 22, 2012
AWS power outage aftermath
As it turns out, it doesn't take all that long to run fsck on a large filesystem comprised of multiple AWS Elastic Block Storage (EBS) volumes:
[root@cloudora ~]# df -h /dev/md0
Filesystem Size Used Avail Use% Mounted on
/dev/md0 4.9T 2.9T 1.7T 64% /arcstore
[root@cloudora ~]# fsck -y /dev/md0
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
/dev/md0 is mounted.
WARNING!!! Running e2fsck on a mounted filesystem may cause
SEVERE filesystem damage.
Do you really want to continue (y/n)? yes
/dev/md0 has gone 454 days without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Error allocating icount link information: Memory allocation failed
e2fsck: aborted
I saw a few posts about coaxing e2fsck to use the filesystem for scratch space rather than memory, but unfortunately the older version of the program available on this EC2 instance does not support it.
So I think that we may end up blowing away this copy of archival storage and replacing it with a fresh one.
Wednesday, December 21, 2011
Amazon's new AWS icons
Below is a completely unreadable schematic of ICPSR's replica of its web infrastructure in Amazon's cloud. This is just one way that we're using the cloud, and Amazon in particular.
Two weeks ago, Amazon released a nice set of icons for use in common drawing and presentation software. The set contains an icon for all of the Amazon Web Services (AWS) services and types of infrastructure, and it also contains generic, gray icons for non-AWS elements. I used the icons to create the nice schematic above.
The diagram is based on one of the examples Amazon includes in the PPTX-format set of icons. I needed to delete a few services and servers that we don't use (e.g., Route 53 for DNS). The diagram shows the ICPSR machine room on the left, and the three main systems that deliver our production web service: a big web server, an even bigger database server, and an even bigger still EMC storage appliance. We synchronize the content from these systems into corresponding systems in the AWS cloud.
We use EC2 instances in the US-East region to host our replica. Unlike physical hardware where we sometimes host multiple IP addresses on a single machine, we maintain a one-to-one mapping between virtual machines and IP addresses in EC2. And so one physical web server in ICPSR's machine room ends up as a pair of virtual servers in Amazon's cloud.
We initiate a failover by changing the DNS A (address) record for www.icpsr.umich.edu and www.cceerc.org. This change can take place on either a physical DNS server located at ICPSR or a virtual DNS server located in AWS. The time-to-live (TTL) is very low, only 300 seconds, and so once we initiate the failover procedure, web browsers will start using the replica very soon. (However, we have noticed that long-lived processes which do not regularly refresh name-to-address resolution for URLs, like crawlers, take much longer to failover.)
The replica supports most of the common services on the production ICPSR web site, such as search, download, analyze online, etc, but it does not support services where someone submits content to us, such as the Deposit System.
It is important to note that our replica is intended as a disaster recovery (DR) solution, not a high availability solution. That is, the purpose of the replica is to allow ICPSR to recover quickly from failure, and to avoid a long (e.g., multi-day) period of unavailability. The replica design is not at all a solution for a high-availability web site, one that would never be down even for a second. It would take a significant investment to change the architecture of ICPSR's delivery platform to meet such a requirement.
 |
| Click the picture to enlarge. |
Two weeks ago, Amazon released a nice set of icons for use in common drawing and presentation software. The set contains an icon for all of the Amazon Web Services (AWS) services and types of infrastructure, and it also contains generic, gray icons for non-AWS elements. I used the icons to create the nice schematic above.
The diagram is based on one of the examples Amazon includes in the PPTX-format set of icons. I needed to delete a few services and servers that we don't use (e.g., Route 53 for DNS). The diagram shows the ICPSR machine room on the left, and the three main systems that deliver our production web service: a big web server, an even bigger database server, and an even bigger still EMC storage appliance. We synchronize the content from these systems into corresponding systems in the AWS cloud.
We use EC2 instances in the US-East region to host our replica. Unlike physical hardware where we sometimes host multiple IP addresses on a single machine, we maintain a one-to-one mapping between virtual machines and IP addresses in EC2. And so one physical web server in ICPSR's machine room ends up as a pair of virtual servers in Amazon's cloud.
We initiate a failover by changing the DNS A (address) record for www.icpsr.umich.edu and www.cceerc.org. This change can take place on either a physical DNS server located at ICPSR or a virtual DNS server located in AWS. The time-to-live (TTL) is very low, only 300 seconds, and so once we initiate the failover procedure, web browsers will start using the replica very soon. (However, we have noticed that long-lived processes which do not regularly refresh name-to-address resolution for URLs, like crawlers, take much longer to failover.)
The replica supports most of the common services on the production ICPSR web site, such as search, download, analyze online, etc, but it does not support services where someone submits content to us, such as the Deposit System.
It is important to note that our replica is intended as a disaster recovery (DR) solution, not a high availability solution. That is, the purpose of the replica is to allow ICPSR to recover quickly from failure, and to avoid a long (e.g., multi-day) period of unavailability. The replica design is not at all a solution for a high-availability web site, one that would never be down even for a second. It would take a significant investment to change the architecture of ICPSR's delivery platform to meet such a requirement.
Wednesday, November 30, 2011
ICPSR and the cloud
How is ICPSR using "the cloud?"
I've been getting this question a lot lately, and it feels like it's time to put together a blog post on this question.
From a functional standpoint ICPSR is using the cloud for identity and authentication, content delivery, archival storage, and data producer relationship management. And if I include services based at the University of Michigan, I might also include data curation, and customer relationship management.
From a vendor standpoint here's a roster of some of the organizations with whom we're doing business, and how their piece of the cloud helps us run our business.
A typical transaction on the ICPSR web site looks like this: Search. Select content for download. Create an ICPSR-specific identity. Authenticate using that identity. Download the content. Do not return to ICPSR for at least a year.
Given that the ICPSR-specific identities are weak (i.e., web site visitors create them by entering an arbitrary email address and password) and given that they identity is often used only once, it seemed like a good idea to eliminate the need to create such an identity. We don't need strong identities, but we do need identities that would be available to anyone. Technologies like OpenID, Facebook Connect, and the like seemed promising, but who wants to build infrastructure which talks to all of them?
Janrain does.
We use Janrain Engage as one part of our identity and authentication strategy. Janrain acts as a third party between the content provider (ICPSR) and the identity providers. And so when someone needs to log in to ICPSR's portal, they see a screen that looks something like this:
So there's no need to create an account and password at ICPSR. And if someone does return later, they don't have to log in to our site if they've already logged in to their identity provider's site. (This is Single Sign-On or SSO.)
We're hosting several web portals in Amazon's cloud. We're using Amazon's Infrastructure as a Service (IaaS) to stand-up Linux systems in the Amazon Elastic Computing Cloud (EC2) that are identical to the images we host locally. We back the instances with Elastic Block Storage (EBS) volumes so that the content persists when we need to terminate and restart a computing instance.
We also host a replica of our on-site delivery system in Amazon's cloud for disaster recovery (DR) purposes. We find that we have the opportunity to "test" this replica at least once per year when ICPSR's headquarters loses power for several hours due to high winds, ice storms, or other acts of nature.
The Amazon service has been very reliable overall (despite a few highly publicized events), and certainly more reliable than our own on-site facilities. We also like that we can scale resources up and down very quickly, and that we have clear costs associated with the infrastructure. (Anyone at an institution of higher learning who has tried to calculate the actual cost of electricity used knows what I mean.)
I've posted many times about our relationship with DuraCloud, and how we're using it as a mechanism for storing archival copies in the cloud. In many ways DuraCloud fulfills a role similar to that of Janrain Engage by providing a layer of abstraction between ICPSR's technical infrastructure and that of multiple service providers. In this case we manage one vendor and one set of bills, but have the ability to store content in the cloud storage service of multiple providers (Amazon, Rackspace, Microsoft).
The acquisitions team at ICPSR keeps an eye on grants funded by places like the National Science Foundation and the National Institutes of Health. If a grant looks like it may be producing data the team makes a note to contact the primary investigator (PI). The goal is to have a conversation with the PI to see if there will indeed be data produced, and to see if it might be a good fit for ICPSR's holdings. If so, we then try to convince the PI that depositing the content with ICPSR would be good for everyone (more data citations for the data producer; more re-use of the data by other researchers; etc.).
We had been using a home-built application to manage this content, but we found it to be a losing battle. There was never enough money or time to build the types of relationship management reporting systems that the acquisition team wanted. And so rather than trying to build a better mousetrap, we decided to rent a better mousetrap by moving the content into a professional contact/customer relationship management (CRM) system. Like Salesforce.
The University of Michigan central IT organization (ITS) also delivers a handful of services that I would consider "the cloud" even though they do not package and market them that way. File storage, trouble ticketing, and Drupal-hosting are all available from ITS, and they all look like cloud services to us because we pay for only what we use, we can scale them up and down reasonably quickly, and we do not have to deploy any local hardware or software to use them.
I've been getting this question a lot lately, and it feels like it's time to put together a blog post on this question.
From a functional standpoint ICPSR is using the cloud for identity and authentication, content delivery, archival storage, and data producer relationship management. And if I include services based at the University of Michigan, I might also include data curation, and customer relationship management.
From a vendor standpoint here's a roster of some of the organizations with whom we're doing business, and how their piece of the cloud helps us run our business.
Given that the ICPSR-specific identities are weak (i.e., web site visitors create them by entering an arbitrary email address and password) and given that they identity is often used only once, it seemed like a good idea to eliminate the need to create such an identity. We don't need strong identities, but we do need identities that would be available to anyone. Technologies like OpenID, Facebook Connect, and the like seemed promising, but who wants to build infrastructure which talks to all of them?
Janrain does.
We use Janrain Engage as one part of our identity and authentication strategy. Janrain acts as a third party between the content provider (ICPSR) and the identity providers. And so when someone needs to log in to ICPSR's portal, they see a screen that looks something like this:
So there's no need to create an account and password at ICPSR. And if someone does return later, they don't have to log in to our site if they've already logged in to their identity provider's site. (This is Single Sign-On or SSO.)
We also host a replica of our on-site delivery system in Amazon's cloud for disaster recovery (DR) purposes. We find that we have the opportunity to "test" this replica at least once per year when ICPSR's headquarters loses power for several hours due to high winds, ice storms, or other acts of nature.
The Amazon service has been very reliable overall (despite a few highly publicized events), and certainly more reliable than our own on-site facilities. We also like that we can scale resources up and down very quickly, and that we have clear costs associated with the infrastructure. (Anyone at an institution of higher learning who has tried to calculate the actual cost of electricity used knows what I mean.)
We had been using a home-built application to manage this content, but we found it to be a losing battle. There was never enough money or time to build the types of relationship management reporting systems that the acquisition team wanted. And so rather than trying to build a better mousetrap, we decided to rent a better mousetrap by moving the content into a professional contact/customer relationship management (CRM) system. Like Salesforce.
The University of Michigan central IT organization (ITS) also delivers a handful of services that I would consider "the cloud" even though they do not package and market them that way. File storage, trouble ticketing, and Drupal-hosting are all available from ITS, and they all look like cloud services to us because we pay for only what we use, we can scale them up and down reasonably quickly, and we do not have to deploy any local hardware or software to use them.
Monday, April 25, 2011
For crying out cloud it has been an interesting weekend
The timing of the outage wasn't that bad for us. To the extent we use the Amazon cloud to perform research as part of our NIH "cloud grant" losing access over a long weekend wasn't that bad. At worst we lost one day (Friday) and one afternoon (Thursday) where we wouldn't have been able to do everything we might have wanted.
We also got lucky that our production web site didn't suffer any problems - and wasn't scheduled for any maintenance - this weekend. We run our replica system in Amazon's cloud, and had we lost the main site here at ICPSR, we would have been in very bad shape.
We also keep an encrypted copy of our holdings in Amazon's cloud, and over the past few days we haven't been able to keep it sync'd with master copies from here. But since we have so many copies in so many different locations, this wasn't all that worrisome. In fact, if we synchronized content weekly instead of daily, and if we always performed the synchronization on, say, Wednesday, the outage would have been a non-issue for this purpose. (But, of course, if the weekly synchronization was performed on Saturdays instead of Wednesday, we'd be drifting even more out of sync.)
The primary pain point was with a production web site we run exclusively in the cloud, TeachingWithData.org. For all practical purposes this site was off-line from Friday morning (EDT) until the middle of the day on Saturday. That's not good. But it's also the case that the service was never designed for 24 x 7 production, and so it's not surprising that it could suffer an extended outage. Building in fault-tolerance costs more.
The biggest lesson for us is that we need to make sure that some of our oldest, most long-lived instances are moved to EBS-backed instances. This is something that we've been meaning to do for some time, and this serves as a reminder of why it would be good to make it a higher priority.
Tuesday, September 14, 2010
Amazon introduces new "micro" instances
ICPSR is taking advantage of a new "micro"-sized virtual machine offered by Amazon Web Services (AWS). Amazon describes the new instance this way:
Micro Instance 613 MB of memory, up to 2 ECUs (for short periodic bursts), EBS storage only, 32-bit or 64-bit platformThis looked like a good fit for the "stealth" DNS server that we run in Amazon's cloud, and so we converted it from a Small Instance - Reserved ($350 for a three year term + $0.03/hour) to a Micro Instance - Reserved ($82 for a three year term + $0.007/hour).
We have other lightly used instances running in Amazon's cloud, and we'll likely convert them over too.
Friday, November 20, 2009
ICPSR and the Cloud
Interest in the cloud is heating up on the University of Michigan campus. In the past day or two I've seen surveys asking campus IT leaders to comment on their exploratory interest in the cloud, and have also answered email queries about who is dabbling with the cloud.
ICPSR started exploring the Amazon cloud in late 2008, and by early 2009 we had setup our first production service, a stealth authoritative DNS server for our domains, and a replica of our Web service infrastructure. We're primarily uses of the Elastic Computing Cloud (EC2) and Simple Storage Service (S3). We're also looking at CloudFront, but to be honest, I'm not sure we generate the volume of traffic that would make it super useful to us.
Since them we've also launched an instance in Amazon's EU zone for disaster recovery purposes, and launched a handful of new sites on cloud instances rather than local hardware. Here's a complete roster of instances as of today:
- Web server replica
- Oracle database replica
- CCERC Web replica
- Solr search replica
- Stealth authoritative DNS server (used in our DR process)
- Teaching With Data NSDL Pathway production service
- Teaching With Data NSDL Pathway staging server
- LDAP server supporting TWD
- SSDAN next generation production service
- SSDAN next generation development and staging server
- Content server for deep DR (located in the EU zone)
I think it's likely we'll move some of the replicas we run on behalf of the Minnesota Population Center to the cloud as well.
We've had great experiences with Amazon Web Services (AWS) so far: very low barrier to entry, and a very nice management tool in Elasticfox. The on-going operations have been stable and secure, and the price is right. And while I'm not sure I'd rely solely on the cloud for my archival storage solution, using the cloud to stash away one additional copy is very attractive.
Tuesday, October 27, 2009
Exciting News from Amazon
Amazon announced three new offerings in their cloud platform today. All sound very interesting, and all have potential utility to ICPSR.
One, Amazon now offers a bona fide relational database (MySQL-type) in the cloud. They handle the patching, scaling, and other classic DBA functions; you provide the data. We use Oracle heavily today, but make little use of Oracle-only features.
Two, they are now offering "high-memory" instances: High-Memory Double Extra Large Instance 34.2 GB of memory, 13 EC2 Compute Units (4 virtual cores with 3.25 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform ($1.20/hour); and, High-Memory Quadruple Extra Large Instance 68.4 GB of memory, 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform ($2.40/hour).
Three, they are dropping the price of "on-demand" instances by 15% effective Nov 1. We've switched to reserved instances for some of our long-lived virtual systems, but we still have a handful of on-demand systems, and so this will have an immediate positive impact on our monthly bill.
Definitely a nice "treat" from Amazon this Halloween!
Thursday, October 1, 2009
Teaching With Data Launched
ICPSR launched a new portal this week - TeachingWithData. This is part of ICPSR's grant to build a National Science Digital Library (NSDL) Pathway dedicated to social science data and teaching. I won't say too much more about the content in this post, but will instead focus on our technology selection.
The two main technical requirements were: (1) Fedora-based, and (2) fast deployment. We think we've achieved both with the site.
We implemented the site on an Amazon Web Services (AWS) Elastic Computing Cloud (EC2) instance; this allowed us to stand-up a base platform very quickly. We now have nearly a dozen instances running in AWS. Some are delivering production services, like our search technology and this Pathway, and others are used for science experiments and front-ends for storage. We've found that it's more convenient to synchronize and organize content in AWS by sticking a Linux instance in front so that we can use tools like rsync. Using Simple Storage Service (S3) directly is less convenient.
After looking at a variety of stacks that sit on top of Fedora, we selected Muradora for three main reasons:
- Open source software
- Active development community
- Best technology fit with other technology platforms in use at ICPSR
Thursday, June 11, 2009
Cloudy day for the cloud?
I use Google Alerts to keep tabs on a variety of people, places, and things of all sort, and something interesting hit my Gmail inbox today about "Amazon Web Services":
Looks like Amazon Web Services’ Elastic Compute service went down for an extended period this evening.
Now this was news to me, especially since we host our production study search service in the AWS EC2, and our replica delivery infrastructure too. Both the University of Michigan Network Operations Center (NOC) and Merit (NOC) monitor our systems, testing availability every minute of every day. And whenever there is an outage, the on-call engineer gets a page (or many pages!). And I'm the oncall this week. :-)
So what really happened?
Here's a piece of the story from the AWS Service Health Dashboard:
7:33 PM PDT We wanted to give you a quick update. A lightning storm caused damage to a single Power Distribution Unit (PDU) in a single Availability Zone. While most instances were unaffected, a set of racks does not currently have power, so the instances on those racks are down. We have technicians on site, and we are working to replace the affected PDU. We do not yet have an ETA, but we expect to be able to recover the instances when we restore power. Besides these affected instances, all other instances, and all other Availability Zones, are operating normally. Users with affected instances can launch replacement instances in any of the US Region Availability Zones or wait until their instance(s) are restored.
Some instances in one of AWS's availability zones (e.g., they have three for the US alone) failed. That's not wonderful news, especially if one of the failed instances belongs to you, but it is hardly a failure of the entire EC2 service.
To me this is somewhat like ICPSR messing up an entry in a database, making one of our studies unavailable by mistake, and someone blogging that ICPSR's on-line delivery service went down.
Net net for me: Like with any story with a sensational headline, one always has to read the body of the text to get the real story. And preferably, read the story from various sources to triangulate on the reality of the situation.
Looks like Amazon Web Services’ Elastic Compute service went down for an extended period this evening.
Now this was news to me, especially since we host our production study search service in the AWS EC2, and our replica delivery infrastructure too. Both the University of Michigan Network Operations Center (NOC) and Merit (NOC) monitor our systems, testing availability every minute of every day. And whenever there is an outage, the on-call engineer gets a page (or many pages!). And I'm the oncall this week. :-)
So what really happened?
Here's a piece of the story from the AWS Service Health Dashboard:
7:33 PM PDT We wanted to give you a quick update. A lightning storm caused damage to a single Power Distribution Unit (PDU) in a single Availability Zone. While most instances were unaffected, a set of racks does not currently have power, so the instances on those racks are down. We have technicians on site, and we are working to replace the affected PDU. We do not yet have an ETA, but we expect to be able to recover the instances when we restore power. Besides these affected instances, all other instances, and all other Availability Zones, are operating normally. Users with affected instances can launch replacement instances in any of the US Region Availability Zones or wait until their instance(s) are restored.
Some instances in one of AWS's availability zones (e.g., they have three for the US alone) failed. That's not wonderful news, especially if one of the failed instances belongs to you, but it is hardly a failure of the entire EC2 service.
To me this is somewhat like ICPSR messing up an entry in a database, making one of our studies unavailable by mistake, and someone blogging that ICPSR's on-line delivery service went down.
Net net for me: Like with any story with a sensational headline, one always has to read the body of the text to get the real story. And preferably, read the story from various sources to triangulate on the reality of the situation.
Wednesday, April 1, 2009
Weather report at ICPSR? Cloudy
We've been really impressed with the services available from Amazon Web Services. Amazon makes a handful of very nice Firefox Add-ons available that make it very easy to start using their services. Elasticfox is the main one we've been using; it takes only a few moments to install it, configure it with one's AWS credentials, and then to start launching and managing virtual machines. Of course, Amazon also makes it very, very easy to start using their AWS service. All one needs to do is create an account and supply a credit card number.
In February we upgraded to the latest version of Autonomy (nee Verity nee Inktomi) Ultraseek for our study/web search. We installed the software on a virtual machine in Amazon's Elastic Computing Cloud (EC2), their "computer virtualization" service so that we could make use of the search both from our primary web location in Ann Arbor as well as our back-up web locations in EC2 and (soon) San Diego. (More on the back-up web locations in a later post.)
No dobut this'll jinx the search service, but since we moved to the EC2 platform the search has worked very well; no service interruptions, no downtime. And keeping the search and index in one location (Amazon's cloud) and the content is it indexing in a different location seems to work just fine.
In addition to the production study/web search capability and our back-up web location, we've also been using EC2 for ad hoc, one-off computing requests. In the past we would have purchased new hardware, or re-purposed under-powered desktop hardware, but now we just launch the right sized virtual machine in the cloud, and we're ready to go in just minutes. And by making good use of Amazon's related products - Elastic Block Storage and Simple Storage Solution - we also have a reasonable disaster recovery story to tell.
In February we upgraded to the latest version of Autonomy (nee Verity nee Inktomi) Ultraseek for our study/web search. We installed the software on a virtual machine in Amazon's Elastic Computing Cloud (EC2), their "computer virtualization" service so that we could make use of the search both from our primary web location in Ann Arbor as well as our back-up web locations in EC2 and (soon) San Diego. (More on the back-up web locations in a later post.)
No dobut this'll jinx the search service, but since we moved to the EC2 platform the search has worked very well; no service interruptions, no downtime. And keeping the search and index in one location (Amazon's cloud) and the content is it indexing in a different location seems to work just fine.
In addition to the production study/web search capability and our back-up web location, we've also been using EC2 for ad hoc, one-off computing requests. In the past we would have purchased new hardware, or re-purposed under-powered desktop hardware, but now we just launch the right sized virtual machine in the cloud, and we're ready to go in just minutes. And by making good use of Amazon's related products - Elastic Block Storage and Simple Storage Solution - we also have a reasonable disaster recovery story to tell.
Subscribe to:
Posts (Atom)