ICPSR Web Availability - 2012-2013

Here are the final numbers for ICPSR's web site availability over our last fiscal year:

Click to embiggen

The year did not start off so well, and we reached the nadir quickly.  August 2012 was our worst period of availability in a very bad year for us overall.  January, March, and June 2012 also had very poor numbers.

The main antagonist we faced was a new and unusual problem with our Oracle database server.  For many years we would export the content for backup purposes each evening, and it worked well for a decade.  However, suddenly in 2012 we began to experience an outage just AFTER each export.  Despite intensive analysis by ourselves and local Oracle exports, we never could isolate the root cause of failure.

We eventually "solved" the problem by exporting our database only once per week v. once per day.  That left us more exposed to loss, of course, but it seemed to limit the outages to once per week v. once per day.

We then replaced the hardware with a new machine with a bit more processor and memory, but with blindingly fast solid-state drives. With the new machine deployed we returned to our daily export schedule, and the machine -- and our web availability -- have been in pretty good shape ever since. The machine went into service in April 2012, and the chart above makes it clear that life has been a little less hectic for our on-call engineer since then.

Web availability at ICPSR - March 2013

ICPSR's content delivery system showed very high availability in March 2013:  a bit over 99.95% uptime.  We had only two problems in March.  One was a power outage that affected our headquarters on the University of Michigan campus, and we experienced a small amount of downtime as we moved service to our replica in Amazon's cloud.  The second was a 21-minute outage due to a continuing -- but now solved, we think -- problem with exporting content from our Oracle database server.

Here are the overall numbers for ICPSR's 2012-2013 fiscal year:

click to enlarge

We replaced our aging Oracle database server with a new machine which has twice the memory, twice the computing power, and perhaps most impressively, has 300 times the disk I/O speed(!).  The new machine has an array of solid-state drives (SSDs), and we use this for all of our database storage.  (The operating system resides on conventional disk drive technology.)

Web availability at ICPSR - October 2012

October was a very good month for system uptime - over 99.9% availability:

Click chart to enlarge

That's good news after a much rougher September.  So far things look good this month, although a number of very short-lived outages have already pushed us below 99.9% for the month.

September 2012 deposits at ICPSR

The numbers from September are in:

# of files	# of deposits	File format
29	1	F 0x07 video/h264
145	17	application/msword
5	1	application/octet-stream
292	11	application/pdf
14	5	application/vnd.ms-excel
1	1	application/vnd.ms-powerpoint
120	1	application/x-arcview
31	1	application/x-dbase
1	1	application/x-rar
24	6	application/x-sas
14	6	application/x-spss
13	5	application/x-stata
5	5	application/x-zip
1	1	image/jpeg
32	1	image/x-3ds
70	2	multipart/appledouble
10	3	text/plain; charset=unknown
62	18	text/plain; charset=us-ascii
2	1	text/rtf
29	2	text/xml

Interesting month in that we have the usual stuff in the usual quantities, but we also have a large number of unusual formats hitting the doorstep, such as ArcView and Apple Double.  And we also have a usual format in an unusually high quantity (MS Word).

September 2012 web availability

September was an OK, but not great month for web availability:

Click to enlarge

We eliminated one frequent, but short-lived source of downtime when we stopped exporting the content of our Oracle database nightly.  We are now doing it only on the weekend, and while that adds some risk, we're gaining significant uptime.  (For some reason that we do not understand, our Oracle instance stops answering queries for 15-20 minutes about ten minutes AFTER the export completes.)  We have a new server racked and ready to install, and we're hoping that a fast new machine with solid-state drives will solve the problem for us.

We did run into some trouble mid-month when some routine maintenance went awry, and we had to fail over to our replica over the weekend of September 15 and 16.  The total amount of downtime was about 90 minutes total over the course of the weekend, but the replica kept the problem from clobbering our service completely.

After that we had pretty smooth sailing for the rest of the month.  Just 16 minutes of downtime for the rest of the month.

August 2012 deposits

Light month for deposits:

# of files	# of deposits	File format
56	30	application/msword
69	26	application/pdf
9	8	application/vnd.ms-excel
2	1	application/vnd.wordperfect
4	1	application/x-dosexec
24	7	application/x-sas
47	21	application/x-spss
7	2	application/x-stata
26	2	application/x-zip
1	1	image/gif
18	4	image/jpeg
1	1	message/rfc8220117bit
3	3	text/plain; charset=iso-8859-1
17	10	text/plain; charset=unknown
32	9	text/plain; charset=us-ascii
3	2	text/plain; charset=utf-8
39	7	text/rtf
1	1	text/x-mail; charset=iso-8859-1
4	2	text/x-mail; charset=unknown
2	2	text/x-mail; charset=us-ascii
1	1	text/xml

Just the usual stuff, but in pretty low quantities.

ICPSR web availability - August 2012

August was not our best month.

We did a bit better than 99.3% uptime.  Almost all of the downtime is due to a recurring, as-yet-unsolved problem we are having with our Oracle database platform.  The primary symptom is that the database platform stops fielding queries for about 5-15 minutes, which disables our production web site.  The platform does this about 30-45 minutes AFTER it has finished a full export using the Oracle datapump system.

Because our existing Oracle hardware is old and has a relatively slow disk I/O system, we're going to try to solve this problem by throwing hardware at it.  For well under $10k we can replace our five-year-old hardware with something much newer.  Goodbye RAID-5 SCSI, hello SSD.

July 2012 web availability at ICPSR

July 2012 was a pretty good month:

Clicking the image will open a larger, easier-to-read chart.

We only has 36 minutes of downtime in July, and 28 were due to maintenance as we tried (but failed) to update apache, perl, and mod_perl on our production web server.  We discovered some interesting idiosyncrasies in some perl libraries during the maintenance.  (Summary:  Multi-word time zones like "New York" are trouble.)

July 2012 deposits at ICPSR

The totals for July 2012:

# of files	# of deposits	File format
1	1	F 0x07 video/h264
5201	1	application/dicom
1	1	application/msaccess
112	23	application/msword
121	4	application/octet-stream
241	41	application/pdf
6	5	application/vnd.ms-excel
1	1	application/vnd.ms-powerpoint
1	1	application/x-7z-compressed
44	1	application/x-arcview
83	1	application/x-dbase
73	3	application/x-dosexec
5	1	application/x-empty
11	3	application/x-sas
1	1	application/x-shellscript
150	21	application/x-spss
23	6	application/x-stata
28	4	application/x-zip
10024	5	image/jpeg
8	1	image/png
4	1	image/x-ms-bmp
2	2	message/rfc8220117bit
15	1	multipart/appledouble
5	5	text/html
1	1	text/plain; charset=iso-8859-1
4	3	text/plain; charset=unknown
157	38	text/plain; charset=us-ascii
7	4	text/plain; charset=utf-8
7	4	text/rtf
4	3	text/x-mail; charset=us-ascii
43	4	text/xml
25	1	video/unknown

Lots of image content this month to go with the usual stuff (e.g., SPSS) in the usual volumes.

ICPSR 2012 technology recap - where did the money go?

A post from a week or so ago showed the sources of money flowing into the technology organization at ICPSR.  This week's post will focus on how that money is spent.

I should note that the focus of this post is on what we call the "Computer Recharge" at ICPSR.  This is a tax paid by all FTEs at ICPSR that is levied on an hourly basis.  Each time an employee completes his or her timesheet and allocates time to a project (exceptions: sick, holiday, and vacation time), a small amount of money also accumulates in the "Recharge."  In FY 2012 this accounted for about 45% of the technology revenue.

Unlike a direct charge for technology where a project or grant may be paying for a dedicated server, extra storage space, or a fraction of a software developer working on custom systems, the Computer Recharge dollars are used to fund technology expenses that benefit the entire organization.  This includes expenses as prosaic as printers and desktop computers, but also includes systems and software development for our delivery systems, ingest systems, and digital preservation systems.

Here's the breakdown in chart form:

No surprise that most of the money goes to pay people:  systems administrators, desktop support specialists, software developers, and a group of very hands-on managers who do a lot of the same work plus project management and business analysis.  This accounts for $862k out of $1218k.

Equipment ($214k) represents desktop machines, new storage capacity, printers, virtual machines, cloud storage, software licenses, and almost every non-salary expense.

Transfers ($103k) is an interesting category.  This is money that we collect as major systems depreciate.  But since the U-M doesn't really "do" depreciation, we use this interesting process instead:

1. Buy the item (using money from Equipment pot)
2. Estimate the lifetime of the item (say, five years)
3. Collect 1/5 of the purchase price each year for five years
4. At the end of each year, move the 1/5 collected into Transfers
5. When the item needs to be replaced, use the money in the Transfers pot

So the $103k represents money that was collected in FY 2012 for items that were purchased in earlier years, and which will need to be replaced in FY 2013 or beyond.

All of our other expenses are tiny by comparison:  $19k for Internet access, $7k for telephones and service monitoring, $6k for travel, $5k for maintenance contracts, and $2k for miscellaneous fees and expenses.

One take-away from this is that the essential element in technology budgeting isn't the purchase price of the server, or the annual cost of cloud storage, bur rather the recurring cost of the people who will build, maintain, enhance, and customize the technology portion of the business.

ICPSR 2012 technology recap - where did the money come from?

We're putting together some summary numbers for technology spending and investments at ICPSR for FY 2012.  (The ICPSR fiscal year is the same as the University of Michigan's, and runs from July 1 to June 30.  We've just recently closed FY 2012.)

The first set of numbers shows the allocation of effort in FY 2012 by funding source. The unit of measurement in this pie chart is HOURS (not DOLLARS) that were expended in FY 2012 by each funding source.  (We originally wanted to calculate dollars, but that turns out to be an even bigger effort.)  Here's an interactive chart:



This is an interactive Google Docs chart.  If you click slices of the pie, it will identify the funding source.

The main source of technology effort funding comes from the Computer Recharge, an hourly "tax" that ICPSR levies against all projects.  Although it is one single funding source (nearly 45% of hours worked in FY 2012 were billed against this source), I have split it into two sub-categories, one for what I am calling "IT" and one for "SW" (software).

The "SW" portion includes the effort of all staff who are professional software developers.  The type of work performed by this team using this account includes enhancements and maintenance for ICPSR's core data curation and data management systems, and investments in new products and services such as software developed to support our IDARS system for applying for access to datasets.

The "IT" portion includes the effort of the remainder of the staff which tends to include systems administrators, architects, network managers, and desktop support specialists.  I also allocate my own time to this bucket since the majority of my non-contract, non-grant effort over the past year has been in building and architecting technology systems.

Other big slices of the "IT pie" include the work of staff members who are explicitly funded by projects such as our CCEERC and RCMD web portals; our two Bill and Melinda Gates Foundation grants; the ICPSR Summer Program, and many more.  In fact, there are over 20 separate funding sources used to support technology at ICPSR; the pie chart shows 18 because I grouped several small ones into a category called "Misc."

If this gives the impression that there are many, many projects and activities at ICPSR that involve technology, that's good!  That is certainly the case.

However, if "focus wins" then we're in a little bit of trouble.  My sense is that each of this 20-some funding sources has at least one unique project with its own business analysis and project management needs, and it is sometimes the case that different projects have antithetical technology needs.  I see this play out in all phases of the OAIS lifecycle.  ("I want you to build a system that makes it as easy as possible to fetch datasets from ICPSR" v.  "I want you to build a system that requires significant effort and oversight to fetch datasets from ICPSR.")

June 2012 deposits at ICPSR

Deposit numbers from June:

# of files	# of deposits	File format
1	1	video/h264
368	17	application/msword
3	1	application/octet-stream
276	26	application/pdf
9	4	application/vnd.ms-excel
1	1	application/vnd.ms-powerpoint
1	1	application/x-rar
8	3	application/x-sas
118	16	application/x-spss
713	8	application/x-stata
1	1	image/jpeg
6	1	image/x-3ds
22	1	multipart/appledouble
4	4	text/html
36	1	text/plain; charset=iso-8859-1
31	3	text/plain; charset=unknown
389	10	text/plain; charset=us-ascii
1	1	text/plain; charset=utf-8
4	3	text/x-mail; charset=us-ascii
1	1	video/unknown

Quite a bit of Stata this month, much more than normal.

Amazon Web Services makes Tech@ICPSR weep

June 2012 was looking to be a great, great month for uptime.  We were on track to have our best month since November 2011 this fiscal year - just 60 minutes of downtime across all services and all applications.  It was going to be beautiful.

And then Amazon Web Services had another power failure.

And then we wept.

The power failure took the TeachingWithData portal out of action.  (To be fair, it was already having significant problems due to its creaky technology platform, but this took it all the way out of action.)  The failure also took our delivery replica out of action, and gave Tech@ICPSR the joy of rebuilding it over the weekend.

But the real trouble was with a company called Janrain.

Janrain sells a service called Engage.  Engage is what allows content providers (like ICPSR) to use identity providers (like Google, Facebook, Yahoo, and many more) so that their clients (like you) do not need to create yet another account and password.  Engage is a hosted solution that we use for our single sign-on service using existing IDs, and it works 99.9% of the time.

However, this hosted solution lives in the cloud.  We just point the name signin.icpsr.umich.edu at an IP address we get from Janrain, plug in calls to their API, and then magic happens.

Except when the cloud breaks.

Amazon took Engage off-line for nearly four hours.  And then once it came back up, it was thoroughly confused for another three hours.  Ick.

So, counting all of that time as "downtime" our fabulous June 2012 numbers suddenly became our awful June 2012 numbers.  Here they are:

If you click on the image above, Blogger will make it bigger.

Of course, during a lot of that downtime, all of the features on the web site except for third-party login worked fine.  And most of the problem happened late on a Friday night and Saturday morning during the summer, so that's a good time for something bad to happen, if it has to happen at all.

May 2012 deposits at ICPSR

Stats?  Stats.

# of files	# of deposits	File format
1	1	application/msaccess
564	22	application/msword
334	3	application/octet-stream
136	22	application/pdf
4	3	application/vnd.ms-excel
16	1	application/x-arcview
4	1	application/x-dbase
31	7	application/x-sas
281	19	application/x-spss
17	3	application/x-stata
5	3	application/x-zip
18	7	image/jpeg
2	2	message/rfc8220117bit
4	3	text/html
4	4	text/plain; charset=iso-8859-1
125	3	text/plain; charset=unknown
332	20	text/plain; charset=us-ascii
1	1	text/plain; charset=utf-8
9	3	text/rtf
1	1	text/x-makefile; charset=us-ascii

Nothing too interesting this month.  We have the usual formats, and in the usual proportions.  We did seem to get an unusually large number of MS Word files last month, and we also have a pretty large set of unidentified files (at least in terms of MIME type).

May 2012 web availability

Web availability was good, but not great, in May 2012:

Click to enlarge

Five main episodes of 25 to 59 minutes account for almost all of the 242 minutes of unavailability in May 2012.

On May 13 the production web server seemingly lost power, and it required an additional reboot and some TLC to bring the system back on-line fully (33 minutes).

On May 18 the search index on our CCEERC web portal became corrupted, and that disabled much of the usefulness of the site for nearly an hour (49 minutes).

On May 22 we saw the first of two episodes where the proxy (AJP) between Apache httpd and Apache tomcat faulted.  This did not recover on its own and required some help from the technology team.  This resulted in a medium-duration outage (27 minutes), and another similar fault occurred on the evening of May 31 (24 minutes).

On May 31 our production database server faulted, requiring a manual power-cycle, and also requiring the production web server to be rebooted (59 minutes).

My sense is that while we're in much better shape with regard to the instability caused by khugepaged, we are starting to see something a little amiss with the Apache proxy system.  It isn't clear to us at this time if the issue is faulty software or sub-optimized configuration on our part.

ICPSR system outage - 5/31/2012

ICPSR's content delivery systems faulted at approximately 8:30pm EDT on Thursday, May 31, 2012.  The oncall engineer discovered that the production Oracle database server had become unresponsive, and this disabled most features of most of our web portals.

After arriving on-site she rebooted the database server, but by then the production web server had become hopelessly confused.  She then rebooted that system as well, and all systems were back in service a bit before 9:30pm EDT.

The ICPSR technology team is reviewing system logs and access records to see if any further corrective action is required.

Our apologies for the inconvenience this no doubt caused to many of you.

April 2012 deposits at ICPSR

Chart?  Chart.

# of files	# of deposits	File format
177190	6	application/dicom
49	23	application/msword
157	6	application/octet-stream
184	36	application/pdf
8	4	application/vnd.ms-excel
1	1	application/x-7z-compressed
1717	7	application/x-dosexec
6	2	application/x-empty
13	4	application/x-sas
21	15	application/x-spss
3	3	application/x-stata
7	7	application/x-zip
6	2	image/gif
157948	13	image/jpeg
208	6	image/png
104	6	image/x-ms-bmp
6	6	message/rfc8220117bit
5	2	multipart/appledouble
26	7	text/html
2	2	text/plain; charset=unknown
371	35	text/plain; charset=us-ascii
104	6	text/plain; charset=utf-8
59	12	text/rtf
416	6	text/xml

Things look pretty light in terms of the usual formats, but a handful of big, big deposits stick out.

DICOM - Digital Imaging and Communications in Medicine - is a format used for medical imaging, and we received well over 150k such files last month plus an equally large batch in the more prosaic JPEG format.  I'm not sure how we'll be curating and delivering this content.....

Related to this same set of deposits is a large number of Windows executable files (EXE and DLL file extensions) which will also be an interesting challenge for delivery.

April 2012 Web availability

April was a much, much better month for our systems:

Click to enlarge

The real game changer seemed to be disabling the transparent hugepage system on our RHEL 6 systems.  Once we did that, our fortunes changed for the better.  And so we sing:

The main culprits behind the small amount of downtime we had in April were a misfire during an attempt to introduce yet another rewrite rule to our Apache httpd config (which is always risky), a filesystem filling up on the production web server which tanked the search engine for nearly thirty minutes, and a brief outage with Janrain's Engage service, which allows people to use their Facebook ID or Google ID to sign in to the ICPSR web site.

March 2012 Web availability

March 2012 was not kind to us.

Clicking the image will display a full-size chart.  But please don't.  It is too ugly.

The main culprit in March was a continuing problem with the reliability of the production web server.  The environment - cooling, electricity, humidity - was fine, and the individual web applications were also fine, but something is not quite right with the kernel.  I think.  (If you would like to join the team as our new Senior Systems Architect and help solve the problem, see my post from last week.)

In March we saw multiple outages, each lasting over an hour.  The script always went something like this:

1. Load average increases by 5000-10000%
2. One web application stops responding and logging
3. KERN.INFO error messages from khugepaged and jsvc appear in syslog
4. Attempt to restart web application
5. Fail
6. Attempt to restart all web apps and their containers
7. Fail
8. Attempt to reboot machine
9. Fail
10. Optional:  Drive into office if weekend or early morning
11. Attempt to cycle power on machine
12. Mix of foul language and prayer
13. Repeat step #12
14. Success - machine is working again

Because we use the cloud instead of local, physical servers for many services, and because we haven't had all that many times where the machine needed its power cycled to solve a problem, we don't have things set-up for remote power access.  We'd like to address that.  (If you would like to join the team as our new Senior Systems Architect and help solve the problem, see my post from last week.)

So here's the plan to have a better April:

1. Disable khugepaged, hoping this might stop the machine from seizing up
2. Drive faster to the Perry Building, hoping this might result in faster applications of turning the power off and on
3. Hire the Senior Systems Architect, hoping that having a third pair of eyes on the problem might reveal its true cause and solution
4. Mix of foul language and prayer, hoping it will ease the pain

And, more seriously, we have also updated a few apps (like Solr) to use local storage rather than NFS-mounted storage for their work, particularly if the app tends to do a lot of writing to the filesystem.  NFS seems to be part of the mystery too.

March deposits at ICPSR

Chart?  Chart.

# of files	# of deposits	File format
112	30	application/msword
9	3	application/octet-stream
81	27	application/pdf
3	2	application/vnd.ms-excel
4	2	application/vnd.wordperfect
22	14	application/x-sas
83	21	application/x-spss
3	3	application/x-stata
2	2	application/x-zip
2	2	image/jpeg
1	1	image/x-3ds
1	1	message/rfc8220117bit
9	4	text/html
5	2	text/plain; charset=unknown
87	16	text/plain; charset=us-ascii
3	2	text/rtf
1	1	text/x-c; charset=unknown
7	1	text/x-c; charset=us-ascii

A blissfully normal month of deposits.  Usual types.  Usual volumes.

Still need to tweak the automated MIME type detector to stop reporting that it is finding C source code.  The eight files above are most likely plain text files that just happen to have something like a pound-sign or "slash-star" sequence starting in the first column.

Not shown here - because it isn't passing through the deposit system - is a considerable volume of video content from the Gates Foundation.  We have a bit over 6TB that we received in early 2012, and about 1TB of a 20TB collection that will arrive in a steady stream over the next 12-16 months.

If our policy is that the ICPSR deposit system is just one of many mechanisms for ICPSR to accept content, then this seems OK.

But, if we expect the deposit system to be the complete and correct record of ALL incoming content, then we do have a problem.  A 7TB problem that is will grow up to be a big and strong 26TB problem at some point.