ICPSR web availability through November 2011

Now that's what I'm talking about!  (Click the image to see a more readable chart.)

ICPSR FY 2012 web availability through November 2011

After some shamefully low availability numbers in September and October, we've rebounded nicely in November (over 99.9%).

We saw two main problems in the month. 

One was a short outage where our search engine (Solr) faulted and required a restart.  We think this is due to a memory leak in Solr, and we are hoping that we can avoid the problem more completely once we move from our older 32-bit web hardware to our new 64-bit machine.  I'm hoping this happens before the end of the calendar year.

The other outage was due (we think) to a campus power blip that seemed to cause a fault with our EMC storage appliance.  While ICPSR never lost power, and while the machine room has a large, new UPS system, we speculate that the EMC got confused when it lost contact with UMROOT Windows Domain Controllers across campus due to the network path fluctuating.  The problem solved itself after 15 minutes, and it was the only anomaly that was coincident with the EMC hanging.