AWS power outage aftermath

As it turns out, it doesn't take all that long to run fsck on a large filesystem comprised of multiple AWS Elastic Block Storage (EBS) volumes:

[root@cloudora ~]# df -h /dev/md0
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              4.9T  2.9T  1.7T  64% /arcstore 

[root@cloudora ~]# fsck -y /dev/md0
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
/dev/md0 is mounted.
WARNING!!!  Running e2fsck on a mounted filesystem may cause
SEVERE filesystem damage.
Do you really want to continue (y/n)? yes
/dev/md0 has gone 454 days without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Error allocating icount link information: Memory allocation failed
e2fsck: aborted

I saw a few posts about coaxing e2fsck to use the filesystem for scratch space rather than memory, but unfortunately the older version of the program available on this EC2 instance does not support it.

So I think that we may end up blowing away this copy of archival storage and replacing it with a fresh one.