ICPSR 2012 technology recap - where did the money go?

A post from a week or so ago showed the sources of money flowing into the technology organization at ICPSR.  This week's post will focus on how that money is spent.

I should note that the focus of this post is on what we call the "Computer Recharge" at ICPSR.  This is a tax paid by all FTEs at ICPSR that is levied on an hourly basis.  Each time an employee completes his or her timesheet and allocates time to a project (exceptions: sick, holiday, and vacation time), a small amount of money also accumulates in the "Recharge."  In FY 2012 this accounted for about 45% of the technology revenue.

Unlike a direct charge for technology where a project or grant may be paying for a dedicated server, extra storage space, or a fraction of a software developer working on custom systems, the Computer Recharge dollars are used to fund technology expenses that benefit the entire organization.  This includes expenses as prosaic as printers and desktop computers, but also includes systems and software development for our delivery systems, ingest systems, and digital preservation systems.

Here's the breakdown in chart form:

No surprise that most of the money goes to pay people:  systems administrators, desktop support specialists, software developers, and a group of very hands-on managers who do a lot of the same work plus project management and business analysis.  This accounts for $862k out of $1218k.

Equipment ($214k) represents desktop machines, new storage capacity, printers, virtual machines, cloud storage, software licenses, and almost every non-salary expense.

Transfers ($103k) is an interesting category.  This is money that we collect as major systems depreciate.  But since the U-M doesn't really "do" depreciation, we use this interesting process instead:

1. Buy the item (using money from Equipment pot)
2. Estimate the lifetime of the item (say, five years)
3. Collect 1/5 of the purchase price each year for five years
4. At the end of each year, move the 1/5 collected into Transfers
5. When the item needs to be replaced, use the money in the Transfers pot

So the $103k represents money that was collected in FY 2012 for items that were purchased in earlier years, and which will need to be replaced in FY 2013 or beyond.

All of our other expenses are tiny by comparison:  $19k for Internet access, $7k for telephones and service monitoring, $6k for travel, $5k for maintenance contracts, and $2k for miscellaneous fees and expenses.

One take-away from this is that the essential element in technology budgeting isn't the purchase price of the server, or the annual cost of cloud storage, bur rather the recurring cost of the people who will build, maintain, enhance, and customize the technology portion of the business.

ICPSR 2012 technology recap - where did the money come from?

We're putting together some summary numbers for technology spending and investments at ICPSR for FY 2012.  (The ICPSR fiscal year is the same as the University of Michigan's, and runs from July 1 to June 30.  We've just recently closed FY 2012.)

The first set of numbers shows the allocation of effort in FY 2012 by funding source. The unit of measurement in this pie chart is HOURS (not DOLLARS) that were expended in FY 2012 by each funding source.  (We originally wanted to calculate dollars, but that turns out to be an even bigger effort.)  Here's an interactive chart:



This is an interactive Google Docs chart.  If you click slices of the pie, it will identify the funding source.

The main source of technology effort funding comes from the Computer Recharge, an hourly "tax" that ICPSR levies against all projects.  Although it is one single funding source (nearly 45% of hours worked in FY 2012 were billed against this source), I have split it into two sub-categories, one for what I am calling "IT" and one for "SW" (software).

The "SW" portion includes the effort of all staff who are professional software developers.  The type of work performed by this team using this account includes enhancements and maintenance for ICPSR's core data curation and data management systems, and investments in new products and services such as software developed to support our IDARS system for applying for access to datasets.

The "IT" portion includes the effort of the remainder of the staff which tends to include systems administrators, architects, network managers, and desktop support specialists.  I also allocate my own time to this bucket since the majority of my non-contract, non-grant effort over the past year has been in building and architecting technology systems.

Other big slices of the "IT pie" include the work of staff members who are explicitly funded by projects such as our CCEERC and RCMD web portals; our two Bill and Melinda Gates Foundation grants; the ICPSR Summer Program, and many more.  In fact, there are over 20 separate funding sources used to support technology at ICPSR; the pie chart shows 18 because I grouped several small ones into a category called "Misc."

If this gives the impression that there are many, many projects and activities at ICPSR that involve technology, that's good!  That is certainly the case.

However, if "focus wins" then we're in a little bit of trouble.  My sense is that each of this 20-some funding sources has at least one unique project with its own business analysis and project management needs, and it is sometimes the case that different projects have antithetical technology needs.  I see this play out in all phases of the OAIS lifecycle.  ("I want you to build a system that makes it as easy as possible to fetch datasets from ICPSR" v.  "I want you to build a system that requires significant effort and oversight to fetch datasets from ICPSR.")

An open letter to our storage system

Dear Storage System:

I think it is time that we had a talk.  Not a friendly chat, but a serious, heart-to-umm-bus talk.  About real issues.

You know, we used to have one of your older cousins live with us.  Your cousin's official name was just a bunch of letters and numbers, but we always said "the Naz" for short.  Kind of like "the Fonz."

The Naz was sweet.  Everyone loved the Naz.  When the Naz felt the least bit sick, in a flash there would be a call home, and then before you could say, "broken disk drive," some dispatcher would be on the phone with us, offering to bring out a replacement part that same day.  Sometimes we'd even get the phone call before we knew there was a problem.  Those were the times!

Sure, some people on campus made fun of us, but we loved the Naz anyway.  "Hey, why do you sepnd so much on storage for your Naz?  Don't you know you can buy storage from us way, way cheaper?"  Yes, they would say such things.  Other storage providers can be so cruel.

But we didn't care.  We loved our Naz.  Dependable, reliable, but, sure, a little expensive.  But we thought that we were spending our money wisely on the Naz.  Not a bit of trouble.  We loved our Naz.

But, as all good things go, so did the Naz.  Parts got old.  The cabinet got dusty.  Sure, a 300GB disk drive seemed big enough back in 2005, but now it just seemed, I don't know, quaint.  And, you know, the Naz never had a second data mover....

And, so, after a lot of planning, and a lot of work, we put the Naz out to pasture.  The Naz isn't storing and serving data like in the old days; it just mills about the machine room, chewing on electricity, and soaking up the air conditioning.  Visitors come by to visit, and they never stay with the Naz for very long.

Of course, as we were reading sweet bedtime stories to the Naz, we were grooming you for the job.  You were new and shiny.  We didn't know you very well, but we new our Naz, and we knew you were supposed to be just like our Naz.  But newer.  And better.  And with more blue lights.  And we were excited.

Things started well.  In many ways you were just like the Naz, but better.  Your disk drives were hefty.  You had newer software.  You even seemed a little faster, just like your car does after you wash it.  This was great.

But then the behavior problems started.  You know what I mean.

Like the time that you knew full well that there was a problem, but did you call home for help?  No.  You made us do it.  Why?  Why wouldn't you use the nice telephone line we left for you?

And, sure, after we finally convinced you to call home, you then told the foulest lies.

"We've tried to log in to your system, but the password doesn't work!" the techs would say.  Why did you give them the wrong password?  Do you think this is a game?

"Please download DiskDebunker v4.3 from our mirror site.  Install if on a Windows ME machine, configured for use on a private network, and use it to assess if the storage processor valve flanges are flush.  This will produce a 700GB file called DataGrommit.zip, and you should then upload that to the Easy Web support site." they would say other times.  I can only imagine the wild tales you must have told to confuse them so.  Why couldn't you just be honest and authentic, and tell them that one of your disk drives faulted, and all we needed was a replacement?  Why couldn't you be more like the Naz?

"I have read and understand all site messages." said the service requests.  Yeah, sure.  Why did our little troublemaker say this time?

This behavior must stop!

We're scheduling a little something that some people call "an intervention."  We want you to hear the problems that your mischief is causing.  We want you to hear the stories.  We want you to hear it from the Naz.

We don't want it to end this way.  We want it to work.  But we've had enough.  Don't make us call the people at Property Disposition to come get you.  You won't like where they will take you.

Let's give it one more try.

Your friends @ ICPSR.

IT Recharge Projects - 2010 Q1

The steering committee selected seven projects for attention in Q1 of 2010. Here's the list with a brief description of each one.

One, Restricted Use Contracting System. This is an on-going project to build a suite of tools to support the contracting process for access to restricted-use research data. There's a portal where researchers complete tasks such as uploading documents (research plan, IRB approval, CVs, etc), submitting security plans to protect the data, and enumerating who will have access to the data. There's also an internal administrative tool for managing contracts, and an automated workflow system that generates alerts and reminders.

Two, Turnover Notification to Depositors. This is a small project that closes the loop with a depositor when his/her data results in the release of an ICPSR study.

Three, Citation Data Feed. This is another small project that will feed our bibliography to partner organizations via an automated system.

Four, Expose Deposited Files via Deposit Viewer. This is a slightly larger project which updates our internal Deposit Viewer/Manager tool so that one may use it to access the payload of a deposit. Currently staff must use a separate mechanism to accomplish this.

Five, SAMHDA Site Redesign. Another medium-size project to retool the SAMHDA site to our new technology platform.

Six, Variable Metadata Management System Design. A preliminary project to scope how we might design and build a variable-level metadata management system to complement our study-level metadata management system.

Seven, Invoice Management. Our parent organization, the Institute for Social Research, is our de facto accounts receivable office, and this is a project to analyze the process by which information flows between Organization Representatives, ICPSR, and ISR.

ICPSR's IT Recharge

ICPSR supports its IT infrastructure through the use of a recharge mechanism. Here's a blurb from an overview document I wrote:

ICPSR hosts an array of both government- and member-funded activities and projects. While each has its own clients, goals, and areas of specialization, they all share the same essential technology needs. To deliver efficient and effective data products and services to its constituency each project requires access to tools, systems, and technical resources that ingest, curate, preserve, and deliver research data. ICPSR has long recognized this shared need, and operates a Recharge Center (RC) to deliver these services.

The RC is operated by ICPSR's Computer and Network Services team, which is comprised of systems analysts, software developers, systems and network administrators, certified IT security professionals, and a management team. The RC is funded through an annually reviewed University of Michigan Recharge account. The RC operates via this recharge mechanism, allocating expenses equally regardless if the client is a federal grant or contract, a membership-funded project, or an activity funded through internal University of Michigan funds. The rate does not discriminate between federally and non-federally funded activities.

Recently ICPSR began a new process to allocate RC-funded resources that conduct business analysis and software development. Here's how it works:

During the course of normal business each standing committee at ICPSR (e.g., the Dissemination Committee, the [Research Data] Processing Workflow Committee, etc) records a list of technology projects it would like to see undertaken. The head of each committee works with someone from my team to scope and size each project. Those lists are prioritized, and then all of the committee heads meet together once per quarter to create a single, consolidated list of projects. We manage the size of the list such that the amount of work for the quarter is equal to the resources available in the quarter.

My team then commits to working each project in the upcoming quarter. Some of the projects are short-lived, and can be completed within the quarter; others are long-lived and may span quarters. In this latter case, only one part of a bigger project would be completed in a single quarter.

I thought I would dedicate one post per quarter (starting with this one) where I would list the set of projects, and describe each one briefly. In some cases this will provide a "sneak preview" of new products and services that will roll out in the months to come.