Sixteen products or one?

A recent conversation with Nathan Adams, ICPSR's Assistant IT Director for Software Development got me thinking about this....

It's no secret that ICPSR uses a package called Survey Documentation and Analysis (SDA) from UC Berkeley as our on-line analysis system.  But people may be surprised to learn that this one product forms the underpinnings of more than a dozen closely related ICPSR on-line analysis products.

One, Anonymous Analysis : This is where we make a dataset available via SDA and there is no authentication allowed.

Two, Authenticated Analysis : One must authenticate using MyData, Google, or Facebook.

Three, Member Analysis :  One must authenticate and also be using a computer located on the campus (even virtually) of a member institution.

Four, Private Analysis : One must authenticate and the identity used must be a member of a previously created group of identities.

Five through eight, Secure Analysis : Like any of the options above, but where the raw, proprietary, binary data files reside on a separate server, and where the ICPSR web server accesses the content via HTTPS rather than through the filesystem.

Nine through Sixteen, Non-disclosed Analysis : Like any of the eight options above, but where SDA's disclosure.txt controls have been used to attempt to prevent unintentional disclosure.

So sixteen different combinations!  And it is easy to imagine even more cropping up in the months ahead.

My experience is that one ends up with sixteen different online analysis "products" when things grow organically over time.  When things evolve due to a small tweaks in response to requests like, "Hey, could we use SDA for this, but with just one small change ..... ?"

It is easy to see how it happens.  But when things grow over time like this, they end up suffering from a profound lack of design, and end up costing more to maintain.  They are fragile.  They break when you change things, like the hardware.  Or the OS.  Or the NAS.  Or the authentication scheme.  Or the oil in your car.

So probably time to pull back a bit, pull together a team of content owners, and start asking some questions.

If we were going to start fresh today with an on-line analysis system, what should we build?

What sort of access controls are needed to prevent bad guys from using it?

What sort of disclosure mitigation capabilities are required to prevent accidents from happening?

To which populations might we need to restrict access?

What does the user experience look like?  Is this geared for the novice or for expert-in-a-hurry?  Or do we have multiple audiences and so need to build more than one experience?

Time to design.

Job posting - again

We're posting a job description for a senior software developer for a third time.  If there is a recession in the IT business in SE Michigan, somebody forgot to tell our pool of potential applicants.

This position is very much like other software developer positions at ICPSR.

In practice there is a blend of business analysis, system design, software development, and second-line on-going support for the stuff you write.  Building stuff in java to run under tomcat is a must.  Experience with Oracle, Eclipse, and one or more frameworks is very useful.

The main support for this position is our Bill and Melinda Gates Foundation MET Extension project.  This is a two year grant to build systems that will delivery video content to researchers in the social sciences and education.  We've had a lot of success - like a 0% failure rate - at hiring people to work on project X, and then moving them over to project Y a few years down the road.  Project X is in this case is MET Extension; project Y is unknown.  But so was the MET Extension project 10 months ago...

If you have interest in the position and would like more info, please feel free to drop me a note.  And here's a short-lived, but direct, link to the job http://umjobs.org/job_detail/66372/software_developer_senior

Provenance metadata and the OAIS Receive Submission

The FLAME (File-Level Archival Management Engine) project continues to articulate functional requirements for the software system.  So far the process has looked something like this:

1. Select one of the functional areas of OAIS
2. Drill down into one of the sub-functions within that area
3. Enumerate a list of high-level statements that should be true for that sub-function
4. Translate those high-level statements into medium-level specifications for the software

For example, we tackled one such area before the recent holiday break:

1. Ingest
2. Receive Submission
3. The producer provided basic provenance information at deposit

Of course, this raises the question:  What do we consider "basic provenance information" at the time of deposit?  What information can we collect from the deposited content, and what information do we need to collect from the person performing the deposit?

Here is the draft list we created:

a. FLAME should enable the ability to transfer digital content to ICPSR through web-based file upload

i. Uploading files should NOT require MyData authentication
ii. The act of uploading files serves as a signature for the transfer

b. FLAME should enable the ability to document receipt of content transferred to ICPSR through non-electronic means

i. Date package arrived (required)
ii. Shipping company (required)
iii. Tracking ID number (required)
iv. Other details about the shipment (optional)

c. FLAME should capture the following provenance information from the content provider:

i. Self-reported identity of the content provider, or identity from MyData profile (for electronic transfer)

1. Name of depositor (required)

ii. Self-reported contact information for the depositor, or from MyData profile

1. Email-address (required)
2. Telephone number (optional)
3. Mailing address (optional)

iii. Self-reported descriptive provenance information from the depositor

1. Name or title of deposit (required)
2. Summary or description of the deposit (optional)
3. Name of organization that sponsored the research, or "not applicable" (required)
4. Number of ID of the grant or contract, or "not applicable" (required)

d. FLAME should capture the following provenance information from the files after each content transfer:

i. Date and time at which each file is received
ii. Checksum of each file
iii. MIME type of each file
iv. Original name of each file
v. Packaging information (e.g., file was part of a Zip archive)

What do you think basic provenance information should include?  Does our list look like it captures everything one could reasonably expect to collect at the time of deposit?

Starting the FLAME

In an earlier post I described a major new project at ICPSR called FLAME.  FLAME is the File-Level Archival Management Engine, and will become the new repository technology platform ICPSR uses to curate and preserve content.  As the name implies, the main molecule of information upon which FLAME will operate is a "file" which is different than the main molecules used at ICPSR today, the "deposit" and the "study."  In the big picture the activities at ICPSR will not change much:  we will still collect social science research data, curate them, preserve them, and make them available in a wide variety of formats and modes.  But when one looks at the details, an awful lot will change.

So when one is going to change everything, where does one start?

Fortunately we have a ready-made starting point with the Open Archival Information System (OAIS) reference model.  While this does not give us a blue print of what to build, it does give us a model to use as we construct our blueprints.  I believe this is very much what the folks at Archivematica have done.

So the question becomes:  How do we translate a high-level reference model that contains functions such as Receive Submission to the low-level blue prints one needs to reconfigure process and build software?  What kind of web applications do I need for Receive Submission?  What should they do?  Should that box that contains the submitter's identity be an email address?  A text string?  An ORCID?

So how to start?

One of my colleagues, Nancy McGovern, suggested we brainstorm 6-12 medium-level statements for each of the functions in the OAIS reference model.  We started with Receive Submission, and indeed generated 12 statements.  (The analogue at Archivematica is Receipt of SIP.)  One example is:

The producer provided basic provenance information at deposit

If the metaphor for building FLAME is building a house, then OAIS plays the role of high-level best practices.  The statements (like above) play the role of floor plans and elevations; those things to which most people can relate and make decisions.  So this is moving in the right direction, but we're still lacking the blueprints.

The next step is to take a statement like the one above and turn it into requirements for software (and for process).  One example requirement that flows from the statement above is:

FLAME should capture the following provenance information from the files after each content transfer:

i. Date and time at which each file is received

ii. Checksum of each file

iii. MIME type of each file

iv. Original name of each file

v. Packaging information (e.g., file was part of a Zip archive)

We can then discuss these low-level requirements with stakeholders, such as the acquisitions team, and with the technology team, such as a software developer who may have additional questions (e.g., "Well, what sort of checksum do you want - MD5, SHA-1, or something else?").

Right now we are working through the details of Receive Submission, and the next few stops on the roadmap will likely be in Ingest as well.  We're documenting both the high-level statements and the low-level requirements in a Drupal CMS that we use as our Intranet.

Collaborators, not depositors

ICPSR should stop accepting deposits.

Instead ICPSR should be recruiting collaborators.

To be sure ICPSR receives a great deal of its content via US Government agencies who have decided to outsource the digital preservation of their content to a trustworthy repository like ICPSR.  In this case the relevant contract, grant, or inter-agency agreement makes it clear what content will be coming to ICPSR to be curated and preserved.  In some cases the agency has little interest in depositing content ("Isn't that what we pay you for?"), and so the formal act of depositing content falls to the ICPSR staff anyway.

However, we also receive a considerable volume of content through our web portal where the depositor is external.  Sometimes we have worked hard to acquire the content, and the deposit is one milestone on a very long road, but other times the content comes to us unsolicited.  (I like to call these "drive-by deposits.")

In some cases the depositor is quite eager and able to help ICPSR with much of the curation work:  drafting rich descriptive metadata; organizing survey data and documentation into coherent groups; packaging other types of content into logical bundles (such as with our Publication-Related Archive); and, reviewing the data for possible disclosure risks.  Depositors may have access to resources like graduate students who can help with these tasks, and if the depositor is also the data producer, then s/he has valuable, unique insight into the data and documentation.  Unfortunately ICPSR is not well poised to tap into that expertise and those resources.

What would it take to get there?

ICPSR could separate the transactional step of submitting content (i.e., file upload concurrent with signature) from the iterative step of preparing metadata applicable to the submitted content.  In fact, one could even prepare metadata well before the submission transaction if the data producer had the interest and resources to prepare that information, but was not quite ready to share the data yet.  And, it would be equally permissible to submit the data for preservation and sharing, and then build the metadata slowly during the weeks and months following the upload.

If the data producer could also export the metadata in machine actionable formats, say, DDI XML for content which maps well to the classic "study" object that ICPSR has curated and preserved for decades, then there may be additional value to the producer. And introducing the structure that comes along with an XML schema like DDI might also be valuable to the producer in terms of thinking about and organizing the documentation, even for his/her own use.

In this world the ICPSR deposit system becomes a much shorter, much simpler web application.  And the ICPSR data management infrastructure would need to be opened up -- but with serious access controls -- so that content providers could access, create, and revise their documentation and metadata.  But the best thing about this world is that ICPSR gains a lot of collaborators, some who would be quite eager to work with us, I think.

ICPSR launches the Restricted Contract System portal

The ICPSR Restricted Contract System (RCS) portal is officially open for business. We launched the new portal late in August, making the National Survey of Parents and Youth available through the system.

The portal is the researcher-facing piece of the system, and we use it to guide the researcher through the contract process of applying for access to restricted-use data. The system is highly configurable which allows us to collect information through traditional web forms and document uploads (e.g., proof of IRB approval, where required).

One key innovation with the new system is an attempt to make the IT security portion of the process as painless as possible. Historically ICPSR and other data providers have required researchers to submit detailed IT security plans for protecting the data, a process which often required a great deal of labor, but which did not actually make any actual measurements about security. In the RCS we've replaces the IT security plan with three new components.

One, we pull questions from our "Question Bank" that are tailored to the specific IT environment of the researcher (e.g., Windows machine connected to the Internet) and to a specific person: the researcher or the researcher's IT person. For example, one question might ask the researcher to confirm that s/he will lock the office door when the data are unattended. And another question might ask the IT person to confirm that the data will be kept in a place where they will not be backed up to tape for disaster recovery purposes.

Two, we ask the researcher to install and run an audit utility which inspects the computer for common security problems. The software does NOT require administrative access for installation or to run, and we limit its checking to a small number of essential areas, such as checking to see if a screen saver with password has been enabled.

Three, we also partner with the University of Michigan to run a remote vulnerability scan of the computer(s), looking for common problems which can be exploited remotely by attackers.

If the questions are answered appropriately, and if the audit and scan do not reveal any problems, then the researcher has completed the IT security portion of the process, and no written IT security plan is required. (We do, however, give researchers the option of writing an IT security plan if they would rather not submit to the scan and audit.)

The goal of the new portal is to lower the barrier for accessing restricted-use data, but still collecting enough information to ensure that the data will be safe.

The complete RCS suite of software also includes internal utilities to automate the contract administration process, such as generating reminder emails about contract renewal.

Rework

I've been reading Rework by the 37signals guys. It's a very fast read, consisting of a series of brief essays grouped into different thematic chapters.

Most of the essays are bracingly irreverent, but I've been finding that many of them contain at least some elements that match my own perceptions having worked at small companies, large companies, and in both the public and private sectors.

Here are a few tidbits from one of my favorites, their essay Meetings are toxic(!).

The worst interruptions of all are meetings. Here's why: ... They require thorough preparation that most people don't have time for. They frequently have agendas so vague that nobody is really sure of the goal...

and

If you decide that you absolutely must get together, try to make your meeting a productive one by sticking to these simple rules: ... Always have a clear agenda. Begin with a specific problem. ...

I've really enjoyed reading this, and would recommend it to almost anyone, but particularly those that work at places that rely heavily on tech.

ICPSR: Then and Now: Technology Human Resources: Part II

As I mentioned in the last post, the team faced two main challenges in 2002: How to grow its capacity for managing IT resources without adding more people; and, how to expand its capacity for delivering solutions, and becoming a true partner at ICPSR.

We addressed the first challenge by asking our administrative assistant to step into a technology support role. This transition was largely successful, but when the administrative assistant retired at the end of 2005, we refilled the position with someone who had already been working in the IT sector. We also made one new hire in this area, adding Asmat Noori as as assistant IT director with responsibility for operations. Asmat's team supports over twice as many systems as 2002, and his introduction of tools such as Altiris and Wise has allowed us to keep the size of the team the same. That said, we're hoping to use a Challenge Grant to fund a new person who will lead ICPSR's adoption of cloud computing and cloud storage technologies.

We addressed the second challenge through a combination of writing grants and contracts, and recruiting software developers with expertise and experience in technologies such as Java. We also encouraged internal ICPSR businesses, such as the Summer Program, to fund directly portions of software developers when there is a need for sustained development and enhancement, such as the new Summer Program Portal.

This has been a very successful combination with new software developers joining the team in 2003 (to work on the Child Care and Early Education Research Center project), 2005 (to automate key data processing and data pipeline work flows at ICPSR), 2007 (one to build tools for the Minority Data Resource Center and one to build the Summer Program Portal), and 2008 (to build technology for the Quantitative Social Science Digital Library). Because the software development team had become so large and worked with so many partners across ICPSR, we also hired an assistant director for software development, Nathan Adams, in late 2008.

Cole Whiteman joined ICPSR in 2004, and brought his skills of process forensics to our data management activities. Cole later joined the Computer and Network Services team, and continues his work to analyze processes and build software systems. Cole built and support systems for managing most of our metadata and the deposits that arrive via our on-line deposit system.

And Peter Joftis returned to his technology roots, re-joining the CNS team in 2009 after leading the Child Care and Early Education Research Center project for the past six years. Peter's current focus is on CCEERC content, and how best to store it in a Fedora repository.

And so the story ends, for now, in 2009 with an IT organization that looks very different.

Like in 2002 it still manages and operates ICPSR's considerable technology infrastructure, and despite the growth in those assets, does so with just about the same number of people as in 2002.

However, in 2009, the number of software developers has grown from two to eight, and the capacity to work in a broad array of technologies has increased dramatically. Also, its ability to work with stakeholders at ICPSR to analyze processes, design solutions, co-write grant applications, and deliver new products and services has grown even more.

ICPSR: Then and Now: Technology Human Resources: Part I

When I joined ICPSR in 2002 the technology team - called Computer and Network Services - had eight people. In addition to myself, there was an administrative assistant, two software developers, three systems administrators, and one technology generalist who did a little bit of everything. Longtime ICPSR staffer Peter Joftis was also on the team, but soon left to lead ICPSR's Child Care and Early Education Research Center.

The team managed about 75 desktop workstations, a small number of servers, the local area network (LAN), a dozen or so printers, and no doubt a handful of other technology assets I've lost track of over time. With only two software developers the team spent most of its time delivering incremental changes to the web delivery system, and tending to core infrastructure, such as our database and web applications for managing information about the membership. At the time we were very much a classic IT shop.

People worked very hard, but we were on the margin of the business. And we were perceived that way. The IT people were the ones who fixed your PC when it had a virus. They patched computers when Microsoft announced yet another security flaw in Windows or Office. They took care of backups, and retrieved that file you deleted accidentally. These were valuable services to be sure, but they weren't core to the business. They don't think up solutions to our problems; they just implement the technology solutions we think up. We were not partners.

Because ICPSR is so clearly in the information business, but made almost no entrepreneurial investments in information technology, 2002 was a very dangerous time for the organization. But what to do?

When interviewing for the position it was clear that the team fell into two distinct functional subgroups. One group delivered those classic IT support functions, and the other group delivered new products and services.

It would be important for the first group to remain about the same size, but expand its capacity to support more of everything: more servers, more storage, more desktop workstations, more printers, etc. And so we would need to invest in tools and processes to build this capacity without adding significantly to the number of people on the team.

And it would be important for the second group to grow. A lot.

One, it would need to be a bigger team. The capacity to deliver new products and services, to explore new technologies, and to automate the many manual processes at ICPSR all needed to be expanded dramatically.

Two, it would need to be a more partner-oriented team. The team needed to expand its ability to work hand-in-hand with data processors, archive managers, and grant writers to understand key business problems and opportunities, and to recommend and build solutions to address those needs. Many of the software developers would need to become project managers and systems analysts.

And, three, it would need to expand its repertoire of technologies. The team had been working largely in CGI/Perl to build web applications, and Perl alone to build command-line utilities, and those were the appropriate, dominant technologies of the 90's. But by 2002 there were many other technologies available, and the team needed to select the best, and build its collective muscle around them.

Next: The team evolves