A look back to 2000 (part 4)

Continuing from the 2000 ICPSR Annual Report....

Internet Access

Web technology continues to develop extremely rapidly. As new tools and techniques become available, CNS will continue to attempt to utilize them to provide innovative and useful services via the Web. CNS has moved in the past year toward standardizing the Data Analysis System software developed at the University of California for on-line subsetting and limited analysis of a small but significant number of studies. A major redesign of ICPSR's Web site is under way, and is scheduled for implementation in the next year. Over the past two years, CNS has cooperated with the Council for European Social Science Data Archives (CESSDA) to integrate ICPSR's catalog into a truly international index covering the collections of ICPSR and the national archives of Australia, Israel, and most of Europe.

SDA is certainly the de facto standard at ICPSR for on-line analysis of datasets, but it is also the case that we have turned this web application into more of a platform for a wide range of different services.  Had the team known how this would progress over the next ten years it might have selected something different; more of a bona fide platform for building services rather than a finished web application twisted into a platform.

Integrating ICPSR's catalog into larger catalogs turns out to be a business problem, not a technology problem.  Packaging ICPSR metadata into standard formats (like DDI) and exposing it to search engines and harvesters is simple; making the business decisions about who has access, who can redistribute/deliver our holdings, where the search actually takes place, etc are the difficult problems.  And none of them are technical.

As long as ICPSR continues to operate a significant "retail business" for data delivery, I don't see the landscape changing.

With the authorization of Council, CNS has begun testing and implementation of an open Web-based ordering system. Under this approach, anyone on a participating member's campus will be able to directly download all ICPSR data and machine-readable documentation without requiring the intervention of their campus Official Representative. The testing and development of this service is scheduled to be completed in the first quarter of 2001.

This year CNS released the Web-based replacement for CDNet, the Consortium's pioneering Internet-based data order and retrieval system. CNS staff continue to participate in the development of an XML implementation of a social science codebook standard in conjunction with the international Data Documentation Initiative. CNS staff will continue to work on the integration of this and other XML standards into ICPSR's internal metadata catalogues and processing systems.

This project was completed long ago, of course.  It is hard to believe that ICPSR only started making its content accessible via the web in 2001, but I suspect that this was another case where the issue was bound up in business process rather than technology.  For example, if the business process at the time was to keep the Organizational Representative (OR) firmly in the middle, positioned between ICPSR's delivery system and the consumer, one can imagine how revolutionary it would be to tell the ORs that they would now be on the sidelines.

And now for a few more photos from the annual report.  As with past posts in this series, mouse over the photo to see the name of the person.

And another:

And one more:

A look back to 2000 (part 3)

Returning to the 2000 ICPSR Annual Report:

Operations Support

ICPSR continues to pursue a strategy of distributed and networked computing systems. ICPSR staff use increasingly powerful individual personal computers and workstations electronically networked to more powerful servers. This hierarchy of computing devices allows ICPSR to take advantage of the good price-performance ratios in desktop computers and still have the higher performance servers to provide the computing power and mass storage needed to handle the large volume of data processed and disseminated each year.

It is ironic, but even though ICPSR staff have much more powerful desktop machines today, they perform fewer work functions on those machines.  Instead, most of the data management and processing work takes place within the confines of the Secure Data Environment (SDE), and therefore most of the work takes place on a virtual desktop machine that we "rent" from the University of Michigan's central IT provider, ITS.

All staff members have cost-effective Pentium or Macintosh desktop workstations with connectivity to powerful, specialized servers. All staff members have access to a standard set of desktop applications (word processing, spreadsheets, local area network services, World Wide Web access, electronic mail) as well as to specialized software necessary to perform particular functions (statistical packages, desktop publication software, specialized editing packages, database management systems, etc.).

And today it is all managed centrally - machine images, software packages, operating system patches, etc.

ICPSR currently runs servers that provide high-capacity magnetic disk storage, magnetic tape access (l/2-inch reel-to-reel and 3480, 4mm, 8mm, I/4-inch cartridge, and digital linear tape), database management facilities, high-capacity printers, image and Optical Character Recognition (OCR) scanning, CD-ROM mastering, and wide area network gateways. All of these services are available on ICPSR's internal local area network, and are provided by a set of five SPARCstations.

Well, not so much anymore.  The only thing we manage that ever sees a tape is a special-purpose tape library that we use to back up our EMC NS 120 storage appliances.  And that's LTO-4 tape.

And the five SPARCstations have become five dozen different pizza box servers (when we need lots of local disk), blade servers, and virtual servers running in Amazon's EC2.

ICPSR's current main computer servers are a pair of SPARCserver 1000s running Solaris and connected via a dedicated 100 megabit-per-second subnet. CNS plans to upgrade this equipment next year, replacing the SPARCservers with more powerful Sun 4500 Enterprise Servers.

The E4500 systems - one for the web server and one for databases and data processing - were still reasonably new when I arrived in 2002.  But we replaced them with Dell servers running RH Linux in 2007 (I think), and then replaced those with bigger, better, badder 64-bit machines this year.

Over the more than 30-year history of ICPSR, the Computing and Network Services group has undertaken several major in-house programming projects to provide ICPSR with more effective data processing tools. FAST and CDNet are specialized archival processing and management systems that CNS developed. CNS continues to maintain and upgrade ICPSR's core orderinventory and record-keeping systems.

I think the gang had killed off CDNet even before I arrived, and we eliminated FAST a year or two into my tenure.  Everything is much more automated now, but the overall workflow and business process hasn't changed, and that is a problem.  The act of storing an object in the repository and the act of publishing an object on the web site are still very much intertwined, and changing the business process - and the software systems - is one of the key goals for FLAME.

ICPSR came through the Y2K "crisis" relatively unscathed. All of our systems were tested and upgraded where necessary during 1999, and were taken off-line as a precaution over the New Year's holiday. Coincidentally, a non-Y2K-related hardware failure occurred when the system was powered back up on New Year's Day, and the system stayed down until January 3rd.

And now we have a replica of ICPSR's web-based delivery system in Amazon's cloud.

In the coming year, CNS plans to add another terabyte of high-speed RAID disk storage arrays to accommodate the needs of the expanding archive and to provide space for migration from our library of 3480 tape cartridges. We have also continued an aggressive program of upgrading staff workstations.

I don't know how much disk storage ICPSR had in 2000, but we have 50+ TBs on-site in our EMC NAS units, another 10TB with ITS, another 6TB at Michigan State University, and at least this much again at DuraCloud and again and again in Amazon's EBS and S3 systems.

If the number of computing systems has grown 10x, I would estimate that the amount of managed storage has grown by 100x.

And that doesn't count the additional 50-100TB we need for the Measurements of Effective Teaching video collection that we have.

But not so many tapes....

And this week's photo contest entry.  (Mouse over the image to see the name.)

I think this one is too easy.

A look back to 2000 (part 2)

From the 1999-2000 ICPSR Annual Report section on Computer and Network Services:

The mission of the Computing and Network Services Group is to provide the technical  resources necessary to support ICPSR's mission: the preservation, acquisition, and distribution of social science data and the education of the social science community in its use. Computing and Network Services (CNS) has four major areas of responsibility:

• CNS consults with users at ICPSR member institutions regarding software and hardware selection and works to resolve any technical problems encountered in the data delivery process. 

• CNS supports the hardware and software used in the routine operations of ICPSR's member services, archival development, educational, and administrative groups. 

• CNS participates in the assessment and evaluation of ICPSR's operations, providing technical input into the ongoing improvement of processes and procedures. CNS also participates in the development and acquisition of innovative systems. 

• In the long term, CNS is responsible for developing and promulgating a vision of the technical future, for anticipating the impact of new technologies on services, and for ensuring that ICPSR is well-positioned to take advantage of these new technologies.

The high-level mission of the Computer and Network Services team hasn't changed:  It continues to support other parts of the organization.  However, it also plays a much larger leadership role than perhaps it did 10+ years ago, selecting technologies and services that enable capabilities that were heretofore unavailable.  For example, no one asked the CNS team to "use the cloud"; evaluating, selecting, and adopting cloud technologies and cloud infrastructure providers was one of the leadership activities we undertook.

If we dig into the bullet-points, things have changed a bit more, however.

Bullet #1 : I insisted that we change this responsibility soon after arriving at ICPSR in 2002.  Based on past experience I felt it was important that the customer-facing portion of the organization own the relationship with the customer, and while the IT team would provide second-level support to our front-line folks, there was no way we could routinely engage with customers and be effective.  The team just wasn't staffed to do that sort of thing.

Bullet #2 is still true today, but the scope also includes the vendors and virtual infrastructure that we use to deliver services both to internal clients and external customers.

Bullet #3 has been one of the main areas of focus for the team, and continues to be a high priority even today.  I might describe the work we led between 2005-2008 as analyzing and documenting the existing business processes at ICPSR, and then automating the heck out of them wherever possible.  The next phase - starting with FLAME - will be to re-invent business processes at ICPSR so that they are more in harmony with standards like TRAC and OAIS, and then adding automation where it makes sense.

Bullet #4 is definitely still true today, but I think there is room for improvement.  It is too easy to get caught up in the demands of the "next big project" and to stop looking further over the horizon at what's coming next.

And now for the photo contest portion of the post.  Can you identify this ICPSR staff member from 2000?  Mouse over the photo to check your answer.

A look back to 2000 (part 1)

I was looking through the Internet Archive's Wayback Machine the other day, trying to find an electronic copy of a printed document I have.  I wasn't able to find the document I wanted, but I did stumble upon a copy of the 1999-2000 ICPSR annual report.  This appears to have been published during the interregnum between Richard Rockwell and Myron Gutmann.  Myron took over as the Executive Director of ICPSR in 2001, and hired me in 2002 to lead the Computer and Network Services team.

As I read the report it occurred to me that some parts of it were just as relevant today as 10+ years ago.  In some cases the issues and observations are exactly the same; only the players and the technologies have changed.

I thought it might be interesting to compare excerpts from the Computer and Network Services section of that report, and compare them to today's ICPSR.  There's quite a bit of text in that old annual report, and I think it will make sense to break it down into smaller pieces for easier reading and commentary.

I also found a wonderful array of old photographs of my colleagues at ICPSR.  Some of them have left ICPSR, and some of them still work here today.  I'll include one of them in each of the follow-up posts too.  If you mouse over the picture, it should identify the person.

ICPSR: Then and Now: Technology Human Resources: Part II

As I mentioned in the last post, the team faced two main challenges in 2002: How to grow its capacity for managing IT resources without adding more people; and, how to expand its capacity for delivering solutions, and becoming a true partner at ICPSR.

We addressed the first challenge by asking our administrative assistant to step into a technology support role. This transition was largely successful, but when the administrative assistant retired at the end of 2005, we refilled the position with someone who had already been working in the IT sector. We also made one new hire in this area, adding Asmat Noori as as assistant IT director with responsibility for operations. Asmat's team supports over twice as many systems as 2002, and his introduction of tools such as Altiris and Wise has allowed us to keep the size of the team the same. That said, we're hoping to use a Challenge Grant to fund a new person who will lead ICPSR's adoption of cloud computing and cloud storage technologies.

We addressed the second challenge through a combination of writing grants and contracts, and recruiting software developers with expertise and experience in technologies such as Java. We also encouraged internal ICPSR businesses, such as the Summer Program, to fund directly portions of software developers when there is a need for sustained development and enhancement, such as the new Summer Program Portal.

This has been a very successful combination with new software developers joining the team in 2003 (to work on the Child Care and Early Education Research Center project), 2005 (to automate key data processing and data pipeline work flows at ICPSR), 2007 (one to build tools for the Minority Data Resource Center and one to build the Summer Program Portal), and 2008 (to build technology for the Quantitative Social Science Digital Library). Because the software development team had become so large and worked with so many partners across ICPSR, we also hired an assistant director for software development, Nathan Adams, in late 2008.

Cole Whiteman joined ICPSR in 2004, and brought his skills of process forensics to our data management activities. Cole later joined the Computer and Network Services team, and continues his work to analyze processes and build software systems. Cole built and support systems for managing most of our metadata and the deposits that arrive via our on-line deposit system.

And Peter Joftis returned to his technology roots, re-joining the CNS team in 2009 after leading the Child Care and Early Education Research Center project for the past six years. Peter's current focus is on CCEERC content, and how best to store it in a Fedora repository.

And so the story ends, for now, in 2009 with an IT organization that looks very different.

Like in 2002 it still manages and operates ICPSR's considerable technology infrastructure, and despite the growth in those assets, does so with just about the same number of people as in 2002.

However, in 2009, the number of software developers has grown from two to eight, and the capacity to work in a broad array of technologies has increased dramatically. Also, its ability to work with stakeholders at ICPSR to analyze processes, design solutions, co-write grant applications, and deliver new products and services has grown even more.

ICPSR: Then and Now: Technology Human Resources: Part I

When I joined ICPSR in 2002 the technology team - called Computer and Network Services - had eight people. In addition to myself, there was an administrative assistant, two software developers, three systems administrators, and one technology generalist who did a little bit of everything. Longtime ICPSR staffer Peter Joftis was also on the team, but soon left to lead ICPSR's Child Care and Early Education Research Center.

The team managed about 75 desktop workstations, a small number of servers, the local area network (LAN), a dozen or so printers, and no doubt a handful of other technology assets I've lost track of over time. With only two software developers the team spent most of its time delivering incremental changes to the web delivery system, and tending to core infrastructure, such as our database and web applications for managing information about the membership. At the time we were very much a classic IT shop.

People worked very hard, but we were on the margin of the business. And we were perceived that way. The IT people were the ones who fixed your PC when it had a virus. They patched computers when Microsoft announced yet another security flaw in Windows or Office. They took care of backups, and retrieved that file you deleted accidentally. These were valuable services to be sure, but they weren't core to the business. They don't think up solutions to our problems; they just implement the technology solutions we think up. We were not partners.

Because ICPSR is so clearly in the information business, but made almost no entrepreneurial investments in information technology, 2002 was a very dangerous time for the organization. But what to do?

When interviewing for the position it was clear that the team fell into two distinct functional subgroups. One group delivered those classic IT support functions, and the other group delivered new products and services.

It would be important for the first group to remain about the same size, but expand its capacity to support more of everything: more servers, more storage, more desktop workstations, more printers, etc. And so we would need to invest in tools and processes to build this capacity without adding significantly to the number of people on the team.

And it would be important for the second group to grow. A lot.

One, it would need to be a bigger team. The capacity to deliver new products and services, to explore new technologies, and to automate the many manual processes at ICPSR all needed to be expanded dramatically.

Two, it would need to be a more partner-oriented team. The team needed to expand its ability to work hand-in-hand with data processors, archive managers, and grant writers to understand key business problems and opportunities, and to recommend and build solutions to address those needs. Many of the software developers would need to become project managers and systems analysts.

And, three, it would need to expand its repertoire of technologies. The team had been working largely in CGI/Perl to build web applications, and Perl alone to build command-line utilities, and those were the appropriate, dominant technologies of the 90's. But by 2002 there were many other technologies available, and the team needed to select the best, and build its collective muscle around them.

Next: The team evolves

ICPSR: Then and Now: Archival Storage

Archival Storage, the OAIS function responsible for storing and retrieving content, was built on DLT IV tapes at ICPSR in 2002. Files that we wanted to keep indefinitely were moved to a pair of DLT tapes; one copy was retained at ICPSR, and the other was stored at an off-site location in Ann Arbor, Michigan.

And, unfortunately, we also had a large number of older tape formats as well: IBM 3480 cartridge and 9-track. Again there were two copies, but in this case, both were off-site.

As you might expect with an off-line system such as this, it was very expensive to retrieve any item from Archival Storage. Also, if the requestor was a little fuzzy about the exact item of interest, that would also add to the cost. There was no good way to browse the holdings, and retrieval time was measured in days not minutes.

Today we've moved the master copy of each file from tape to disk, and we replicate each file off-site using a variety of techniques, such as rsync and the Storage Resource Broker Srsync utility. We also keep a copy on tape too, but instead of DLT IV, we're using LTO-3 tape which is ten times more dense. And so this gives us more copies in more locations, and a high degree of confidence that the copies are synchronized.

The next step in Archival Storage is a move away from file-based solutions to object-based solutions. We've been evaluating Fedora as a possible storage platform for social science datasets and documentation, and the results are very promising so far.

ICPSR: Then and Now: Desktop Workstations

In many ways the world of desktop computing has changed very little over the past seven years at ICPSR. Each ICPSR staff member receives a t workstation that includes a tower, display, keyboard, and mouse. In general we did not assign staff peripherals such as cameras, speakers, or external storage in 2002, and we still do not today.

The front face of the display is about the same size now as it was then, 17 inches. Some staff today may have a 20" widescreen display, but the overall square footage for display hasn't changed much. However, the display on a desktop workstation today is much lighter than it was in 2002 (LCD v. CRT), and greater screen resolution makes the display feel larger still. We also have more staff who are interested in funding an upgraded display such as dual monitors or a single very large display. So perhaps if the metric is area, things have changed very little, but if the metric is weight or number of pixels, things have changed quite a bit.

The appearance of the workstation itself has changed little. In 2002 each person had a tower-style workstation under the desk, and most people have a machine with the same general shape. In 2002 most of the machines had a Dell nameplate, but today they are much more likely to have an H-P template. The price difference wasn't a driver for us to move to H-P; the quality control of the Dell tower systems had really slipped in our experience.

The 2002 Dell was likely a GX 240 with 256Mb of memory, and was running Windows 2000 or Windows XP. The 2009 H-P is 4Gb of memory, and some sort of dual-core processor. It still runs Windows XP.



The 2009 workstation likely has a huge local disk - it's difficult to buy a machine with a small disk - that is largely unused (at least for business reasons). The disk will have a system partition which most staff can't modify, and which holds applications and the operating system. The rest of the disk forms a second very large partition, but since we don't back up desktop workstations, it shouldn't be used for most work processes other than as scratch space. Valuable work products are stored on our NAS, the central file server.

The next step for desktop workstations at ICPSR may be to begin using ultra-thin clients: machines that have little storage and processing power because all of the heavy lifting takes place in the cloud. In some ways this is a return to the X terminals that were so popular twenty years ago.

ICPSR: Then and Now: Servers

In 2002 ICPSR had two main systems - a pair of Sun E3500s with 4GB of memory. One machine served as our production web server, and the second did everything else: general-purpose computing for data processing, Oracle database service, file service (NFS and CIFS via samba), DNS service, etc. We also had a very small number of additional machines, such as a system for testing new web applications. All of the machines were built by Sun Microsystems, used Sun's SPARC processors, and ran Sun's operating system, Solaris. We entered into a maintenance contract with Sun in case either of the machines had a problem, and my recollection is that it ran around $15k/year to cover the two big machines plus a handful of external storage arrays. To Sun's credit they were very solid machines.

In 2009 ICPSR has more servers than I can describe easily in a blog post. We still have a pair of machines for delivering web content and general-purpose computing, but they were built by Dell, use Intel processors, and run Red Hat Linux. Today's machines have much more memory and many more processors, and they too have been solid. But we also have many smaller machines with very specific roles: delivering network services (DNS, DHCP, etc); operating our LOCKSS network; staging new web content; replicating services for the Minnesota Population Center; hosting MySQL and Oracle databases; and so on. And, of course, in 2009 Sun Microsystems is about to be swallowed by Oracle.



However, this proliferation of server computing systems has likely reached its apogee at ICPSR. With the rise of virtualization and particularly the rise of the cloud, we're much more likely to build future systems in Amazon's Elastic Computing Cloud (EC2) rather than building them on real (or virtual) machines at ICPSR. For every rack-mount server we have at ICPSR, we probably have one much smaller blade server, and for every blade server, we probably have one EC2 instance running in the cloud.



My sense is that we'll continue this trend, and that where practicable, we'll deploy new systems in a cloud environment rather than purchasing new hardware. In addition to Amazon's cloud offering, the University of Michigan is deploying its own virtualization service, and that will be an attractive choice for systems that consume a lot of network I/O. Amazon charges for network I/O, but U-M does not.

We may also replace several virtual machines in the cloud with an out-sourced service: we already use SalesForce.com as our platform for managing "data leads." It's easy to imagine us adopting OpenID via a service provider such as RPX rather than hosting our own service locally or in a cloud, for example,

ICPSR: Then and Now

As many of you have already heard, Myron Gutmann, the director of ICPSR, has accepted a position with the National Science Foundation, and will be leaving ICPSR at the end of October. If you have not heard the news, here is a link to the press release.

With the departure of Myron and the start of a new academic year, the time feels right for a retrospective series of posts. I'm intending to post two or three per week over the next few weeks, and the general theme will be "ICPSR: Then and Now."

Myron hired me in September 2002, and so late 2002 will form the bulk of my "then" for this series. We'll take a brief stroll down memory lane, looking at how ICPSR - and especially technology at ICPSR - has changed over the past seven years.