The spring of our discontent

This has been a rough spring for the availability of our delivery system.  We are not having fun.  We are not amused.

The drill goes something like this:  The load average on the production web server soars from something around 1 to 50 or 100 or higher, and one of the web applications stops doing any useful work.  It is shown as running in the process table, but it does not answer queries and does no logging.

Killing the web application processes - a pair of tomcat daemons - works OK, but the processes remain in a defunct state, holding onto resources, and preventing a new web application from starting.  The parent PID is 1, but init isn't reaping the dead process.

Worse, if one tries to reboot the machine, say via /sbin/reboot, the machine hangs.  One has to cycle power on the machine to get it to restart cleanly.  Thank goodness for fast boot times and journaled filesystems!

Often these drills take place early in the morning (4am, 5am) when we run big automated jobs, like updating the search index.  And we always see a little something like this in /var/log/messages:

Mar 29 04:43:35 top kernel: INFO: task khugepaged:503 blocked for more than 120 seconds.
Mar 29 04:43:35 top kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 29 04:43:35 top kernel: khugepaged    D 0000000000000000     0   503      2 0x00000000
Mar 29 04:43:35 top kernel: ffff8804747d9c90 0000000000000046 0000000000000000 ffff880000058d80
Mar 29 04:43:35 top kernel: ffff8804747d9c20 0000000000000002 0000000000000003 ffff880000069b00
Mar 29 04:43:35 top kernel: ffff8804747b26b8 ffff8804747d9fd8 000000000000f4e8 ffff8804747b26b8
Mar 29 04:43:35 top kernel: Call Trace:
Mar 29 04:43:35 top kernel: [<ffffffff8116a1c8>] ? __mem_cgroup_try_charge+0x78/0x420
Mar 29 04:43:35 top kernel: [<ffffffff814eef25>] rwsem_down_failed_common+0x95/0x1d0
Mar 29 04:43:35 top kernel: [<ffffffff814ef083>] rwsem_down_write_failed+0x23/0x30
Mar 29 04:43:35 top kernel: [<ffffffff81276d83>] call_rwsem_down_write_failed+0x13/0x20
Mar 29 04:43:35 top kernel: [<ffffffff814ee582>] ? down_write+0x32/0x40
Mar 29 04:43:35 top kernel: [<ffffffff8116f140>] khugepaged+0x790/0x12c0
Mar 29 04:43:35 top kernel: [<ffffffff81090a90>] ? autoremove_wake_function+0x0/0x40
Mar 29 04:43:35 top kernel: [<ffffffff8116e9b0>] ? khugepaged+0x0/0x12c0
Mar 29 04:43:35 top kernel: [<ffffffff81090726>] kthread+0x96/0xa0
Mar 29 04:43:35 top kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20
Mar 29 04:43:35 top kernel: [<ffffffff81090690>] ? kthread+0x0/0xa0
Mar 29 04:43:35 top kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20

This can't be good.  It is never good when the kernel logs messages.  A happy kernel is a silent kernel.

Our experience has been that heavy use of NFS - especially by writers - tickles this bug more often, and so we've taken to off-loading some content from our EMC NAS to the local filesystem.  This also helps performance sometimes, but makes it more problematic to back up content given how we've architected things.

We've also done some digging into the deepest Red Hat forums and IRC channels looking for answers.  Maybe khugepaged is just the victim here?  Maybe the real culprint is the memory management in the kernel?  Or Colonel Mustard in the library?

This entry got us to try one more thing.

So we'll see how it goes next week.

If only we had Richard III to help.  Or Steinbeck.

The cloud is not a hard drive

I recently read a piece in one of the IT trade publications about why buying infrastructure via a cloud provider (Amazon in particular) was a bad deal, and how one would be much better off buying storage in-house.  As is often the case, the opinion-piece compares apples to oranges, and the analysis is so flawed, one can't really make any use of the conclusion.  However, my experience is that the piece is hardly unique in its flaws, and that motivated today's post.

The usual analysis is to compare the cost of storage when purchased as a consumer-grade disk drive v. storage when purchased via a cloud-based infrastructure provider.  The numbers -- $100 for a 2TB drive from Best Buy v. $100/mo for 1TB of S3 space at Amazon -- are thrown out as directly comparable, and thus it is plain to see that it is better to spend that $100 just once to get storage rather than spending the same amount of money each month (and for less storage to boot).

The analysis then usually performs some hand-waving about how disk drives sometimes fail (really?) and how they don't last forever (shocking!), but these facts are just in the noise, and it really is clear how much better one is purchasing storage rather than "renting" it in the cloud.

However, this sort of comparison is like looking at the cost of one's grocery bill for a given meal v. the cost of having a meal at a restaurant.  The first is merely one part of the second, and they simply are not the same thing.

A disk drive from Best Buy can be a very useful thing.  I myself own one of these very same drives, and I use it every couple of weeks to back-up a PC we have at home.  (Clearly my disaster recovery process is not good.)  Because the storage only needs to be "lit" every so often, and only needs to be accessed from a single location, it works well.

However....

If instead I needed my storage to be available 24 x 7, and to have some credible DR plan, this wouldn't do.  Or if I wanted to be able to use a lot more storage some days (or weeks or months) and less storage on others, then buying storage for peak isn't so good.  Also, when it is "lit" my storage consumes some electricity to operate.  So maybe an analysis that includes things like mirrored copies, electricity, front-end access systems, the network, and .....  gets to a more apples-to-apples comparison.

The cloud may be the best solution to some problems, and a dreadful solution to other problems.  Maybe way overpriced.  But a lot of different stuff makes up "cloud storage" and the underlying media is only one of many; in fact, it may be the least expensive component.

A look back to 2000 (part 4)

Continuing from the 2000 ICPSR Annual Report....

Internet Access

Web technology continues to develop extremely rapidly. As new tools and techniques become available, CNS will continue to attempt to utilize them to provide innovative and useful services via the Web. CNS has moved in the past year toward standardizing the Data Analysis System software developed at the University of California for on-line subsetting and limited analysis of a small but significant number of studies. A major redesign of ICPSR's Web site is under way, and is scheduled for implementation in the next year. Over the past two years, CNS has cooperated with the Council for European Social Science Data Archives (CESSDA) to integrate ICPSR's catalog into a truly international index covering the collections of ICPSR and the national archives of Australia, Israel, and most of Europe.

SDA is certainly the de facto standard at ICPSR for on-line analysis of datasets, but it is also the case that we have turned this web application into more of a platform for a wide range of different services.  Had the team known how this would progress over the next ten years it might have selected something different; more of a bona fide platform for building services rather than a finished web application twisted into a platform.

Integrating ICPSR's catalog into larger catalogs turns out to be a business problem, not a technology problem.  Packaging ICPSR metadata into standard formats (like DDI) and exposing it to search engines and harvesters is simple; making the business decisions about who has access, who can redistribute/deliver our holdings, where the search actually takes place, etc are the difficult problems.  And none of them are technical.

As long as ICPSR continues to operate a significant "retail business" for data delivery, I don't see the landscape changing.

With the authorization of Council, CNS has begun testing and implementation of an open Web-based ordering system. Under this approach, anyone on a participating member's campus will be able to directly download all ICPSR data and machine-readable documentation without requiring the intervention of their campus Official Representative. The testing and development of this service is scheduled to be completed in the first quarter of 2001.

This year CNS released the Web-based replacement for CDNet, the Consortium's pioneering Internet-based data order and retrieval system. CNS staff continue to participate in the development of an XML implementation of a social science codebook standard in conjunction with the international Data Documentation Initiative. CNS staff will continue to work on the integration of this and other XML standards into ICPSR's internal metadata catalogues and processing systems.

This project was completed long ago, of course.  It is hard to believe that ICPSR only started making its content accessible via the web in 2001, but I suspect that this was another case where the issue was bound up in business process rather than technology.  For example, if the business process at the time was to keep the Organizational Representative (OR) firmly in the middle, positioned between ICPSR's delivery system and the consumer, one can imagine how revolutionary it would be to tell the ORs that they would now be on the sidelines.

And now for a few more photos from the annual report.  As with past posts in this series, mouse over the photo to see the name of the person.

And another:

And one more:

TRAC: A5.5: Coping with surprises

A5.5 If repository ingests digital content with unclear ownership/rights, policies are in place to address liability and challenges to those rights.

The repository’s policies and mechanisms must be vetted by appropriate institutional authorities and/or legal experts to ensure that responses to challenges adhere to relevant laws and requirements, and that the potential liability for the repository is minimized.

Evidence: A definition of rights; citations for relevant laws and requirements; policy on responding to challenges; documented track record for responding to challenges in ways that do not inhibit preservation; examples of legal advice sought and received.

---

This is the last post in my series where I try to document ICPSR's standing wrt TRAC.  I posted the first item a long time ago on September 30, 2009.  At one post/week (and a couple of multi-week gaps) it takes a long time to work through the document.

I thought I might take a second pass through TRAC, but in this case I would use the (somewhat) newly release "magenta book" as my guide.  Ne sure to put on your sunglasses first if you decide to click the link in this paragraph.

But on to the last of the "old style" TRAC requirements....

ICPSR's deposit agreement requires the depositor to assert that s/he owns the rights to the intellectual property being deposited, and that while s/he retains those rights, s/he also grants ICPSR the non-exclusive right to preserve and disseminate (eventually) the content.  In general we don't encounter very many instances where we find that a depositor didn't have rights, and so we need to take action.

My sense is that if we found that a depositor didn't have rights - or decided that they wanted to break their agreement with ICPSR - we would pull the content from the dissemination and archival systems, and would return the content (if desired).

iFLAME?

ICPSR's project to stand up a repository system continues to move along, but slowly. 

Some of this is due to changes in staffing; for example, our colleague Nancy McGovern has joined the team at MIT, and is now Head, Curation and Preservation Services at MIT Libraries.

Some of this is due to people getting busy with their day jobs at ICPSR and finding less opportunity to discuss requirements for the project, or to work through existing lists to ensure that they make sense.

And a big part is due to changes on my team.  We lost four people to a retirement incentive opportunity, and a pair of new Gates Foundations project created a demand for the equivalent of over three full-time positions, and so this is a net swing of seven people.  It's hard to sustain effort on projects with that level of change.

One of my colleagues dropped by earlier this week to relate a metaphor for FLAME (the File Level Archival Management Engine).  She likened the current ICPSR content environment to be like the world of LPs.  The unit of content is the "album" and that is where most of the metadata lives.  At ICPSR our "album" is the "study."

FLAME, however, aims to take the unit of content to the file level, and to collect and store more data at that level.  This is particularly essential in a world where less and less content is a "study" and where good preservation practices call for collecting and retaining information at the file level.

To continue Laurie's metaphor, she described the move to FLAME as similar to the move Apple made in music with iTunes.  That is, the unit of content is now the song rather than the album.  I really like this metaphor, and I think it captures the issues well.

For example, if one is capturing metadata at the song level, one will still want to group songs into different lists: albums, playlists, etc.  And one may want to edit group-level metadata for a list of songs that is arbitrarily large, and to do so without editing lots and lots of songs individually.  And so on.

Zynga says "buh bye" to Amazon

So one big bit of news is how Zynga, the company responsible for such Facebook games as Farmville and Words With Friends, is building out its own IT infrastructure (zCloud) to host its games rather than continuing to rely solely upon Amazon for this infrastructure.

The obvious question:

Why?


Building your own cloud is a big bet:  data centers, racks and racks of equipment, servers, network switches, cables.... lots of cables.  That's a very big investment to make for a platform that's just overhead to your main business: developing software (games).

However.....

If the platform is actually your business, then it makes a lot of sense.  In that world the product isn't software; the product is the platform.  In that world you want to own the platform.  You want the control (costs, performance, etc).

Zynga says that it will still use AWS for spikes in service that it cannot service within its own zCloud platform, and so it isn't the complete end of the relationship for the two companies.  But it is still a very big change for Zynga.

February 2012 deposits at ICPSR

Wow.

# of files	# of deposits	File format
15	1	F 0x07 video/h264
3	3	application/msaccess
23	11	application/msword
55	11	application/msword application/msword
15	6	application/octet-stream
58	18	application/pdf
2	1	application/vnd.wordperfect
1	1	application/x-dosexec
2	2	application/x-empty
1	1	application/x-rar
5	4	application/x-sas
2	2	application/x-shellscript
84	12	application/x-spss
3	2	application/x-stata
5	5	application/x-zip
1	1	image/jpeg
1	1	image/tiff
2	2	message/rfc8220117bit
396597	10	text/html
19	5	text/plain; charset=iso-8859-1
3	3	text/plain; charset=unknown
2978	21	text/plain; charset=us-ascii
1	1	text/plain; charset=utf-8
20	2	text/x-c++; charset=us-ascii
13	6	text/x-c; charset=us-ascii
3	3	text/x-mail; charset=us-ascii
9	4	text/xml

So we have some of the usual suspects in the usual quantities.  And the usual miscues (e.g. C++ source code) due to entries in the magic database that we should really address.

And we have some new funkiness with magic or file with the strange video MIME type and the "double" MS-Word MIME type.

A few types we see but not often:  WordPerfect, Access, and some shell scripts (purportedly).  Nice.

But nearly 400k HTML files?  Wow.  Double wow.

I'm not sure what that is, but it seems like something big.

A look back to 2000 (part 3)

Returning to the 2000 ICPSR Annual Report:

Operations Support

ICPSR continues to pursue a strategy of distributed and networked computing systems. ICPSR staff use increasingly powerful individual personal computers and workstations electronically networked to more powerful servers. This hierarchy of computing devices allows ICPSR to take advantage of the good price-performance ratios in desktop computers and still have the higher performance servers to provide the computing power and mass storage needed to handle the large volume of data processed and disseminated each year.

It is ironic, but even though ICPSR staff have much more powerful desktop machines today, they perform fewer work functions on those machines.  Instead, most of the data management and processing work takes place within the confines of the Secure Data Environment (SDE), and therefore most of the work takes place on a virtual desktop machine that we "rent" from the University of Michigan's central IT provider, ITS.

All staff members have cost-effective Pentium or Macintosh desktop workstations with connectivity to powerful, specialized servers. All staff members have access to a standard set of desktop applications (word processing, spreadsheets, local area network services, World Wide Web access, electronic mail) as well as to specialized software necessary to perform particular functions (statistical packages, desktop publication software, specialized editing packages, database management systems, etc.).

And today it is all managed centrally - machine images, software packages, operating system patches, etc.

ICPSR currently runs servers that provide high-capacity magnetic disk storage, magnetic tape access (l/2-inch reel-to-reel and 3480, 4mm, 8mm, I/4-inch cartridge, and digital linear tape), database management facilities, high-capacity printers, image and Optical Character Recognition (OCR) scanning, CD-ROM mastering, and wide area network gateways. All of these services are available on ICPSR's internal local area network, and are provided by a set of five SPARCstations.

Well, not so much anymore.  The only thing we manage that ever sees a tape is a special-purpose tape library that we use to back up our EMC NS 120 storage appliances.  And that's LTO-4 tape.

And the five SPARCstations have become five dozen different pizza box servers (when we need lots of local disk), blade servers, and virtual servers running in Amazon's EC2.

ICPSR's current main computer servers are a pair of SPARCserver 1000s running Solaris and connected via a dedicated 100 megabit-per-second subnet. CNS plans to upgrade this equipment next year, replacing the SPARCservers with more powerful Sun 4500 Enterprise Servers.

The E4500 systems - one for the web server and one for databases and data processing - were still reasonably new when I arrived in 2002.  But we replaced them with Dell servers running RH Linux in 2007 (I think), and then replaced those with bigger, better, badder 64-bit machines this year.

Over the more than 30-year history of ICPSR, the Computing and Network Services group has undertaken several major in-house programming projects to provide ICPSR with more effective data processing tools. FAST and CDNet are specialized archival processing and management systems that CNS developed. CNS continues to maintain and upgrade ICPSR's core orderinventory and record-keeping systems.

I think the gang had killed off CDNet even before I arrived, and we eliminated FAST a year or two into my tenure.  Everything is much more automated now, but the overall workflow and business process hasn't changed, and that is a problem.  The act of storing an object in the repository and the act of publishing an object on the web site are still very much intertwined, and changing the business process - and the software systems - is one of the key goals for FLAME.

ICPSR came through the Y2K "crisis" relatively unscathed. All of our systems were tested and upgraded where necessary during 1999, and were taken off-line as a precaution over the New Year's holiday. Coincidentally, a non-Y2K-related hardware failure occurred when the system was powered back up on New Year's Day, and the system stayed down until January 3rd.

And now we have a replica of ICPSR's web-based delivery system in Amazon's cloud.

In the coming year, CNS plans to add another terabyte of high-speed RAID disk storage arrays to accommodate the needs of the expanding archive and to provide space for migration from our library of 3480 tape cartridges. We have also continued an aggressive program of upgrading staff workstations.

I don't know how much disk storage ICPSR had in 2000, but we have 50+ TBs on-site in our EMC NAS units, another 10TB with ITS, another 6TB at Michigan State University, and at least this much again at DuraCloud and again and again in Amazon's EBS and S3 systems.

If the number of computing systems has grown 10x, I would estimate that the amount of managed storage has grown by 100x.

And that doesn't count the additional 50-100TB we need for the Measurements of Effective Teaching video collection that we have.

But not so many tapes....

And this week's photo contest entry.  (Mouse over the image to see the name.)

I think this one is too easy.

TRAC: A5.4: Tracking agreements

A5.4 Repository tracks and manages intellectual property rights and restrictions on use of repository content as required by deposit agreement, contract, or license.

The repository should have a mechanism for tracking licenses and contracts to which it is obligated. Whatever the format of the tracking system, it must be sufficient for the institution to track, act on, and verify rights and restrictions related to the use of the digital objects within the repository.

Evidence: A policy statement that defines and specifies the repository’s requirements and process for managing intellectual property rights; depositor agreements; samples of agreements and other documents that specify and address intellectual property rights; demonstrable way to monitor intellectual property; results from monitoring.

---

ICPSR has a plethora of teams and systems that track content.

Some systems are technical, such as databases of deposits, holdings, and agreements.

Some systems are human, such as archive directors and managers who track content they are managing, preserving, and delivering.

And some systems are a mix, such as an Acquisitions team that monitors content flowing through the ingest process.

Job posting : Associate Director, ICPSR

I don't know how long this link will be live, but I've cut-and-pasted the original job posting below.    You can some work with me at ICPSR.  No tech experience required. :-)

Job Summary
		Position Summary The Inter-university Consortium for Political and Social Research (ICPSR) seeks to appoint an Associate Director to participate in the Consortium's leadership and to manage its collection development activities. The holder of this position will provide senior leadership in partnership with the Consortium's Director and will manage a portfolio of internally and externally-funded projects that acquire data and prepare them for future research. She or he will also work closely with the members of ICPSR's governing Council. This position involves the direct supervision of seven senior archivists and research scientists, with a total staff of over 50 and an annual budget of $7 million. The holder of this position will also develop new archival collections, interact with current and prospective sponsors, and write grant proposals and applications. Extensive oral and written communication is required, along with significant travel. The individual who holds this position will carry on a research program of his or her own, in collaboration with researchers in ICPSR or other units of the Institute for Social Research. The individual selected for this position will hold a research faculty appointment in ICPSR and the Institute for Social Research at the level of Research Associate Professor or Research Professor. The selected candidate's qualifications will determine the level. The selected individual will report to the Director of ICPSR. The position may be a joint appointment with other units within the Institute for Social Research (Center for Political Studies, Population Studies Center, Research Center for Group Dynamics, or Survey Research Center). A joint appointment with another unit of the University of Michigan is also possible. ICPSR offers a competitive total compensation package, with full access to University of Michigan benefits. Organizational Overview The Inter-university Consortium for Political and Social Research, established in 1962, maintains and provides access to a vast archive of social science data for research and instruction, and offers training in quantitative methods. To ensure that data resources are available to future generations of scholars, ICPSR acquires, processes, and preserves data. In addition, ICPSR provides user support. A unit within the Institute for Social Research  (ISR) at the University of Michigan, ICPSR is a membership-based organization, with more than 700 member colleges and universities around the world. A Council of leading scholars and data professionals guides and oversees its activities. More information can be found about ICPSR at www.icpsr.umich.edu. ICPSR's Collection Development Unit is one of five major activities, the others being Collection Delivery, Educational Activities, Computer and Network Services, and Administration. Collection Development has responsibility for identifying data resources, arranging for their acquisition, and processing them to ensure their future usability. Its budget is provided by ICPSR member dues and by grants and contracts from public and private sponsors. Collection Development is organized into archives, including the member-funded General Archive and eight specialized Topical Archives.  Each archive is headed by a senior archive director, who reports to the Associate Director. The Associate Director has general oversight of the management and budget of these archival units, as well as responsibility for establishing overall policies.
	Responsibilities*
		Lead ICPSR's Collection Development group by providing overall management to the unit and direct supervision to its archival project directors. This includes operational and budgetary oversight, and the establishment of operational procedures for the entire group. Consult with data producers and data users about the needs of the research community, and use that information to develop new archival collections and to write proposals for external funding to support those collections. Participate fully in ICPSR's senior management team, and when necessary substitute for the --Director in relations with the ICPSR Council, with other units at the University of Michigan, and the larger social science community. Conduct and publish research in the candidate's area of expertise.
	Required Qualifications*
		Minimum: Ph.D in relevant field of study. Seven years professional experience in the field, with a minimum of two years of experience in the management of externally-sponsored research activities. Demonstrated scholarly productivity, along with the proven ability to work both independently and collaboratively. Demonstrated expertise in the creation, management, and statistical analysis of social science research data. Knowledge of procedures for ensuring the privacy and protection of research subjects. Experience in writing grant proposals and applications, and in serving as a principal investigator. Experience in a supervisory role providing direction to staff. Excellent communication skills, both oral and written, in the English language.
	Desired Qualifications*
		Desired: Previous experience in teaching or training in the field of interest. Knowledge and experience archiving data and providing secondary analysis of those data. Applicants should submit a letter of application, a CV, three letters of reference, and relevant writing samples to: George Alter, Director, Inter-university Consortium for Political and Social Research, Institute for Social Research, P.O. Box 1248, Ann Arbor, Michigan 48106-1248. Screening of applications will begin shortly and continue until the position is filled.
	U-M EEO/AA Statement
		The University of Michigan is an equal opportunity/affirmative action employer.

ICPSR and Datanet : Sustainable Environment-Actionable Data

ICPSR is a participant in an NSF Datanet project called SEAD:  Sustainable Environment-Actionable Data.  The PI, Margaret Hedstrom, gave a talk at the December 2011 CNI member meeting:

My colleague, Robert McDonald at IU, has a nice post about the project, and I've stolen the photo of the team from the post to reproduce here:

Margaret is in the middle, third from the left in the front row.  Robert is third from the left in the back row.  ICPSR Director George Alter and I are on either side of Robert.

A look back to 2000 (part 2)

From the 1999-2000 ICPSR Annual Report section on Computer and Network Services:

The mission of the Computing and Network Services Group is to provide the technical  resources necessary to support ICPSR's mission: the preservation, acquisition, and distribution of social science data and the education of the social science community in its use. Computing and Network Services (CNS) has four major areas of responsibility:

• CNS consults with users at ICPSR member institutions regarding software and hardware selection and works to resolve any technical problems encountered in the data delivery process. 

• CNS supports the hardware and software used in the routine operations of ICPSR's member services, archival development, educational, and administrative groups. 

• CNS participates in the assessment and evaluation of ICPSR's operations, providing technical input into the ongoing improvement of processes and procedures. CNS also participates in the development and acquisition of innovative systems. 

• In the long term, CNS is responsible for developing and promulgating a vision of the technical future, for anticipating the impact of new technologies on services, and for ensuring that ICPSR is well-positioned to take advantage of these new technologies.

The high-level mission of the Computer and Network Services team hasn't changed:  It continues to support other parts of the organization.  However, it also plays a much larger leadership role than perhaps it did 10+ years ago, selecting technologies and services that enable capabilities that were heretofore unavailable.  For example, no one asked the CNS team to "use the cloud"; evaluating, selecting, and adopting cloud technologies and cloud infrastructure providers was one of the leadership activities we undertook.

If we dig into the bullet-points, things have changed a bit more, however.

Bullet #1 : I insisted that we change this responsibility soon after arriving at ICPSR in 2002.  Based on past experience I felt it was important that the customer-facing portion of the organization own the relationship with the customer, and while the IT team would provide second-level support to our front-line folks, there was no way we could routinely engage with customers and be effective.  The team just wasn't staffed to do that sort of thing.

Bullet #2 is still true today, but the scope also includes the vendors and virtual infrastructure that we use to deliver services both to internal clients and external customers.

Bullet #3 has been one of the main areas of focus for the team, and continues to be a high priority even today.  I might describe the work we led between 2005-2008 as analyzing and documenting the existing business processes at ICPSR, and then automating the heck out of them wherever possible.  The next phase - starting with FLAME - will be to re-invent business processes at ICPSR so that they are more in harmony with standards like TRAC and OAIS, and then adding automation where it makes sense.

Bullet #4 is definitely still true today, but I think there is room for improvement.  It is too easy to get caught up in the demands of the "next big project" and to stop looking further over the horizon at what's coming next.

And now for the photo contest portion of the post.  Can you identify this ICPSR staff member from 2000?  Mouse over the photo to check your answer.

TRAC: A5.3: Agreement completeness

A5.3 Repository has specified all appropriate aspects of acquisition, maintenance, access, and withdrawal in written agreements with depositors and other relevant parties.

The deposit agreement specifies all aspects of these issues that are necessary for the repository to carry out its function. There may be a single agreement covering all deposits, or specific agreements for each deposit, or a standard agreement supplemented by special conditions for some deposits. These special conditions may add to the standard agreement or override some aspects of the standard agreement. Agreements may need to cover restrictions on access and will need to cover all property rights in the digital objects. Agreements may place responsibilities on depositors, such as ensuring that Submission Information Packages (SIPs) conform to some pre-agreed standards, and may allow repositories to refuse SIPs that do not meet these standards. Other repositories may take responsibility for fixing errors in SIPs. The division of responsibilities must always be clear. Agreements, written or otherwise, may not always be necessary. The burden of proof is on the repository to demonstrate that it does not need such agreements—for instance, because it has a legal mandate for its activities.

An agreement should include, at a minimum, property rights, access rights, conditions for withdrawal, level of security, level of finding aids, SIP definitions, time, volume, and content of transfers. One example of a standard to follow for this is the CCSDS/ISO Producer-Archive Interface Methodology Abstract Standard.

Evidence: Submission agreements/deposit agreements/deeds of gift; written standard operating procedures.

---

My sense is that two main sets of documents are relevant to this TRAC requirement.

One document is the deposit agreement we use for unsolicited, "drive-by" deposits where content comes to us unsolicited.  In this case the agreement is between ICPSR and the depositor, and s/he grants us several rights to the content, such as the right to replicate for preservation purposes, but notably, not copyright.

The other is the set of documents that ICPSR (via the U-M and ISR) signs with agencies and foundations to preserve and disseminate specific collections of content.  These tend to be much larger documents, often including language that runs well beyond "basic" digital preservation, such as detailed requirements and rules for how the content is to be curated before it even enters the repository.

Job posting : Director of Curation Services

ICPSR is hiring.  You can find the job below here.

Director of Curation Services

Job Summary

Director of Curation Services

The Inter-university Consortium for Political and Social Research (ICPSR) seeks to appoint a Director of Curation Services to maintain and develop a comprehensive approach to data management and digital preservation. The holder of this position will lead a team with responsibility for policy implementation and planning across the data lifecycle, including metadata standards, repository management, digital preservation, and compliance with standards (such as Trusted Repositories Audit & Certification). The selected individual will report directly to the Director of ICPSR and will be a member of the ICPSR senior leadership. He or she will represent the organization to ICPSR's extensive network of partners in both the social science and the digital archiving communities.

The individual selected for this position will hold a research faculty appointment in ICPSR and the Institute of Social Research. The position may be in the Archivist (Assistant, Associate or Full), Research Scientist (Associate or Full) or Research Professor (Associate or Full) tracks, depending upon qualifications. Joint appointment with other University of Michigan units is possible.

Organizational Overview
The Inter-university Consortium for Political and Social Research, established in 1962, maintains and provides access to a vast archive of social science data for research and instruction, and offers training in quantitative methods. To ensure that data resources are available to future generations of scholars, ICPSR acquires, processes, and preserves data. In addition, ICPSR provides user support. A unit within the Institute for Social Research (ISR) at the University of Michigan, ICPSR is a membership-based organization, with more than 700 member colleges and universities around the world. A Council of leading scholars and data professionals guides and oversees its activities. More information can be found about ICPSR at www.icpsr.umich.edu.

Responsibilities*

Maintain and develop ICPSR's comprehensive digital preservation policy framework

Plan collaboratively with others to oversee the continuous evaluation of the digital repository, including all policies, standards, and workflows

Identify the necessary descriptive, technical, structural, and preservation metadata for ICPSR's diverse digital content

Evaluate and maintain quality control of metadata operations

Research other metadata standards to determine their relevance for ICPSR

Participate in the Data Documentation Initiative Alliance and in development of the DDI standard

Advise on methods for handling new types of digital content

Manage ICPSR's institutional records in collaboration with others

Promote the role of the organization within the digital preservation and data archive communities through a program of presentations, papers, and articles at conferences and meetings, and in journals representing key domains

Participate in the development of standards and good practice for the digital archiving community at national and international levels

Participate in training on lifecycle data management

Prepare proposals and applications for external funding to support both research into digital archiving and practical activities designed to enhance ICPSR's repository

Conduct and publish in the candidate's area of expertise

Required Qualifications*

M.A. or Ph.D. in an area that relates to information systems, such as Library Science or Information Technology, but we welcome and will consider otherwise qualified candidates with credentials that support the requirements of the position. Five to eight years professional experience in the field, with a minimum of two years of experience in the management of digital archiving activities. Excellent communication skills, both oral and written, in the English language.

Desired Qualifications*

Demonstrated ability to analyze, assess, and find creative digital curation solutions in a complex and dynamic research-based environment. Experience with creating and applying metadata standards to describe digital collections. Familiarity with data used in social science research and protection of confidentiality in research on human subjects. Experience in a managerial role (including fiscal responsibility and supervision of staff) on multiple projects. Demonstrated scholarly productivity, along with the proven ability to work both independently and collaboratively in a diverse environment. Experience in writing grant proposals and applications, and in serving as a principal investigator.

Applicants should submit a letter of application, a CV, three letters of reference, and relevant writing samples to: George Alter, Director, Inter-university Consortium for Political and Social Research, Institute for Social Research, P.O. Box 1248, Ann Arbor, Michigan 48106-1248. Screening of applications will begin shortly and continue until the position is filled.

U-M EEO/AA Statement

The University of Michigan is an equal opportunity/affirmative action employer.

ICPSR Web Availability - February 2012

Things looked much better in February than January.  Chart?  Chart.

Click to enlarge

The big difference this month is that we didn't perform a major upgrade in hardware and software, and aren't suffering from the associated technology hangover.  That said, there continue to be many short-lived (under five minutes) periods where a single element of the delivery service (e.g., the bibliography search) perform so poorly that we count the service as being "down."  Tracking down the root cause of these mini-outages has been only slightly more rewarding than hunting for snipe.

We did make use of our replica in Amazon's cloud for an extended maintenance window in the middle of the month.  The U-M central IT organization replaced our somewhat long-in-the-tooth network gear that connects us to the campus backbone with newer systems.  My sense is that we don't push the network all that hard at ICPSR, but this may change as we begin delivering video in the future.

A look back to 2000 (part 1)

I was looking through the Internet Archive's Wayback Machine the other day, trying to find an electronic copy of a printed document I have.  I wasn't able to find the document I wanted, but I did stumble upon a copy of the 1999-2000 ICPSR annual report.  This appears to have been published during the interregnum between Richard Rockwell and Myron Gutmann.  Myron took over as the Executive Director of ICPSR in 2001, and hired me in 2002 to lead the Computer and Network Services team.

As I read the report it occurred to me that some parts of it were just as relevant today as 10+ years ago.  In some cases the issues and observations are exactly the same; only the players and the technologies have changed.

I thought it might be interesting to compare excerpts from the Computer and Network Services section of that report, and compare them to today's ICPSR.  There's quite a bit of text in that old annual report, and I think it will make sense to break it down into smaller pieces for easier reading and commentary.

I also found a wonderful array of old photographs of my colleagues at ICPSR.  Some of them have left ICPSR, and some of them still work here today.  I'll include one of them in each of the follow-up posts too.  If you mouse over the picture, it should identify the person.

TRAC: A5.2: Transfer of rights

A5.2 Repository contracts or deposit agreements must specify and transfer all necessary preservation rights, and those rights transferred must be documented.

Because the right to change or alter digital information is often restricted by law to the creator, it is important that digital repository contracts and agreements address the need to be able to work with and potentially modify digital objects to keep them accessible. Repository agreements with depositors must specify and/or transfer certain rights to the repository enabling appropriate and necessary preservation actions for the digital objects within the repository. (This requirement is linked to A3.3.)

Because legal negotiations can take time, potentially preventing or slowing the ingest of digital objects at risk, it is acceptable for a digital repository to take in or accept digital objects even with only minimal preservation rights using an open-ended agreement and then deal with expanding to detailed rights later. A repository’s rights must at least limit the repository’s liability or legal exposure that threatens the repository itself.

Evidence: Contracts, deposit agreements; specification(s) of rights transferred for different types of digital content (if applicable); policy statement on requisite preservation rights.

---

The ICPSR deposit agreement is pretty clear about this:  The depositor is granting us non-exclusive rights to preserve and disseminate the content, and in order to perform these tasks effectively, we reserve the right to transform the content.

In fact, unlike a more conventional archive where the goal might be to preserve the item "as is" indefinitely, much of our work - particularly in the topical archives - involves changing the content in significant ways.