Clients v. customers | services v. products

Seth Godin has another excellent post.  This one notes the distinction between customers (who decide whether or not to buy your product) and clients (who pay you to make things for them).

Seth Godin

In the context of ICPSR I think we have a product we call "ICPSR membership."  Customers buy it (or not), and if they do, they receive a reasonably well defined set of services, largely centered around the ability to access high quality datasets and documentation.  We have many hundreds of customers for this product.  I think our Summer Program is also a product, and that too has may hundreds of customers.

We also have a smaller number, perhaps a dozen or so, of clients.  In the best case we have a handful of clients who all pay us to perform a similar set of tasks for them: curate their datasets and documentation, preserve the curated artifacts, and publish the content on a specially "skinned" version of the ICPSR web site for all the world to see.  Adding more clients who want us to do this kind of work benefits all of the other clients, and, often, our customers too.

And like any organization which draws much of its revenue from contract work for clients, we also have those that push us in new, different directions, sometimes for the better, and sometimes for the worse.  The trick, of course, is not too try to head off in too many different directions at once.  And to favor those clients who pull us in better, not worse, directions.

Great FLAMEing file identification service

Some parts of the FLAME project will lend themselves to a microservices approach.  Microservices, like cloud computing, is a trendy, useful concept, but without a crystal clear definition.  But my take is that a microservice is something that performs one small, but useful bit of work, and which can be swapped in and out of an overall architecture at a component level.  It needs to have very clear inputs and outputs, and cannot contain any "secret sauce" that isn't part of its functional role.

Do not try this street magic at home.

One common activity at ICPSR is automated file identification.  Historically we've done this with the venerable UNIX utility file, but where we modify the magic database heavily, particularly for the formats we see most often.  We also post-process the output from file where we need additional handling above and beyond the capabilities of the magic database (e.g., making decisions based on the name or extension of the file).

Managing the magic database is not for the faint of heart.  (Try updating the Vorbis section.)  And this management has gotten both harder -- RHEL 6 uses a new format for its magic database which is incompatible with RHEL 5 -- and easier -- the new format eliminates the pesky magic.mime database.  However, we've gotten reasonably competent at managing magic and have come to rely on it for file format identification.

In support of the FLAME project we even created a little web service that takes a file's content and its name as input, and delivers a little snippet of XML as the output.  The XML contains the "human readable" answer from our magic database and the "MIME type" too.  This is our first FLAME-inspired web service.

If you'd like to try it, you can use your favorite form-capable URL transfer utility to do so.  Here's an example where I have run curl on one of our RHEL machines:

dhcp-bryan:; curl -F "file=@uuid-comparison.xlsx;filename=uuid-comparison.xlsx" www.icpsr.umich.edu/cgi-bin/wsifile<?xml version="1.0" encoding="utf-8"?><wsifile><ifile>Microsoft Excel</ifile><ifilemime>application/zip; charset=binary</ifilemime><uploadInfo>application/octet-stream</uploadInfo></wsifile>

feeding in an Excel file as the input, and another with a plain text file:

dhcp-bryan:; curl -F "file=@/etc/resolv.conf;filename=resolv.conf" www.icpsr.umich.edu/cgi-bin/wsifile<?xml version="1.0" encoding="utf-8"?><wsifile><ifile>ASCII text</ifile><ifilemime>text/plain; charset=us-ascii</ifilemime><uploadInfo>application/octet-stream</uploadInfo></wsifile>

and an interesting MS Word file:

dhcp-bryan:; curl -F "file=@2011-03CouncilPandAminutes.doc;filename=2011-03CouncilPandAminutes.doc" www.icpsr.umich.edu/cgi-bin/wsifile<?xml version="1.0" encoding="utf-8"?><wsifile><ifile>CDF V2 Document, Little Endian, Os: Windows, Version 5.1, Code page: 1200, Number of Characters: 0, Name of Creating Application: Aspose.Words for Java 4.0.3.0, Number of Pages: 1, Revision Number: 1, Security: 0, Template: Normal.dot, Number of Words: 0</ifile><ifilemime>application/msword; charset=binary</ifilemime><uploadInfo>application/octet-stream</uploadInfo></wsifile>

Feel free to try it out, and to post reactions, suggestions here.

Collaborators, not depositors

ICPSR should stop accepting deposits.

Instead ICPSR should be recruiting collaborators.

To be sure ICPSR receives a great deal of its content via US Government agencies who have decided to outsource the digital preservation of their content to a trustworthy repository like ICPSR.  In this case the relevant contract, grant, or inter-agency agreement makes it clear what content will be coming to ICPSR to be curated and preserved.  In some cases the agency has little interest in depositing content ("Isn't that what we pay you for?"), and so the formal act of depositing content falls to the ICPSR staff anyway.

However, we also receive a considerable volume of content through our web portal where the depositor is external.  Sometimes we have worked hard to acquire the content, and the deposit is one milestone on a very long road, but other times the content comes to us unsolicited.  (I like to call these "drive-by deposits.")

In some cases the depositor is quite eager and able to help ICPSR with much of the curation work:  drafting rich descriptive metadata; organizing survey data and documentation into coherent groups; packaging other types of content into logical bundles (such as with our Publication-Related Archive); and, reviewing the data for possible disclosure risks.  Depositors may have access to resources like graduate students who can help with these tasks, and if the depositor is also the data producer, then s/he has valuable, unique insight into the data and documentation.  Unfortunately ICPSR is not well poised to tap into that expertise and those resources.

What would it take to get there?

ICPSR could separate the transactional step of submitting content (i.e., file upload concurrent with signature) from the iterative step of preparing metadata applicable to the submitted content.  In fact, one could even prepare metadata well before the submission transaction if the data producer had the interest and resources to prepare that information, but was not quite ready to share the data yet.  And, it would be equally permissible to submit the data for preservation and sharing, and then build the metadata slowly during the weeks and months following the upload.

If the data producer could also export the metadata in machine actionable formats, say, DDI XML for content which maps well to the classic "study" object that ICPSR has curated and preserved for decades, then there may be additional value to the producer. And introducing the structure that comes along with an XML schema like DDI might also be valuable to the producer in terms of thinking about and organizing the documentation, even for his/her own use.

In this world the ICPSR deposit system becomes a much shorter, much simpler web application.  And the ICPSR data management infrastructure would need to be opened up -- but with serious access controls -- so that content providers could access, create, and revise their documentation and metadata.  But the best thing about this world is that ICPSR gains a lot of collaborators, some who would be quite eager to work with us, I think.

ICPSR's Secure Data Environment overview

Jenna Tyson is a graphic artist on staff at ICPSR.  Over the fast few years Jenna has helped me out with displays for poster sessions, transforming the mediocre layout I produce with a true work of art.  I've posted some of her work here in the past.

I asked Jenna if she could create a logo for our Secure Data Environment (SDE), and above you can see the one that I liked best.  I leave it as an exercise to the reader to decide if the terrified individual in the picture is a defeated intruder or a frustrated ICPSR data curator.

The blog contains several posts that go into some detail about the software and security components behind the SDE, but I'm not sure that I ever posted a high-level description to set context, scope, and purpose.  And so along with Jenna's logo, I present the story behind the SDE.

---

The ICPSR Secure Data Environment (SDE) is a protected work area that uses technology and process to protect sensitive social science research data from accidental or deliberate disclosure.  The SDE exploits commonly used security technologies such as firewalls, ActiveDirectory group policies, and network segmentation to minimize unwanted access between the SDE and outside world.  Further, it takes advantage of work processes which require strict control of when data may be moved between the SDE and external locations.

Data enter the SDE through ICPSR's deposit system.  Depositors upload their content to a web application on our public web portal where it is encrypted.  An automated process "sweeps" content from the portal several times per hour, moving it to the SDE, where it is then unencrypted.  The content resides on a special-purpose EMC Network Attached Storage (NAS) appliance which services ICPSR's SDE.  The appliance uses private IP address space which is only routed within the University of Michigan enterprise network, and is also protected by a firewall.  Further, NAS shares are exported only to specific machines and only to specific ActiveDirectory groups.

ICPSR data managers must be located on the University of Michigan enterprise network to access the SDE.  (They may use the University of Michigan VPN client to access the network from remote locations, and this requires strong authentication and implements strong encryption.)  Data managers run a simple utility to "log in" to the SDE.  Once logged into the SDE they are assigned to a disposable virtual Windows 7 desktop system which is configured to persist any content on the ICPSR SDE NAS.  Any content stored on the virtual desktop system is destroyed once the image is terminated.

Data curators are not allowed to access the Internet or email within the SDE, and they do not have access to local system ports (e.g., USB).  Clipboards are NOT shared between the SDE and the local machine, and so there is no ability to "cut and paste" between the two environments.  It is possible, of course, for data curators to take notes from what they see on the screen, and to take screen snapshots, but ICPSR management considers these to be acceptable risks.

Data curators may release data from the SDE via two mechanisms.

One, they may submit completed work for release via an internal work system called turnover.  This queues material for placement in archival storage, and also queues related material for release on the web site.  A release manager reviews all content before allowing it on the web site.

Two, they may submit unfinished work for transfer outside of the SDE.  In this case a request appears in the inbox of the data curator's supervisor who may then review the request, and then accept or reject it.  If accepted the content is available to the data curator through a simple file retrieval mechanism, and the transfer is logged.

ICPSR has contracted the services of a "white hat" ethical hacker to assess the security vulnerabilities on the SDE.  ICPSR has already implemented small changes within the SDE based on preliminary reports from the contractor.

October 2011 deposits at ICPSR

Time again for the monthly report of new deposits at ICPSR.  Here is the snapshot from October 2011:

# of files	# of deposits	File format
4	3	application/msaccess
46	3	application/msoffice
2496	31	application/msword
2415	7	application/octet-stream
489	45	application/pdf
93	16	application/vnd.ms-excel
6	1	application/vnd.wordperfect
15	2	application/x-dbase
2	1	application/x-empty
1130	14	application/x-sas
1867	31	application/x-spss
1193	14	application/x-stata
1	1	image/gif
1	1	image/jpeg
2	2	image/png
4	1	image/tiff
1	1	image/x-photoshop
10	5	message/rfc8220117bit
151	2	text/html
2	2	text/html; charset=us-ascii
114	8	text/plain; charset=iso-8859-1
50	8	text/plain; charset=unknown
5047	33	text/plain; charset=us-ascii
67	1	text/plain; charset=utf-8
4	3	text/rtf
66	7	text/x-c++; charset=us-ascii
1	1	text/x-c++; charset=utf-8
211	7	text/x-c; charset=us-ascii
98	6	text/xml

The volumes are a bit higher this month, especially the number of files.  At least some of the deposits must have been large, containing an unusually large number of files.

In addition to the usual suspects - plain text, stat package formats, MS Word, PDF - we have a very large number of unidentified files this month (2400+ application/octet-stream), and we also have a very small number of interesting formats (images, photoshop).

Using DuraCloud for Archiving and Preservation

I'll be joining Michele Kimpton, CEO of DuraSpace, on a webinar next Wednesday (November 2, 2011).  Our topic is DuraCloud, and how one can use this cloud-based service as part of one's digital preservation strategy.

I think sometimes people will view the cloud as an alternative to keeping and maintaining local copies, but at ICPSR we're using the cloud as an easy-to-manage storage location to supplement more conventional locations, such as local NAS storage and the University of Michigan's "Value Storage" service.

Here is a copy of the invite that went out via email:

DuraSpace

You are invited to attend the following event: Using DuraCloud for Archiving & Preservation Wednesday, November 2, 2011 1:00p.m. - 2:00p.m. Eastern Standard Time Presented By: Michele Kimpton, DuraSpace Chief Executive Officer & DuraCloud Project Director & Bryan Beecher, Director of Computing & Network Services,Interuniversity Consortium for Political and Social Research (ICPSR) Having a hard time keeping up with current preservation and archiving practices? Are you finding the task of archiving your content complicated, costly and confusing? Then you need to join us for a free webinar that details how DuraCloud can be part of your preservation and archiving solution. This webinar will discuss how to use DuraCloud as a component of your archiving and preservation strategy. An overview of the service will include what it is, how it works, and the benefits it has to offer. Additionally, Bryan Beecher, Director of Computing & Network Services at ICPSR, will present ICPSR's preservation and archivingstrategy. Bryan will share how DuraCloud and other methods have been implemented to meet ICPSR's preservation and archiving goals. If you are interested in attending please thoroughly complete the registration process (below) to receive your unique login url. Be sure to SAVE the return email you receive from Infinite Conferencing as it will include your unique login information.   To register for this event, use the following link: http://ash-cs7.conferenceservers.com/brands/INF/EP_WebInterpoint/maskpath.html?id=66c84ec486&l=en-US  Below is the call-in information for the event. Via Skype (Free, World): Dial +9900827047086940  Via Phone (Toll, US): Dial +1(201)793-9022 Enter Room Number: 7086940 The maximum capacity for this web seminar is 99 participants. The event will be recorded and slides will be available for viewing after the event at http://duraspace.org/web_seminars. Please contact Kristi Searle at ksearle@duraspace.org with any questions. **Please be aware of Infinite Conferencing System Requirements: -          Internet connection speed of 128 kbps or higher is recommended -          Microsoft Windows XP, Vista, Windows 7, or Server 2003 -          Internet Explorer 6.0 SP2, 7.0, 8.0 & 9.0, Firefox 3.0x/3.5, 4, 5 and Chrome 12 browsers -          Apple Mac with Intel CPU, Mac OS X 10.5/10.6, Safari 4.x, 5.x or Firefox 3.x, Java 1.5+ -          Linux, Unix, or Solaris with Mozilla 1.0+ -          Cookies and Scripting enabled in browser

September 2011 deposits

The report for September:

# of files	# of deposits	File format
1	1	application/msaccess
37	14	application/msword
13	2	application/octet-stream
125	28	application/pdf
37	15	application/vnd.ms-excel
7	4	application/x-sas
77	27	application/x-spss
14	4	application/x-stata
1	1	application/x-zip
4	1	message/rfc8220117bit
4	4	text/html
2	2	text/plain; charset=iso-8859-1
4	4	text/plain; charset=unknown
196	30	text/plain; charset=us-ascii
4	3	text/rtf
1	1	text/x-c; charset=unknown
5	4	text/x-c; charset=us-ascii

Pretty typical formats, and pretty normal volumes.  A few deserve investigation (octet-stream), and a few plain text files have been tagged as C source code (as usual).

August 2011 deposits

Time for the monthly deposit statistics:

# of files	# of deposits	File format
1	1	application/msaccess
2	1	application/msoffice
130	22	application/msword
104	6	application/octet-stream
715	29	application/pdf
30	10	application/vnd.ms-excel
6	2	application/vnd.ms-powerpoint
1	1	application/x-dosexec
1	1	application/x-empty
23	7	application/x-sas
67	12	application/x-spss
14	7	application/x-stata
4	3	application/x-zip
6	2	image/jpeg
6	3	message/rfc8220117bit
34	6	text/html
5	3	text/plain; charset=iso-8859-1
8	4	text/plain; charset=unknown
420	28	text/plain; charset=us-ascii
1	1	text/plain; charset=utf-8
17	2	text/rtf
5	2	text/x-c; charset=unknown
7	1	text/x-c; charset=us-ascii
113	2	text/xml
2	1	very short file (no magic)

Lots of the usual kinds of stuff in August; maybe even a bit more than one would expect given the time of year.

There's the usual mistakes made by our file identity service; we're going to look at replacing or augmenting the current system (the UNIX file utility with a greatly expanded localmagic database + a wrapper that inspects the file extension) with something else.  We've spent just a tiny amount of time tinkering with Tika from the Apache project, and that looks promising.  This might even grow into a web service that we would share with others.

A couple of unusual items that merit closer inspection too, such as the purported DOS executable, and a bunch of (basically unrecognized) bitstreams.

July 2011 deposits at ICPSR

Another month, another deposit summary:

# of files	# of deposits	File format
3	2	application/msaccess
510	16	application/msword
317	2	application/octet-stream
528	22	application/pdf
22	17	application/vnd.ms-excel
1	1	application/vnd.wordperfect
18	1	application/x-123
115	2	application/x-dbase
3	1	application/x-empty
3	3	application/x-sas
212	12	application/x-spss
5	3	application/x-stata
5	5	application/x-zip
1	1	audio/mpeg
1	1	image/jpeg
10	1	message/rfc8220117bit
2	2	text/html
5	2	text/plain; charset=unknown
232	9	text/plain; charset=us-ascii
1	1	text/rtf
1	1	text/x-c++; charset=us-ascii
50	2	text/xml
12	2	video/unknown

The usual suspects appear in the usual volumes:  lots of SPSS, PDF, and MS Word.  There seems to be a lot of dBase in this month's report:  that is unusual, and is worth investigating.  The service that generates MIME type works pretty well most of the time, but is not 100% error-free.  And to that point, I suspect that the purported video files and C++ source code are actually something else.

The two deposits with many "unknown" (application/octet-stream) are worth a look too.  They may be some esoteric format that we do not see all that often.

June 2011 deposits at ICPSR

June 2011 was a very busy time for our deposit system.  The number of deposits was pretty typical, but the number of files was enormous.

# of files	# of deposits	File format
2	1	application/msaccess
23	1	application/msoffice
165	23	application/msword
698	4	application/octet-stream
266	28	application/pdf
144	11	application/vnd.ms-excel
1	1	application/vnd.ms-powerpoint
14	2	application/vnd.wordperfect
141	1	application/x-123
4	1	application/x-arc011lzw
25	1	application/x-dbase
23	1	application/x-dosexec
1	1	application/x-empty
1	1	application/x-rar
19	5	application/x-sas
1307	21	application/x-spss
10	4	application/x-stata
3	3	application/x-zip
20	9	message/rfc8220117bit
8	7	text/html
12	6	text/plain; charset=iso-8859-1
10	6	text/plain; charset=unknown
4386	46	text/plain; charset=us-ascii
2	1	text/plain; charset=utf-8
11	4	text/rtf
7	2	text/x-c++; charset=us-ascii
1	1	text/x-c; charset=us-ascii
1	1	text/x-mail; charset=us-ascii
1	1	text/xml
153	2	video/unknown

In addition to the usual suspects like plain ASCII, SAS, SPSS, MS Word, PDF, we also have some of the usual problems, such as files being reported by the automated checker as containing C or C++ source code, when the truth is that they are likely text/plain instead.

One interesting data point is the pair of deposits that contain video files, and lots of them.  Upon further review these appear to be vintage SPSS files for the IBM PC.  Here's a string that appears in all of the files:

SPSS/PC+ System File Written by Data Entry II

and here is another one:

PCSPSS SYSTEM FILE.  IBM PC DOS, SPSS/PC+ V3.0

From a timestamp located nearby, it looks like these files were from 1994.  Or maybe they were moved from a mainframe to a PC in 1994?


And there are a few others on the list above that would benefit from some human scrutiny as well.

Vaults of Heaven: Visions of Byzantium

Vaults of Heaven: Visions of Byzantium

We made a family trip to one of the University of Michigan's smaller museums, a little gem called the Kelsey Museum of Archaeology.  Even though they added a large new wing for exhibition space, the entire museum is still quite small, and is perfect for smaller children (but who are still old enough to be interested in going to a museum).

We wanted to be sure to check out a special exhibit that ended on May 27, 2001 called Vaults of Heaven:  Visions of Byzantium.  While the museum's permanent collection contains artifacts from the ancient world (Egypt, Greece, Rome), this exhibit featured relatively recent items from between the sixth and fifteenth centuries.  There's an image from the the exhibit on their web site, and I've created a link to it to the left.

One thing that struck me about the items in the exhibit were the similarities -- and the differences! -- between how a museum like the Kelsey preserves objects and makes them available for access, and how a place like ICPSR does it.  Another reminder about how the digital world and the physical world are very, very different.

For example, one item available to view at the Kelsey was a small piece of pottery with an image of a person on it.  No doubt it is very fragile, and it probably needs to be kept in a very safe, climate-controlled location when it isn't on exhibit.  There was a card next to the object that contained some (all?) of the information the museum had collected about it:  When it was likely made, what it was used for, and the identity of the person in the image.  (St Simeon the Stylite.)  This information also needs to be preserved and made accessible, but it would not need to be kept in the same type of storage as the object.  In fact, it may well be the case that the metadata in this case is kept in digital format, and only printed out on a card for access purposes.  And so one of the key preservation tasks would seem to be maintaining a reliable, bi-directional, long-lived link between the metadata and the object.  If the link breaks, then the task of finding the object (or re-discovering what it is) becomes very, very difficult.

In the digital world of ICPSR, we face some of the same issues (climate controlled storage for objects, purchasing and managing storage space, linking metadata to objects), but my sense is that we have a much easier time of it when it comes to linking metadata to objects. 

For one thing, both our objects and our metadata are built from the same stuff - bits - and so keeping them in the same type of storage is easy and makes sense.  (And it sure is much easier to make copies of bits that centuries old pieces of pottery.)

Also, because our stuff resides in the digital world and tends to be kept in a file, there's the filename that one can use to help identify the object, even in the absence of metadata.  And so if I have a digital object without metadata, I still have the filename (and the content, of course) to help me identify it.

And, for some type of files, like PDF, one can bundle a great deal of the metadata inside the file itself.  This creates a very close coupling between the object and its description.  This type of close coupling is also available via some of the stat packages, but becomes less useful if the file may only be read successfully with proprietary software.

April 2011 deposits at ICPSR

April deposits at ICPSR:

# of files	# of deposits	File format
88	13	application/msword
111	20	application/pdf
11	4	application/vnd.ms-excel
15	6	application/x-sas
27	11	application/x-spss
3	2	application/x-stata
1	1	application/x-zip
2	2	message/rfc8220117bit
1	1	text/html
1	1	text/html; charset=us-ascii
3	3	text/plain; charset=unknown
113	17	text/plain; charset=us-ascii
1	1	text/rtf

Nothing too exciting this month.  Lots of the usual types of documentation formats (PDF, Word, plain text), and lots of the usual types of data (SAS, SPSS, Stata, and Excel).

ICPSR has been busy chatting with our good friends at DuraSpace over the past month or two.  We have been an active member of their DuraCloud pilot.  This is a hosted service for content and services where the big cloud providers deliver the compute and the storage, and DuraSpace delivers the software and services.  Our main use has been as a supplement to our archival storage solution.

The project looks like it will soon finish the jump from "research pilot" to "production service."  We participated in a webinar yesterday which introduced the updated management console.  It looks good, but we did volunteer one feature request:  an "financial administrator" role.  The idea is that a login assigned to this role would have read access to invoices and financial statements, but not have any access to the content, services, etc.  This is a role we would love to have with the Amazon AWS IAM stuff, but the Amazon guys still haven't identified the monthly bill as one of the system elements that would benefit from such a role.  (And so that means that someone like me has to navigate through the management console to grab billing information each month, then save it, upload it into the absolutely ghastly UMich financial systems, and ....)

DuraCloud is a nice fit for ICPSR since it gives us a single management interface for syncing content to multiple cloud providers (Amazon and Rackspace today, but Microsoft down the road), and for invoking preservation-oriented services over the content, such as fixity checking.

You can find more info about DuraCloud on their web site, and a nice little piece they wrote about ICPSR too.

February Deposits at ICPSR

February 2011 deposits (and their file formats) at ICPSR:

# of files	# of deposits	File format
1	1	application/msoffice
284	23	application/msword
11	4	application/octet-stream
104	25	application/pdf
7	5	application/vnd.ms-excel
24	10	application/x-sas
80	28	application/x-spss
26	4	application/x-stata
2	1	application/x-stuffit
2	2	application/x-zip
19	6	message/rfc8220117bit
15	3	text/html
9	2	text/plain; charset=iso-8859-1
2	2	text/plain; charset=unknown
122	21	text/plain; charset=us-ascii
9	5	text/rtf
1	1	text/x-c; charset=iso-8859-1
22	7	text/x-c; charset=us-ascii
1	1	text/xml

Nothing too exciting this month. 

[ I thought I had posted this weeks ago, but clearly not. ]

January 2011 deposits at ICPSR

First, the brief snapshot:

# of files	# of deposits	File format
4	2	application/msaccess
19	8	application/msword
11	3	application/octet-stream
162	23	application/pdf
4	3	application/vnd.ms-excel
4	4	application/x-sas
123	26	application/x-spss
1	1	application/x-stata
6	1	image/tiff
2	2	text/plain; charset=iso-8859-1
2	2	text/plain; charset=unknown
74	15	text/plain; charset=us-ascii
1	1	text/plain; charset=utf-8
6	4	text/rtf
1	1	text/x-c++; charset=us-ascii
4	2	text/x-c; charset=unknown
7	4	text/x-c; charset=us-ascii

In most ways this was a pretty typical month; most of the content coming into ICPSR continues to be survey data in either plain text or stat package format, and the accompanying documentation is a mix of PDF and word processing formats.

The last three rows where we purportedly received C and C++ source code are almost certainly wrong; our automated content identification service is based on a locally modified version of file, and we've found that file is a little too quick to peg things as C source code.  No doubt this is due to the environment in which file was created and developed, but it is a hard problem to fix in a sustainable, general way.  Do we hack up our local magic database even more?  Maybe even eliminating the entries for C and C++ source code?  Or do we post-process the output, transforming entries like text/x-c++ to text/plain?  Or do we maintain a separate version of our improved file utility just for incoming deposits?

I think in the long run we might decide to live with the problem, using the output of our identification service as more of a recommendation, and we'll rely on the team of data managers to change those recommendations that don't match reality.

Updated search capability

One of the new things we're releasing in February is an improved search on the ICPSR web site.  The change isn't in the way that people use the search; rather it is in how we build the index.

The ICPSR "study search" index has always used the rich set of metadata that our data managers create during the data curation process.  This contains the usual items one might expect to find in an index, such as the name of the researcher, the name of the study, subject terms and headings, etc.

Our new index uses this same rich metadata, but also makes use of the full-text available in documents such as codebooks and survey instruments.  Our preliminary findings have been very encouraging:  In some cases, studies that would have been hard or impossible to find with a "metadata only" search appear high in the search results with a more broad index.

For example, if you were to search using the terms "warfare" and "africa" in our current search, you would end up with this URL in your browser's address bar:

http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies?archive=ICPSR&q=warfare+africa

and three results.

However, with the new search, you will end up with over 30 results, including many studies that are in the World Military Expenditures and Arms Transfers set.  You can take a sneak peak at the new capability by adding the string "&newSearch=true" to the end of the URL in the address bar.  For instance, to do the search above using the new index, use this URL:

http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies?archive=ICPSR&q=warfare+africa&newSearch=true

and see the difference.

ICPSR Deposits - November 2010

Here is a snapshot of the types of deposits and deposited files that ICPSR received in November 2010.

My sense is that this was a fairly typical month both in terms of volume and the types of files.  We always get lots of documentation and related text in one of the common formats, such as PDF and MS Word.  And the main sources of data come in one of the big stat packages, like SAS and SPSS, or in plain text.

# of files	# of deposits	File format
3	2	application/msoffice
120	14	application/msword
202	33	application/pdf
3	2	application/vnd.ms-excel
22	6	application/x-sas
40	13	application/x-spss
3	1	application/x-stata
2	2	message/rfc822\0117bit
1	1	text/html
3	1	text/html; charset=utf-8
5	5	text/plain; charset=iso-8859-1
16	8	text/plain; charset=unknown
110	31	text/plain; charset=us-ascii
48	1	text/plain; charset=utf-8
1	1	text/rtf
1	1	text/x-c++; charset=us-ascii

Fedora Content Model for Social Science Research Data - Redux

A group of us have been getting together once per week for the past month or two to revisit some of our earlier decisions about social science research data and how we intend to store it in Fedora.  (You can find the original content model by searching the blog for the tag 'eager' -- this work is supported by an NSF INTEROP EAGER grant.)

Our thinking about the type of Fedora objects that we would like to use has shifted from our first thoughts in 2009.  The original objects aimed to group related content within the same object, but in different Datastreams.  We are now thinking of using much simpler objects where the content forms one Datastream, and any related content is packed into its own separate objects, linked together using the RDF syntax available in RELS-TXT.  If the file-level object has metadata which doesn't fit well into existing places, then we may create a second Datastream to collect it.  For example, if we want to record the preservation actions performed on the file/object,we think it makes sense to capture that in a PREMIS-format Datastream stored alongside the actual file/object content.

I'll kick-off this continuing line of posts tomorrow with an example container for what we call a Deposit at ICPSR.  This is the container that a researcher, government agency, or even an ICPSR staffer use to move content into the data curation systems of ICPSR.

NSF Social, Behavioral and Economic Directorate suggests ICPSR for data archiving

The much anticipated NSF guidelines on data management were released earlier this week.  One highlight (especially for those of us working at ICPSR) is that the NSF explicitly recognizes ICPSR as a good option for archiving quantitative social science data.

The SBE Directorate supplements agency-wide guidelines with some of its own, and has this to say:

Quantitative Social and Economic Data Sets 

For appropriate data sets, researchers should be prepared to place their data in fully cleaned and documented form in a data archive or library within one year after the expiration of an award. Before an award is made, investigators will be asked to specify in writing where they plan to deposit their data set(s). This may be the Inter-University Consortium for Politicaland Social Research (ICPSR) at the University of Michigan, but other public archives are also available. The investigator should consult with the program officer about the most appropriate archive for any particular data set.

DuraCloud fixity service testing

Our DuraCloud pilot test is going well. We have uploaded a test collection of nearly 70k files, representing that portion of our archival content that contains public-use datasets. (The datasets are public-use, but our licensing terms restrict access to some of these to our member institutions.)

To the left you can see a snapshot from the DurAdmin webapp that one uses to manage content. I've been using this webapp to view content, check progress, and download files. I've been using a command-line utility called synctool for copying content from ICPSR into DuraSpace, and keeping it synchronized.

The image to the left is the right-side panel from the Services tab of the DurAdmin webapp. I've deployed the Fixity service, and am using it to check the bit-level integrity of the content.

I started the service earlier this morning, and it still has quite a bit of work left to do. The processing-status line shows that the service has started, and that it is checked about 4300 of the files so far.