ICPSR NSF INTEROP project heads to Vancouver for IASSIST 2011

Here is a sneak preview at the poster we've put together for the IASSIST 2011 conference for our NSF INTEROP EAGER project investigating Fedora as a repository solution for social science datasets.

Mary Vardigan, Assistant Director of ICPSR, and Director of Collection Delivery, will be presenting the poster at the IASSIST 2011 conference's poster session on Thursday, June 2, 2011.

In the poster we've tried to capture the thinking that went into our object model for social science research data and documentation, and how we mapped that to Fedora's Content Model Architecture.  We also highlight the other two deliverables of the project, a tool for generating FOXML-format objects for ingest into Fedora, and a suite of still-evolving services that may be applied to the objects.

Fedora Object for overall deposit - example

So far we have seen objects for each of the files belonging to a single deposit, but we have not yet seen a parent-level or aggregate-level container for the deposit itself.  Most of the information we have collected has so far been at the file-level, but what about information that relates to the entire set of files?

For this purpose we'll use a separate Fedora Object.  It's RDF will assert a relationship to each of the file-level Fedora Objects, and as we have already seen, each of them asserts a relationship to this object on the left.

At this time we are using a generic Fedora Object for the higher level deposit object, but it may make sense to create a separate Content Model for it if we know that it must always have a Datastream that contains a history of deposit-related actions.

For the example object we've created the object to the left.  (The object is also a hyperlink to our public Fedora repository.)

The Dublin Core Datastream is relatively empty.  Most of the content is captured in a PREMIS XML Datastream at the end.  At the time of this post the object contains PREMIS which has been built by hand, and so may not be quite correct.  But if the syntax isn't quite correct, we think that the concept is.

The PREMIS is very basic.  An obvious enhancement would be to add additional stanzas to capture the terms to which the depositor agreed to help flesh out the Access Rights portion of the Preservation Description Information (PDI).

Next up I'll share our summary of how we think we will generate a Submission Information Package for each of the items in the deposit.

Fedora Objects for deposited files - example

We are using deposit #15868 as an example to illustrate the Fedora Objects we will use to store social science research data and documentation that has been deposited at ICPSR.  We chose this deposit since each of the files are readily available on a public web site and pose no risk of disclosure.  Each of the images below is also a hyperlink to the corresponding object in our public Fedora Commons repository.

The first file in the deposit contains the survey data.  We assign a unique ID (Reference) for the file that will not change.

In this case the survey data are in a format produced by the statistical analysis software called SAS, and our file format identification software has assigned it the MIME type of application/x-sas.  This content goes into its own Datastream (last one on the left), and we Fedora calculates a message digest to fingerprint the file (Fixity).

We note the original name of the file in the DC Datastream along with the identity of the depositor, the origin of the file, and the identity of the organization that created the file (Provenance).

 We capture its relationship to the higher-level deposit transaction via a relationship in the RELS-EXT Datastream, and we also later capture what role this file plays in the data curation lifecycle at ICPSR (Context).

Not captured or shown at this level are the terms to which the deposit agreed when transferring this content to ICPSR (Access Rights).  We will store those in the aggregate-level object. Typically the depositor grants ICPSR non-exclusive rights to reproduce and publish the content, but this is not exclusively true.

Likewise, we capture similar information for the other three files in the deposit:

In the next blog post on this topic, I'll publish a description of the aggregate object to which these four assert an isPartOf relationship.  Once we have that object as well, we can begin talking about producing an OAIS Submission Information Package (SIP) for each object.

Fedora objects for deposits

Researchers and government agencies (and their proxies at ICPSR) use a web portal called the Data Deposit Form to transfer content to ICPSR.  The form contains many opportunities for a depositor to enter metadata about the transfer, but only a few are required:  the name of the depositor and a name for the deposit.

A deposit may have an arbitrary number of files, and those files may be uploaded individually or as a single "archive" file, such as a Zip or GNU Zip archive.  In a case where the depositor uploads an archive file, ICPSR unpacks it to extract the actual content.  And if the archive file contains an archive file, ICPSR systems continue unpacking recursively.

Our intention is to put each of the deposited files (unpacked, if necessary) in its own Fedora object.  This object will be an off-the-shelf object without any special Content Model.  Here is an example:

(Note that all of the images are also hyperlinks to Fedora Objects in our public Fedora Commons repository.)

This is a standard Fedora Object, conforming only to the Content Model for all objects.

Each deposited file contains a unique ID captured in the PID, and the usual, minimal Fedora object properties.

We also enable the Audit Datastream to record any changes to the object, and use the DC (Dublin Core) Datastream to capture some of the metadata we collect via our Data Deposit Form.

We use a relationship expressed in the RELS-EXT Datastream to point to a parent-level object which is used to link the files within a single deposit and to capture any metadata which applies to the entire deposit, not just the individual files.

The content is highly variable.  In addition to receiving survey data in plain text, ICPSR also receives data in a variety of proprietary formats (e.g., SAS) and related documentation in a wide array of formats (word processor output, plain text documents, and many others).

To illustrate the example further, we created a Fedora Object for each of the files found in one of our recent deposits.  We selected this deposit because the content is entirely public-use, and is readily available from a public web site.  The deposit is also a nice size (only four files).  To keep this blog post at a reasonable size, I'll save the example for tomorrow's post.

Fedora Content Model for Social Science Research Data - Redux

A group of us have been getting together once per week for the past month or two to revisit some of our earlier decisions about social science research data and how we intend to store it in Fedora.  (You can find the original content model by searching the blog for the tag 'eager' -- this work is supported by an NSF INTEROP EAGER grant.)

Our thinking about the type of Fedora objects that we would like to use has shifted from our first thoughts in 2009.  The original objects aimed to group related content within the same object, but in different Datastreams.  We are now thinking of using much simpler objects where the content forms one Datastream, and any related content is packed into its own separate objects, linked together using the RDF syntax available in RELS-TXT.  If the file-level object has metadata which doesn't fit well into existing places, then we may create a second Datastream to collect it.  For example, if we want to record the preservation actions performed on the file/object,we think it makes sense to capture that in a PREMIS-format Datastream stored alongside the actual file/object content.

I'll kick-off this continuing line of posts tomorrow with an example container for what we call a Deposit at ICPSR.  This is the container that a researcher, government agency, or even an ICPSR staffer use to move content into the data curation systems of ICPSR.

Teaching With Data portal

We've launched a new version of our Teaching With Data portal.

TwD is one of the National Science Digital Library (NSDL) pathways. Each pathway has a domain-specific focus or interest, and ours is using quantitative social science research data as an element of teaching.

Jane Wang is the lead software developer on TwD, and she has been building the site using a Fedora stack called Muradora. One of the design requirements from the research team was that the site had to be based on Fedora, and rather than build a complete portal from scratch, Jane selected the stack that would be the best fit for our development environment.

Our TwD site is also interesting in that it is the first portal that we've deployed completely in the cloud; there is no local hardware at ICPSR supporting the project (except for Jane's laptop). All of the content and software resides in Amazon's cloud.

Another interesting aspect of this release is that we're using OpenID for authentication. This means that instead of needing to create yet another login and password to remember, visitors may instead use an existing login and password from one of many OpenID-compliant sources, such as Google, AOL and Yahoo. We're using a third-party service called RPX from JanRain to enable this feature. I'm expecting that we'll enable OpenID and other authentication systems (such as Facebook Connect) on the main ICPSR web site in the near future.

Of course, the site has other new features as well, and I'd like to encourage you to have a look by clicking the image above.

Back to the Fedora: Part 4

This is the final post in the series.

So far we have introduced a pair of Content Model objects: one for social science data, and one for social science data documentation. In this post we introduce a third Content Model object for social science: an aggregate level object that has some content of its own (descriptive metadata and preservation metadata), but serves largely to group together related objects.

The Content Model object is to the left. It must have two Datastreams: one for the descriptive metadata in DDI XML format, and one for preservation metadata in PREMIS XML format. Note that we may discover that we can use DDI for both purposes, and in that case, the PREMIS Datastream will drop out as a required element.

Like past posts, the image to the left is a link to the ICPSR Fedora test repository, and will return the "home page" for the Content Model object pictured.



To the right we have a Fedora data object which conforms to the Content Model above.

Like the Content Model image, this image is also a link to our Fedora test repository, and clicking it will navigate to the matching data object.

This object has one relationship asserted per member object. In this case we assert three hasMember relationships: one for the survey data object; one for the survey documentation object; and, one for the survey instrument object. These correspond to isMemberOf relationships asserted in those objects, and together they assert a series of bilateral relationships.

The object contains the two required Datastreams. In this case the actual XML is somewhat stylized, and may not be "clean" XML. In particular the PREMIS Datastream is very much a work in progress here at ICPSR, and may bear little resemblance to high-quality PREMIS XML.

Back to the Fedora: Part 3

This is the penultimate post in this series. The final post will describe an aggregate object ("study") that will contain relatively little content, but which serves as a grouping element for more basic elements.

The object to the left is a conventional Fedora Data Object, but I include it here as an example where we have important content to preserve and deliver, and where the content is somewhat of a "one off" and doesn't conform to a unique Content Model.

In this case we have the survey instrument that was used to collect the data in icpsr:eager-survey-data-25041.

The instrument is available in two different languages (English and Spanish), and while the original deposit was PDF-format, we have also produced a TIFF-format of each version for preservation purposes. This translates into a simple object with four Datastreams, one for each (language, format) combination.

We assert membership to the aggregate "study" object in RELS-EXT. We also assert a connection to the associated dataset using a custom relationship we minted: isInstrumentFor. It isn't clear (yet) if having a specialized relationship such as this will be any more useful than a less descriptive relationship (e.g., isRelatedTo, to make one up).

Interoperability Between Institutional Data Repositories: a Pilot Project at MIT

Kate McNeill from MIT pointed me to this interesting paper from the IASSIST Quarterly: Interoperability Between Institutional Data Repositories: a Pilot Project at MIT. (PDF format)

As Kate mentioned to me, this paper describes a tool which transformed DDI-format XML into METS, and it would be well worth exploring if this tool could be used in some way to support a deliverable on our EAGER grant: a tool which transforms DDI-format XML into FOXML.

Fedora supports several ingest formats, including METS and its own native FOXML, and so if there is already a tool that generates METS, that would be a good starting point for a FOXML version. Further, an interesting science experiment would be to take DDI, transform it both into METS and FOXML, ingest both objects, and see how if they would differ in any significant manner.

Back to the Fedora: Part 2

To go along with our survey data object, we'll also need a survey documentation object. We'll relate the objects via RDF in the RELS-EXT Datastream, and we'll also relate the documentation object to the higher-level, aggregate object, "social science study." The image to the left is clickable, and will take one to the "home page" for this Content Model object in the ICPSR Fedora test server.

Note that the name of this Content Model object is somewhat of a misnomer. Even though a common use-case is survey data, we may use the same type of object for other social science data that are not survey data, such as government-generated summary statistics about health, crime, demographics, or all sort of other things.

The heart of the Content Model is in the DS-COMPOSITE-MODEL Datastream where we require a large number of Datastreams: a "setups" Datastream for each of the common statistical packages; a DDI XML Datastream that documents the associated survey data object; and a pair of Datastreams for the human-readable technical documentation (the "codebook"). A future refinement might be to replace the pair - one PDF, one TIFF - with a single Datastream which is both durable for preservation purposes, but which also allows the rich display of information (PDF/A?).



At the right we have a data object that conforms to the Content Model object above. Of course, it contains all of the required Datastreams, most of which are stored as simple text files. The DDI is actually a very large bit of XML which is currently being stored in a separate file rather than as in-line XML (i.e., Control Group M rather than Control Group X in the FOXML).

The relationships in the RELS-EXT Datastream are congruent with those in the associated survey data Datastream. Both assert a hasModel relationship to the applicable Content Model, and both assert a isMemberOf relationship to the higher level object that "contains" them. Here, though, we use the isDescriptionOf relationship rather to show that this documentation object is a description of its related survey data object; in that object we asserted a hasDescription relationship to this object.

Of course, there is nothing preventing us from adding additional Datastreams to an object like this when they are available, such as unstructured notes from the original data collector. However, since that content isn't always available, we don't make it a required Datastream in the Content Model.

Clicking the image to the right will take one to its "home page" on the ICPSR Fedora test server. All of the Datastreams are identical to those on the ICPSR web site, except for the TIFF codebook and variable-level DDI, which we usually do not make available.

Back to the Fedora: Part 1

Now that the NSF EAGER grant has arrived, it's time to get restarted on Fedora. We'll start this iteration with a trio of Content Model objects, and kick it off with the first one in this post.

The first - displayed in a clickable, linked, visual format to the left - is a Content Model object for social science survey data. In addition to the objectProperties and the required Datastreams (AUDIT, DC, RELS-EXT), there is also the standard DS-COMPOSITE-MODEL Datastream found in Content Model objects.

For our purposes we'll require each object that purports to conform to a social science survey data object to have three required Datastreams: ORIGINAL, for original survey data that was supplied by the depositor; NORMALIZED, for a plain text version of the file that repository prepares; and, TRANSFORM, which is a record that describes how the ORIGINAL became the NORMALIZED. This last Datastream is typically constructed as an SPSS Setups file at ICPSR, and internally it is often referred to as the "processing history" file. It contains the roadmap of how to move between the two versions of the data.

It may also be the case that we have other Datastreams, perhaps items that will only receive bitwise digital preservation, such as original deposits in SAS or SPSS format. And, in practice, we might want to use Fedora's XACML mechanism to restrict access to the ORIGINAL Datastream since it could contain confidential information.



To the right we have a sample Fedora data object that asserts conformance with our Content Model object above. Like the one above it is also clickable, and will take you to the Fedora repository server ICPSR is using for testing.

In addition to the hasModel relationship, this object also asserts that it is a member of a higher-level object (ICPSR Study 25041), and is described by another object (which we'll look at in the next post).

As required to validate against the Content Model, it has the three required Datastreams. In this particular case, rather than including the original data and processing history transform, I've simply copied the NORMALIZED Datastream content verbatim into the other two Datastreams.

Not shown in the schematic to the right are other possible. optional Datastreams we could include. For instance, it looks like this object was derived from a deposit that began its life at ICPSR as a SAS Transport file. It would certainly be possible to include that as another Datastream that would have value for a limited period of time. Or, another approach would be to collect the deposited items in their own set of Fedora objects, and then assert a relationship to them in the RELS-EXT section.

Next up in this series: the Content Model for technical documentation.

Teaching With Data Launched

ICPSR launched a new portal this week - TeachingWithData. This is part of ICPSR's grant to build a National Science Digital Library (NSDL) Pathway dedicated to social science data and teaching. I won't say too much more about the content in this post, but will instead focus on our technology selection.

The two main technical requirements were: (1) Fedora-based, and (2) fast deployment. We think we've achieved both with the site.

We implemented the site on an Amazon Web Services (AWS) Elastic Computing Cloud (EC2) instance; this allowed us to stand-up a base platform very quickly. We now have nearly a dozen instances running in AWS. Some are delivering production services, like our search technology and this Pathway, and others are used for science experiments and front-ends for storage. We've found that it's more convenient to synchronize and organize content in AWS by sticking a Linux instance in front so that we can use tools like rsync. Using Simple Storage Service (S3) directly is less convenient.

After looking at a variety of stacks that sit on top of Fedora, we selected Muradora for three main reasons:

1. Open source software
2. Active development community
3. Best technology fit with other technology platforms in use at ICPSR

Selecting Muradora allowed us to stand up the portal relatively quickly, and other time we'll evaulate how well the system is meeting our needs.

More Good News: ICPSR Wins NSF Grant

There's more good news for ICPSR on the funding and technology front. I learned last week that our NSF proposal to the EAGER (EArly-concept Grants for Exploratory Research) program was funded. In the proposal, Shared Digital Technologies for Data Curation, Preservation, and Access: A Proof of Concept, we describe a series of objects (Content Models, Service Definitions, Service Deployments) that we will build as an exemplar on how one might store social science datasets and documentation in a Fedora-type repository.

We have been exploring Fedora quite actively since the release of version 3, and this is a wonderful opportunity to move our work from the wings and into center stage. The majority of our efforts so far have been focused more on transferring our legacy content (the keepsakes) from file-based storage into object-based storage, but the work from this grant will be much more forward-looking. This will be a chance to renew work on the early content modeling work I reported on back in May.

My sense is that between this grant and the NIH Challenge Grant, we'll need to look for a reasonably senior systems analyst/developer before the end of the calendar year. (We may even need to add two.)

ICPSR: Then and Now: Archival Storage

Archival Storage, the OAIS function responsible for storing and retrieving content, was built on DLT IV tapes at ICPSR in 2002. Files that we wanted to keep indefinitely were moved to a pair of DLT tapes; one copy was retained at ICPSR, and the other was stored at an off-site location in Ann Arbor, Michigan.

And, unfortunately, we also had a large number of older tape formats as well: IBM 3480 cartridge and 9-track. Again there were two copies, but in this case, both were off-site.

As you might expect with an off-line system such as this, it was very expensive to retrieve any item from Archival Storage. Also, if the requestor was a little fuzzy about the exact item of interest, that would also add to the cost. There was no good way to browse the holdings, and retrieval time was measured in days not minutes.

Today we've moved the master copy of each file from tape to disk, and we replicate each file off-site using a variety of techniques, such as rsync and the Storage Resource Broker Srsync utility. We also keep a copy on tape too, but instead of DLT IV, we're using LTO-3 tape which is ten times more dense. And so this gives us more copies in more locations, and a high degree of confidence that the copies are synchronized.

The next step in Archival Storage is a move away from file-based solutions to object-based solutions. We've been evaluating Fedora as a possible storage platform for social science datasets and documentation, and the results are very promising so far.

If You Give a Mouse a Cookie

I've spent the last few weeks building objects to ingest in our fledgling Fedora repository. There is a lot of new technology to learn, such as various XML schemas for storing preservation metadata, descriptive metadata, content models, and access controls. And there are also somewhat older technologies with which to become re-acquainted, such as LDAP. (I think my last use of LDAP was a few years after we have developed DIXIE, and I was building a command-line utility that helped test some of the LDAP library routines.)

There have been many times during this process where it has felt like I learned just enough about technology X to get part of the project finished, only to discover that technology Y is also critical. And, of course, technology Y is something where I have little current knowledge.

As I repeated this process, it kept reminding me of one of my daughter's books, If You Give a Mouse a Cookie. The book is an entertaining look at unintended consequences: You give the mouse a cookie, but now it wants milk. So you get the mouse some milk, but now it wants a straw. You get the straw, and .... You get the idea.

And so with apologies to Ms Numeroff and Ms Bond...

If You Give a Mouse Fedora

If you give a mouse Fedora, it'll want to be sure that it is version 3.x

If you give it version 3.x, it'll want to use the new Content Model Architecture (CMA).

To use the new CMA, the mouse will ask you to learn a bit more about FOXML and Content Model objects.

As you work through Content Models, you'll come across WSDL in the Service Definitions and Service Deployments.

WSDL will remind you to brush up on Dublin Core.

So the mouse will ask you to generate Dublin Core (DC) for descriptive metadata, but will want you to use PREMIS for preservation metadata.

Once your objects have been expressed in FOXML and PREMIS and DC, the mouse will want to start adding references to Datastreams. The mouse will need a MIME-type for each Datastream.

To get good MIME types for each Datastreams, the mouse will ask you to use the UNIX file command.

If you use the file command, you'll need to create your own local magic.mime database, since you'll likely have stuff that confuses file.

Once you've added MIME types and ingested your objects, the mouse will want to protect them against unauthorized access. So the mouse will want you to learn XACML.

Once you've learned XACML and written policies to protect your resources, the mouse will want an LDAP server to authenticate users and register attributes.

Once you've deployed the LDAP server, you document all of the steps that you have completed. The mouse will want a place to store this documentation you've written.

And chances are if the mouse wants a place to store important information, the mouse is going to want a Fedora repository.

Fedora Land Speed Record?

I don't know if there is an existing "land speed record" for putting the number of Datastreams into a single object into Fedora, but we have ingested a data object with nearly 17k Datastreams into our Fedora repository. To be honest I was not sure that Fedora would be able to handle it, but it did.

This particular object is related to ICPSR Study 13517, and the Datastreams are largely Census 2000 data files that we pulled from the web site of the Census Bureau in 2003. Overall the Datastreams consume a bit over 4GB of disk space, and their corresponding objects (in FOXML format) use only about 10MB of space. They have very little metadata in the in-line DC and RELS-EXT Datastreams, for example.

We're still working on creating the "keepsake" objects I described in an earlier post, but if Fedora can handle this number of Datastreams, there shouldn't be any problems with other less massive studies.

One small complaint: The Fedora admin tool (fedora-admin.bat on the Windows platform) does not do a very good job with objects that have a large number of Datastreams. Because of the way it tiles the Datastreams and lacks scroll bars, it is pretty much useless for something of this scale. The "out of the box" web services, however, do a fine job displaying the object.

Storing Keepsakes

ICPSR has been around for a long time. A really long time. Fifty years. And archive years are like dog years, so this is really old.

During my nearly seven years at ICPSR I've come to the conclusion that we've collected and stored pretty good metadata about most of our electronic wares. For the past few decades we've used consistent buckets to hold information: this came from a depositor; this we made ourselves; this we derived programmatically from an original source elsewhere; etc.

We have rows and rows and columns and columns of metadata stored away in relational databases; and more recently, we also have a formal tracking system with all of the important events and dates in the history of a study. And in the near future I expect that we'll transition all of this well documented content and metadata into a Fedora repository. This is the good stuff.

But then there is the older stuff.

The stuff of questionable value. The stuff that no one else wants. The stuff that we'll keep anyway.

The keepsakes.

The digital keepsakes.

Before the end of the summer we'll move our digital keepsakes from their current home, and put them into Fedora. We won't need anything terribly sophisticated in terms of Fedora's CMA. We'll create a tool that builds a nice Fedora object (in FOXML) for each of ICPSR's current and former studies, and creates a keepsake object for each.

Two models per study or one?

In an earlier post this week, I proposed a method to store a study with a simple shape - ICPSR 8475 which is the American National Election Survey (cumulative file, 1948-2004) - and put it into Fedora. This method used a Content Model for the study, and then two separate Content Models for the main content, one for the data file and one for the technical documentation. This translated into a pair of Fedora objects that would look something like this:

In this case we have a single data object which conforms to two different Fedora Content Models. The datasetCM requires a text/plain datastream to hold the plain ASCII data, and the docsetCM requires a text/xml datastream to hold the dataset-level DDI-format XML. Our simple data object above has both of those datastreams, and so conforms to both Content Models.

Two separate Content Models is probably over-engineering for this study, but it makes sense for a more complex study. Let's take a look at the Intensive Community Study, ICPSR 6849.

This study features many pairs of data files that share a common codebook. In fact, all but one of the data files are part of just such a pair. In this case bundling the documentation directly with the dataset could lead to duplication in the repository, or the creation of special-case rules in software, and so instead we separate the content into three separate data objects: ICS data, ISR data, and the shared documentation:

We also have a parent-level data object, the study, which serves as a container for these three data objects, plus many other trios of similar pairs of data files with a shared codebook:


In this case the study data object would actually have many more values stored in it's RELS-EXT datastream than are shown above.

Content modeling social science data with Fedora

Here's a first cut at how one might model social science data content (like what we have at ICPSR) in Fedora.




My sense is that the "dataset" is the atomic object of interest. Heading down to the variable level feels too fine-grained for a system like Fedora. And so this led to creating two pretty simple Content Model objects:

1. Dataset - this has one datastream, DATA, and the MIME-type is text/plain
   
2. Docset - this has one datastream, DOCS, and the MIME-type is text/xml, and would hold DDI-format metadata at the dataset-level
   

I thought I would also need a Content Model for the basic unit of dissemination we use:

1. Study - this has one datastream, DOCS, and the MIME-type is text/xml, and would hold DDI-format metadata at the study-level

A simple study at ICPSR, say 8475 which has one data file and one documentation file, might then consist of a pair of Fedora data objects. I'll list the persistent ID (PID) first, followed by a list of datastreams in parentheses.



ICPSR-8475 (DC, RELS-EXT, AUDIT, DOCS)

The RELS-EXT datastream would express relationships to show that it contains two members, a Data object following the Dataset Content Model, and a Data object following the Docset Content Model. The RELS-EXT datastream would also assert a hasModel relationship to the Study Content Model. We also need a mechanism for storing access controls and license terms, but I'm still learning about the XACML stuff that might be a good way to do this. It may also make sense to have a datastream for DDI 2.x metadata and one for DDI 3.x metadata rather than just a single one.

Likewise, we would also have:

ICPSR-ANES-1948-2004 (DC, RELS-EXT, AUDIT, DATA, DOCS)

The RELS-EXT datastream would express a memberOf relationship to ICPSR-8475, a hasModel relationship to the dataset Content Model, and a hasModel relationship to the docset Content Model. Like the study Content Model data object above, still need to sort out the mechanism for storing access controls and license terms, and also if two datastreams v. one for DDI would be appropriate.

This then assumes that we might create the dissemination formats, like SPSS Export, on the fly since there isn't a datastream for that stuff. That might work, or we could add additional datastreams to the dataset that would point to the dissemination formats. We might want these to use the Externally Reference Content control group if we consider the dissemination formats to be somewhat ephemeral; the alternative would be Managed Content if we wanted to be able to manage the content in Fedora, perhaps allowing one to roll back to previous versions.

Next: So why two different Content Models, one for data and one for documentation, rather than just a single one?

Fedora at ICPSR

We've been spending a lot of time this month at ICPSR working with the Fedora digital repository software. It looks like a lot of people have downloaded the software (the Fedora Commons web site says that there have been over 25,000 downloads), but the number of organizations using it is still relatively small compared to the number of downloads, less than 200. I suspect that some of those listed organizations, like ICPSR, are using Fedora in sand box environments rather than for production use.

For those not familiar with Fedora, it is an acronym which stands for Flexible Extensible Digital Object Repository Architecture. It's software that was developed originally at Cornell in the late 1990's and is freely available today. After exploring the system for a few weeks now, we'd certainly agree that the name is apt.

One, this really is architecture level software, sometimes called middleware. It isn't a finished content management system (CMS), say like Drupal or Plone, where one performs a quick install and then starts adding content. The software does come with a few demonstration objects already loaded, and a handful of basic mechanisms to browse and display the content, but ultimately one needs to build a "stack" on top of Fedora to really use the system in the way it was intended. This means building your own stack, or using one of the ready-made stacks such as Fez, Muradora, Islandora, and so on. In addition to test-driving Fedora, we've also been looking at the stacks.

Two, the system does force one to think seriously about the precise nature and shape of one's digital content. What are the atomic objects and what are the higher-level digital object molecules they form? What's actually a relationship between two objects v. an attribute of an object? What's the essential content that should be preserved, and what is merely a derivative? If one has a bunch of image-oriented content, there are lots of good examples available for how one might decide to organize the objects; if one has a bunch of social science data, the examples aren't as applicable (but are instructive nonetheless). This is indeed extremely flexible and extensible stuff.

One small complaint I have is with the name: When Thorny Stapes from the Fedora Commons visited us in the fall, he told us the story about how the Cornell folks got the name first, and how they reached a compromise with the Red Hat guys when they created their Fedora distribution of Linux. But the problem for folks like me is that it makes it very hard to find web pages and information about the Fedora repository software without some heroic efforts with search engines. For example, I use Google Alerts as a mechanism to collect information about items of interest, and my query for Fedora is longer than the rest of my Google Alerts combined. And it also turns out that there is a pretty popular college football coach who also has the name Fedora!

Up next: Short summaries of the three Fedora-related projects underway today.