B3.3 Repository has mechanisms to change its preservation plans as a result of its monitoring activities.
The repository must demonstrate or describe how it reacts to information from monitoring, which sometimes requires a repository to change how it deals with the material it holds in unexpected ways. Plans as simple as migrating from format X to format Y when the registries show that format X is no longer supported are not sufficiently flexible—other events may have made format Y a bad choice. The repository must be prepared for changes in the external environment that may make its current plan (to migrate from X to Y in 10 years) a bad choice as the time to implement draws near. The repository should be able to show that it can revise long-range plans in light of changing circumstances.
Another possible response to information gathered by monitoring is for the repository to create additional Representation Information and/or PDI.
Evidence: Preservation planning policies tied to formal or information technology watch(es); preservation planning or processes that are timed to shorter intervals (e.g., not more than five years); proof of frequent preservation planning/policy updates.
My team at ICPSR supports this function, but is not directly responsible for preservation planning, so this post will be brief.
My experience is that ICPSR has a very robust preservation planning function which engages regularly with a wide array of communities, and which therefore keeps a close eye on changes in what the community needs, or changes that are necessitated by technology.
I suppose the best evidence for this might be the major change we undertook a few years ago to move all digital content from a variety of legacy formats to "spinning disk." This was a major project, of course, but allowed ICPSR to stop worrying about the availability of legacy media readers and media which could fail without quick detection.
Saturday, March 26, 2011
February Deposits at ICPSR
February 2011 deposits (and their file formats) at ICPSR:
Nothing too exciting this month.
[ I thought I had posted this weeks ago, but clearly not. ]
| # of files | # of deposits | File format |
| 1 | 1 | application/msoffice |
| 284 | 23 | application/msword |
| 11 | 4 | application/octet-stream |
| 104 | 25 | application/pdf |
| 7 | 5 | application/vnd.ms-excel |
| 24 | 10 | application/x-sas |
| 80 | 28 | application/x-spss |
| 26 | 4 | application/x-stata |
| 2 | 1 | application/x-stuffit |
| 2 | 2 | application/x-zip |
| 19 | 6 | message/rfc8220117bit |
| 15 | 3 | text/html |
| 9 | 2 | text/plain; charset=iso-8859-1 |
| 2 | 2 | text/plain; charset=unknown |
| 122 | 21 | text/plain; charset=us-ascii |
| 9 | 5 | text/rtf |
| 1 | 1 | text/x-c; charset=iso-8859-1 |
| 22 | 7 | text/x-c; charset=us-ascii |
| 1 | 1 | text/xml |
Nothing too exciting this month.
[ I thought I had posted this weeks ago, but clearly not. ]
ICPSR Machine Room maintenance
I don't see an announcement about this on the main web site, so I'll drop a short note here....
The University of Michigan Plant Operations department is performing some electrical work in our machine room starting at 8am this morning. The work will last some unknown number of hours; my guess is 4-6.
We have just diverted traffic from our production web site over to the replica in the cloud. Our most common services such as search, download, and on-line analysis will be available. But services which require our web site to "remember" something -- such as data deposit, Summer Program registration -- are not enabled in the replica.
I'll post a follow-up once the work has completed, and will check back here periodically for comments. If you see something that isn't working as you think it should, please comment here, and we'll take a look.
----
10:22am EDT : We're aware that study search is not working. (Variable search and bibliography search are working.) ICPSR recently updated the search to include the full-text of many types of documents (codebooks, survey instruments, user guides, etc), and the index location changed too. This change was not reflected on the cloud replica, and the index there is now building. Our apologies for the inconvenience this may cause.
ETA for conclusion of the electrical work is between 11am and noon EDT.
-----
11:18am EDT: The power work has been completed. We've restored service to the main site.
The University of Michigan Plant Operations department is performing some electrical work in our machine room starting at 8am this morning. The work will last some unknown number of hours; my guess is 4-6.
We have just diverted traffic from our production web site over to the replica in the cloud. Our most common services such as search, download, and on-line analysis will be available. But services which require our web site to "remember" something -- such as data deposit, Summer Program registration -- are not enabled in the replica.
I'll post a follow-up once the work has completed, and will check back here periodically for comments. If you see something that isn't working as you think it should, please comment here, and we'll take a look.
----
10:22am EDT : We're aware that study search is not working. (Variable search and bibliography search are working.) ICPSR recently updated the search to include the full-text of many types of documents (codebooks, survey instruments, user guides, etc), and the index location changed too. This change was not reflected on the cloud replica, and the index there is now building. Our apologies for the inconvenience this may cause.
ETA for conclusion of the electrical work is between 11am and noon EDT.
-----
11:18am EDT: The power work has been completed. We've restored service to the main site.
Friday, March 11, 2011
TRAC: B3.2: Tracking format obsolescence
B3.2 Repository has mechanisms in place for monitoring and notification when Representation Information (including formats) approaches obsolescence or is no longer viable.
For most repositories, the concern will be with the Representation Information (including formats) used to preserve information, which may include information on how to deal with a file format or software that can be used to render or process it. Sometimes the format needs to change because the repository can no longer deal with it. Sometimes the format is retained and the information about what software is needed to process it needs to change.
In all cases, the repository must show that it has some active mechanism to warn of impending obsolescence. Obsolescence is determined largely in terms of the knowledge base of the designated community(ies). This requirement ensures that the preserved information remains understandable and usable by the designated community(ies). If the mechanism depends on an external registry, the repository must demonstrate how it uses the information from that registry.
Evidence: Subscription to a format registry service; subscription to a technology watch service; percentage of at least one staff member dedicated to monitoring technological obsolescence issues.
ICPSR has two very different stories regarding this requirement.
In terms of content that is has created from deposited materials, the story is very simple. The archival holdings consist of data in plain text format, and related documentation in both PDF and TIFF image format. This content can be used to generate more researcher-friendly formats, such as a SAS Transport file, and it is this latter content that we make available to our community.
For this content obsolescence is easy to track and manage: the pool of content types is very homogeneous.
In terms of content that ICPSR has received from depositors (researchers, federal agencies, news organizations, survey research centers), the content is extremely heterogeneous. Our strategy here is to keep the original content, but only preserve it at the bit-level. We also normalize the content into more durable formats, such as plain text, and those receive what we call "full preservation." For example, if a researcher sends us technical documentation about a dataset in WordPerfect format, we'll keep it, preserve it at the bit-level, and then be prepared to discard it one day if WordPerfect files become unreadable. However, soon after we receive the WordPerfect file, a data manager at ICPSR will transform the content into plain text, and that will be much more durable (although possibly lossy). And as part of our data processing work, this same data manager will produce an ICPSR original documentation file - based heavily on the original WordPerfect, of course - in PDF format (which will also be imaged as TIFF).
Checking the current format types in the repository is a simple database query.
My sense is that the main TODO item for ICPSR is to follow the guidance in the Evidence section of this TRAC requirement, and assign some fraction of two different staff to monitoring and managing older formats. A technology person can create automated jobs to check the repository and broadcast information about risky formats, and can also help write tools and one-time scripts to assist in their migration to newer formats (if needed). A content person can help identify which content is truly useful, and which has important intellectual content that should receive top priority. Also, because this latter person will better understand the meaning of the content, s/he will be poised to know if the migrated content does indeed capture the intellectual properties of the original.
For most repositories, the concern will be with the Representation Information (including formats) used to preserve information, which may include information on how to deal with a file format or software that can be used to render or process it. Sometimes the format needs to change because the repository can no longer deal with it. Sometimes the format is retained and the information about what software is needed to process it needs to change.
In all cases, the repository must show that it has some active mechanism to warn of impending obsolescence. Obsolescence is determined largely in terms of the knowledge base of the designated community(ies). This requirement ensures that the preserved information remains understandable and usable by the designated community(ies). If the mechanism depends on an external registry, the repository must demonstrate how it uses the information from that registry.
Evidence: Subscription to a format registry service; subscription to a technology watch service; percentage of at least one staff member dedicated to monitoring technological obsolescence issues.
ICPSR has two very different stories regarding this requirement.
In terms of content that is has created from deposited materials, the story is very simple. The archival holdings consist of data in plain text format, and related documentation in both PDF and TIFF image format. This content can be used to generate more researcher-friendly formats, such as a SAS Transport file, and it is this latter content that we make available to our community.
For this content obsolescence is easy to track and manage: the pool of content types is very homogeneous.
In terms of content that ICPSR has received from depositors (researchers, federal agencies, news organizations, survey research centers), the content is extremely heterogeneous. Our strategy here is to keep the original content, but only preserve it at the bit-level. We also normalize the content into more durable formats, such as plain text, and those receive what we call "full preservation." For example, if a researcher sends us technical documentation about a dataset in WordPerfect format, we'll keep it, preserve it at the bit-level, and then be prepared to discard it one day if WordPerfect files become unreadable. However, soon after we receive the WordPerfect file, a data manager at ICPSR will transform the content into plain text, and that will be much more durable (although possibly lossy). And as part of our data processing work, this same data manager will produce an ICPSR original documentation file - based heavily on the original WordPerfect, of course - in PDF format (which will also be imaged as TIFF).
Checking the current format types in the repository is a simple database query.
My sense is that the main TODO item for ICPSR is to follow the guidance in the Evidence section of this TRAC requirement, and assign some fraction of two different staff to monitoring and managing older formats. A technology person can create automated jobs to check the repository and broadcast information about risky formats, and can also help write tools and one-time scripts to assist in their migration to newer formats (if needed). A content person can help identify which content is truly useful, and which has important intellectual content that should receive top priority. Also, because this latter person will better understand the meaning of the content, s/he will be poised to know if the migrated content does indeed capture the intellectual properties of the original.
Friday, March 4, 2011
TRAC: B3.1: Documented preservation strategies
B3.1 Repository has documented preservation strategies.
A repository or archiving system must have current, sound, and documented preservation
strategies. These will typically address the degradation of storage media, the obsolescence of media drives, and the obsolescence of Representation Information (including formats), safeguards against accidental or intentional digital corruption. For example, if migration is the chosen approach to some of these issues, there also needs to be policy on what triggers a migration and what types of migration are expected for the solution of each preservation issue identified.
Evidence: Documentation identifying each preservation issue and the strategy for dealing with that issue.
ICPSR documents this thoroughly on the Digital Curation portion of its web site.
A repository or archiving system must have current, sound, and documented preservation
strategies. These will typically address the degradation of storage media, the obsolescence of media drives, and the obsolescence of Representation Information (including formats), safeguards against accidental or intentional digital corruption. For example, if migration is the chosen approach to some of these issues, there also needs to be policy on what triggers a migration and what types of migration are expected for the solution of each preservation issue identified.
Evidence: Documentation identifying each preservation issue and the strategy for dealing with that issue.
ICPSR documents this thoroughly on the Digital Curation portion of its web site.
Friday, February 25, 2011
TRAC: B2.13: Documenting preservation actions
B2.13 Repository has contemporaneous records of actions and administration processes that are relevant to preservation (AIP creation).
These records must be created on or about the time of the actions they refer to and are related to actions associated with AIP creation. The records may be automated or may be written by individuals, depending on the nature of the actions described. Where community or international standards are used, such as PREMIS (2005), the repository must demonstrate that all relevant actions are carried through.
Evidence: Written documentation of decisions and/or action taken; preservation metadata logged, stored, and linked to pertinent digital objects.
ICPSR manages the record of administrative actions in three different ways.
At the micro level, actions performed on individual files, or collections of files, are recorded in an internal workflow system called the Study Tracking System. This records key events and milestones as they occur, and also has a mechanism for free-form annotations (diary entries).
Also at the micro level, ICPSR data managers often produce a consolidate list of commands (syntax files) that document how we have transformed content that we have received. In an idealized case one could take an original survey data file, apply the syntax, and the output would be the file we preserve and distribute.
At a more macro level ICPSR records important events and documents their results in an internal content management system. (We're currently using Drupal for this, but any content management system would be suitable for this purpose.) This system would capture information such as the deployment of a new archival storage location.
These records must be created on or about the time of the actions they refer to and are related to actions associated with AIP creation. The records may be automated or may be written by individuals, depending on the nature of the actions described. Where community or international standards are used, such as PREMIS (2005), the repository must demonstrate that all relevant actions are carried through.
Evidence: Written documentation of decisions and/or action taken; preservation metadata logged, stored, and linked to pertinent digital objects.
ICPSR manages the record of administrative actions in three different ways.
At the micro level, actions performed on individual files, or collections of files, are recorded in an internal workflow system called the Study Tracking System. This records key events and milestones as they occur, and also has a mechanism for free-form annotations (diary entries).
Also at the micro level, ICPSR data managers often produce a consolidate list of commands (syntax files) that document how we have transformed content that we have received. In an idealized case one could take an original survey data file, apply the syntax, and the output would be the file we preserve and distribute.
At a more macro level ICPSR records important events and documents their results in an internal content management system. (We're currently using Drupal for this, but any content management system would be suitable for this purpose.) This system would capture information such as the deployment of a new archival storage location.
Friday, February 18, 2011
TRAC: B2.12: Providing an audit capability
B2.12 Repository provides an independent mechanism for audit of the integrity of the repository collection/content.
In general, it is likely that a repository that meets all the previous criteria will satisfy this one without needing to demonstrate anything more. As a separate requirement, it demonstrates the importance of being able to audit the integrity of the collection as a whole.
For example, if a repository claims to have all e-mail sent or received by The Yoyodyne Corporation between 1985 and 2005, it has been required to show that:
However it may still have no way of showing whether this really represents all of Yoyodyne’s email. For example, if there is a three-day period with no messages in the repository, is this because Yoyodyne was shut down for those three days, or was the e-mail lost before the SIP was constructed? This case could be resolved by the repository amending its description of the collection, but other cases may not be so straightforward.
A familiar mechanism from the world of traditional materials in libraries and archives is an accessions or acquisitions register that is independent of other catalog metadata. A repository should be able to show, for each item in its accessions register, which AIP(s) contain content from that item. Alternatively, it may need to show that there is no AIP for an item, either because ingest is still in progress, or because the item was rejected for some reason. Conversely, any AIP should be able to be related to an entry in the acquisitions register.
Evidence: Documentation provided for B2.1 through B2.6; documented agreements negotiated between the producer and the repository (see B 1.1-B1.9); logs of material received and associated action (receipt, action, etc.) dates; logs of periodic checks.
ICPSR meets this requirement by maintaining as accession register which is a very long (and always growing) list of the files that we preserve. A weekly automated job uses this list as input, and checks to see if each item is still available in archival storage, and also checks to see if the item is intact (i.e., its digital signature has not changed).
In general, it is likely that a repository that meets all the previous criteria will satisfy this one without needing to demonstrate anything more. As a separate requirement, it demonstrates the importance of being able to audit the integrity of the collection as a whole.
For example, if a repository claims to have all e-mail sent or received by The Yoyodyne Corporation between 1985 and 2005, it has been required to show that:
- The content it holds came from Yoyodyne’s e-mail servers.
- It is all correctly transformed into a preservation format.
- Each monthly SIP of e-mail has been correctly preserved, including original unique identifiers such as Message-IDs.
However it may still have no way of showing whether this really represents all of Yoyodyne’s email. For example, if there is a three-day period with no messages in the repository, is this because Yoyodyne was shut down for those three days, or was the e-mail lost before the SIP was constructed? This case could be resolved by the repository amending its description of the collection, but other cases may not be so straightforward.
A familiar mechanism from the world of traditional materials in libraries and archives is an accessions or acquisitions register that is independent of other catalog metadata. A repository should be able to show, for each item in its accessions register, which AIP(s) contain content from that item. Alternatively, it may need to show that there is no AIP for an item, either because ingest is still in progress, or because the item was rejected for some reason. Conversely, any AIP should be able to be related to an entry in the acquisitions register.
Evidence: Documentation provided for B2.1 through B2.6; documented agreements negotiated between the producer and the repository (see B 1.1-B1.9); logs of material received and associated action (receipt, action, etc.) dates; logs of periodic checks.
ICPSR meets this requirement by maintaining as accession register which is a very long (and always growing) list of the files that we preserve. A weekly automated job uses this list as input, and checks to see if each item is still available in archival storage, and also checks to see if the item is intact (i.e., its digital signature has not changed).
Subscribe to:
Posts (Atom)