Showing posts with label archival storage. Show all posts
Showing posts with label archival storage. Show all posts
Monday, September 24, 2012
DuraSpace announces SDSC as storage partner
Friday, July 20, 2012
Amazon's loss is SDSC's gain
Unfortunately, for these particular volumes we have bonded them together to form a virtual RAID. And this RAID is used as a single multi-TB filesystem which is much bigger than fsck can handle. So we are kind of stuck.
One option would be to newfs the big filesystem, and to move the several TBs of content back into AWS, but that would be very slow. And if there is another power outage......
The new copy of our content will now be replicated in...... SDSC's storage cloud. This really brings us full circle at ICPSR since our very first off-site archival copy was stored at SDSC. Back then (like in 2008) it was stored in their Storage Resource Broker (SRB) system, and we used a set of command-line utilities to sync content between ICPSR and SDSC.
Friday, June 22, 2012
AWS power outage aftermath
As it turns out, it doesn't take all that long to run fsck on a large filesystem comprised of multiple AWS Elastic Block Storage (EBS) volumes:
[root@cloudora ~]# df -h /dev/md0
Filesystem Size Used Avail Use% Mounted on
/dev/md0 4.9T 2.9T 1.7T 64% /arcstore
[root@cloudora ~]# fsck -y /dev/md0
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
/dev/md0 is mounted.
WARNING!!! Running e2fsck on a mounted filesystem may cause
SEVERE filesystem damage.
Do you really want to continue (y/n)? yes
/dev/md0 has gone 454 days without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Error allocating icount link information: Memory allocation failed
e2fsck: aborted
I saw a few posts about coaxing e2fsck to use the filesystem for scratch space rather than memory, but unfortunately the older version of the program available on this EC2 instance does not support it.
So I think that we may end up blowing away this copy of archival storage and replacing it with a fresh one.
Friday, May 18, 2012
Data Fetish
David Rosenthal has another good post on digital preservation where he quotes a gent who has coined the term "data fetish" for those that think that data should never be thrown away. Well worth reading, but I'll quote one piece here:
So, we're going to have to throw stuff away. Even if we believe keeping stuff is really cheap, its still too expensive. The bad news is that deciding what to keep and what to throw away isn't free either.
This is so very true.
When my team inherited the responsibility for Archival Storage at ICPSR in 2005-2006, we made a choice to throw away a ton of stuff. Paper hardcopies of electronic files. Magnetic tapes that had been read successfully to spinning disk. Old office supplies. Leases to storage lockers. .... After the smoke cleared we had about 5TB of content moved from a large array of tape types to spinning disk. And this formed the core of ICPSR's Archival Storage.
I think a key question for every archive to answer is "Why are we keeping this?"
If the answer is "We're not sure" (even with some oblique wording), then it is time for that item to be discarded.
Monday, April 16, 2012
The nature of ICPSR's holdings
At the end of 2011 ICPSR had about 9TB of content stored in Archival Storage. This measurement includes everything we have collected over the past 50 years, including content which is not packaged into "studies" for dissemination, such as TIGER/Line files and data packaged for SDA. This content is not compressed, and contains many duplicates[1], and so should be considered an upper bound.
As we head into the start of Q2 in 2012 the quantity of content in Archival Storage has edged up just a little bit; it may be as much as 9.1 TB now. And I would guess that we have another 100GB or so of content in Ingest storage, making its way through the ICPSR data curation process.
The big news, though, is the amount of non-survey content one finds in Ingest storage: 7.4TB. And growing. Fast.
As video content from the Bill and Melinda Gates Foundation Measures of Effective Teaching project continues to arrive it won't be much longer before the amount of video content equals the amount of survey data content. By the end of the calendar year I expect that we will have more video than survey data.
Long-time ICPSR staff tell the story of how the 2000 Census doubled the size of ICPSR's holdings. (I'll speculate that perhaps ICPSR went from about 3TB of content prior to the 200 Census, and then grew to 6TB thereafter.) In 2012-2013 ICPSR is likely to quadruple the size of its holdings, growing from about 9TB to nearly 40TB.
As we head into the start of Q2 in 2012 the quantity of content in Archival Storage has edged up just a little bit; it may be as much as 9.1 TB now. And I would guess that we have another 100GB or so of content in Ingest storage, making its way through the ICPSR data curation process.
The big news, though, is the amount of non-survey content one finds in Ingest storage: 7.4TB. And growing. Fast.
As video content from the Bill and Melinda Gates Foundation Measures of Effective Teaching project continues to arrive it won't be much longer before the amount of video content equals the amount of survey data content. By the end of the calendar year I expect that we will have more video than survey data.
Long-time ICPSR staff tell the story of how the 2000 Census doubled the size of ICPSR's holdings. (I'll speculate that perhaps ICPSR went from about 3TB of content prior to the 200 Census, and then grew to 6TB thereafter.) In 2012-2013 ICPSR is likely to quadruple the size of its holdings, growing from about 9TB to nearly 40TB.
Monday, February 20, 2012
Archival Storage @ ICPSR
I gave this presentation to a group of students who came to ICPSR to hear our Life of a Dataset show.
It contains a couple of pictures from the old ICPSR data warehouse (literally a warehouse) which has been torn down and replaced with a sprawling Costco complex.
And there is also a nice little graphic that shows where we're making copies of things we move into Archival Storage.
It contains a couple of pictures from the old ICPSR data warehouse (literally a warehouse) which has been torn down and replaced with a sprawling Costco complex.
And there is also a nice little graphic that shows where we're making copies of things we move into Archival Storage.
Friday, January 13, 2012
TRAC: A3.9: Self-assessment and certification
A3.9 Repository commits to a regular schedule of self-assessment and certification and, if
certified, commits to notifying certifying bodies of operational changes that will change or
nullify its certification status.
A repository cannot self-certify because an objective, external measurement using a consistent and repeatable certification process is needed to ensure and demonstrate that the repository meets and will likely continue to meet preservation requirements. Therefore, certification is the best indicator that the repository meets its requirements, fulfills its role, and adheres to appropriate standards. The repository must demonstrate that it integrates certification preparation and response into its operations and planning.
Evidence: Completed, dated audit checklists from self-assessment or objective audit; certificates awarded for certification; presence in a certification register (when available); timetable or budget allocation for future certification.
Like a few of the other A-group TRAC requirements, this one really operates at the uppermost level of the organization, and so it is difficult to address it from the IT perspective.
HOWEVA..... One barrier to implementing a regular certification cycle are some fundamental questions:
And so on.
It seems like there is a business opportunity here.
For instance, if ICPSR asserted that it was now in the business of reviewing TRAC requirements for organizations, and a team of ICPSR analysts would either certify your data archive as TRAC compliant or would identify clear action items required to become compliant, would that be a useful thing? Or would other organizations rise up to say, "Hey, who are you, ICPSR, to be granting certifications?"
How should this work?
A repository cannot self-certify because an objective, external measurement using a consistent and repeatable certification process is needed to ensure and demonstrate that the repository meets and will likely continue to meet preservation requirements. Therefore, certification is the best indicator that the repository meets its requirements, fulfills its role, and adheres to appropriate standards. The repository must demonstrate that it integrates certification preparation and response into its operations and planning.
Evidence: Completed, dated audit checklists from self-assessment or objective audit; certificates awarded for certification; presence in a certification register (when available); timetable or budget allocation for future certification.
Like a few of the other A-group TRAC requirements, this one really operates at the uppermost level of the organization, and so it is difficult to address it from the IT perspective.
HOWEVA..... One barrier to implementing a regular certification cycle are some fundamental questions:
Where do I find a list of consultants or analysts that can grant "TRAC certification" to my repository?
Which organization sanctions those consultants and analysts?
What does it mean - precisely - to be "TRAC certified?"
Are there different levels of TRAC certification, much like FISMA levels?
If I'm already FISMA certified, does that automatically grant TRAC certification for certain items (especially in section C)?
And so on.
It seems like there is a business opportunity here.
For instance, if ICPSR asserted that it was now in the business of reviewing TRAC requirements for organizations, and a team of ICPSR analysts would either certify your data archive as TRAC compliant or would identify clear action items required to become compliant, would that be a useful thing? Or would other organizations rise up to say, "Hey, who are you, ICPSR, to be granting certifications?"
How should this work?
Friday, January 6, 2012
TRAC: A3.8: Information integrity measurements
A3.8 Repository commits to defining, collecting, tracking, and providing, on demand, its
information integrity measurements.
The repository must develop or adapt appropriate measures for ensuring the integrity of its holdings. The mechanisms to measure integrity will evolve as technology evolves, but currently include examples such as the use of checksums at ingest and throughout the preservation process. The chain of custody for all of its digital content from the point of deposit forward must be explicit, complete, correct, and current. The repository must demonstrate that the content it has matches the content it received, e.g., with an implemented registry function that documents content from submission onward. Losses associated with migration and other preservation actions should also be documented and made available to relevant stakeholders. (See C1.5 and C1.6.)
If protocols, rules, and mechanisms are embedded in the repository software, there should be some way to demonstrate the implementation of integrity measurements.
Evidence: An implemented registry system; a definition of the repository’s integrity measurements; documentation of the procedures and mechanisms for integrity measurements; an audit system for collecting, tracking, and presenting integrity measurements; procedures for responding to results of integrity measurements that indicate digital content is at risk; policy and workflow documentation.
ICPSR operates very differently than a conventional archive, and it really shows when one looks at this TRAC requirement.
A typical workflow for us looks like this:
The repository must develop or adapt appropriate measures for ensuring the integrity of its holdings. The mechanisms to measure integrity will evolve as technology evolves, but currently include examples such as the use of checksums at ingest and throughout the preservation process. The chain of custody for all of its digital content from the point of deposit forward must be explicit, complete, correct, and current. The repository must demonstrate that the content it has matches the content it received, e.g., with an implemented registry function that documents content from submission onward. Losses associated with migration and other preservation actions should also be documented and made available to relevant stakeholders. (See C1.5 and C1.6.)
If protocols, rules, and mechanisms are embedded in the repository software, there should be some way to demonstrate the implementation of integrity measurements.
Evidence: An implemented registry system; a definition of the repository’s integrity measurements; documentation of the procedures and mechanisms for integrity measurements; an audit system for collecting, tracking, and presenting integrity measurements; procedures for responding to results of integrity measurements that indicate digital content is at risk; policy and workflow documentation.
ICPSR operates very differently than a conventional archive, and it really shows when one looks at this TRAC requirement.
A typical workflow for us looks like this:
- Receive some content in formats like SAS and Word
- Preserve that content "as is" at the bit-level
- Completely re-do all of the data and documentation, preserving the intellectual content (modulo disclosure concerns), but reorganizing it all
- Produce normalized and ready-to-use content based on the re-do
- Preserve the normalized content forever
So at the file-level we track all of the original deposits and all of the content we produce, and we test the integrity of each file every week. Since my team inherited the responsibility to manage archival storage n 2006 I've never seen a problem that wasn't traced back to a transient error that took place as content was being copied into archival storage, and where the solution wasn't solved when the ICPSR staff member re-ran the copy.
We also track the chain of custody at the aggregate level, assigning each "deposit" and "study" to both a workgroup and an individual, and by linking deposits to studies (and vice-versa). We have internal systems to manage both deposits and studies, and they include mechanisms whereby a data manager can edit metadata, assign key dates, and enter diary entries, not unlike a trouble ticket or help desk system.
Friday, December 30, 2011
TRAC: A3.7: Transparency
A3.7 Repository commits to transparency and accountability in all actions supporting the
operation and management of the repository, especially those that affect the preservation
of digital content over time.
Transparency is the best assurance that the repository operates in accordance with accepted standards and practice. Transparency is essential to accountability, and both are achieved through active, ongoing documentation. The repository should be able to document its efforts to make information about its development, implementation, evolution, and performance available and accessible to relevant stakeholders. The usual means of communication an organization uses to provide significant news and updates to stakeholders should suffice for meeting this requirement.
Evidence: Comprehensive documentation that is readily accessible to stakeholders; unhindered access to content and associated information within repository.
This might be an easy requirement for ICPSR to meet. Since we're an "active archive" rather than a "dark archive" (as some would say), we have regular interaction with our designate community. This takes place in the form of downloads from our web site, mailings and blog posts about ICPSR news and events, regular meetings with our Council and our Organizational Representatives, and many other channels.
ICPSR's internal workings are documented in great detail in both paper form and on our Intranet. These documents serve to train new employees, students, and interns during the summer. And since there is always room for improvement, we are working to improve transparency as we develop FLAME, our next-generation, file-level archive management system.
Transparency is the best assurance that the repository operates in accordance with accepted standards and practice. Transparency is essential to accountability, and both are achieved through active, ongoing documentation. The repository should be able to document its efforts to make information about its development, implementation, evolution, and performance available and accessible to relevant stakeholders. The usual means of communication an organization uses to provide significant news and updates to stakeholders should suffice for meeting this requirement.
Evidence: Comprehensive documentation that is readily accessible to stakeholders; unhindered access to content and associated information within repository.
This might be an easy requirement for ICPSR to meet. Since we're an "active archive" rather than a "dark archive" (as some would say), we have regular interaction with our designate community. This takes place in the form of downloads from our web site, mailings and blog posts about ICPSR news and events, regular meetings with our Council and our Organizational Representatives, and many other channels.
ICPSR's internal workings are documented in great detail in both paper form and on our Intranet. These documents serve to train new employees, students, and interns during the summer. And since there is always room for improvement, we are working to improve transparency as we develop FLAME, our next-generation, file-level archive management system.
Friday, December 23, 2011
TRAC: A3.6: Change logs
A3.6 Repository has a documented history of the changes to its operations, procedures,
software, and hardware that, where appropriate, is linked to relevant preservation
strategies and describes potential effects on preserving digital content.
The repository must document the full range of its activities and developments over time, including decisions about the organizational and technological infrastructure. If the repository uses software to document this history, it should be able to demonstrate this tracking.
Evidence: Policies, procedures, and results of changes that affect all levels of the repository: objects, aggregations of objects; object-level preservation metadata; repository’s records retention strategy document.
I'll focus on the technology pieces of this story.
The technology team maintains a change log of major (and not-so-major) system changes. Moving a chunk of storage from Network Attached Storage appliance A to B? It's in the log. Upgrading the hardware of the web server we used to stage new content and software? It's in the log. Updating business productivity software to enforce newly declared business rules that effect how it should work? It's in the log.
We do a pretty good job overall recording technology changes in a well-known, recorded space. (Our Intranet is hosted in the Drupal CMS by the U-M central IT organization.) Of course, there is always room for improvement, but the big stuff gets documented. For instance, I know that we don't always record changes to desktop workstations (e.g., Windows patches) in the change log, even though we do generate an announcement via email.
The repository must document the full range of its activities and developments over time, including decisions about the organizational and technological infrastructure. If the repository uses software to document this history, it should be able to demonstrate this tracking.
Evidence: Policies, procedures, and results of changes that affect all levels of the repository: objects, aggregations of objects; object-level preservation metadata; repository’s records retention strategy document.
I'll focus on the technology pieces of this story.
The technology team maintains a change log of major (and not-so-major) system changes. Moving a chunk of storage from Network Attached Storage appliance A to B? It's in the log. Upgrading the hardware of the web server we used to stage new content and software? It's in the log. Updating business productivity software to enforce newly declared business rules that effect how it should work? It's in the log.
We do a pretty good job overall recording technology changes in a well-known, recorded space. (Our Intranet is hosted in the Drupal CMS by the U-M central IT organization.) Of course, there is always room for improvement, but the big stuff gets documented. For instance, I know that we don't always record changes to desktop workstations (e.g., Windows patches) in the change log, even though we do generate an announcement via email.
Friday, December 16, 2011
TRAC: A3.5: Seeking and acting on feedback
A3.5 Repository has policies and procedures to ensure that feedback from producers and
users is sought and addressed over time.
The repository should be able to demonstrate that it is meeting explicit requirements, that it systematically and routinely seeks feedback from stakeholders to monitor expectations and results, and that it is responsive to the evolution of requirements.
Evidence: A policy that requires a feedback mechanism; a procedure that addresses how the repository seeks, captures, and documents responses to feedback; documentation of workflow for feedback (i.e., how feedback is used and managed); quality assurance records.
I think the market economy that keeps ICPSR in business is the very best evidence that the organization seeks input from its community, and applies that feedback to its operations, content selection, preservation strategies, and nearly every element of its business.
In practice we can see many different types of feedback mechanisms: contract renewals; annual membership renewals; biennial Organizational Representative meetings; regular ICPSR Council meetings; and, regular participation at all sorts of public forms about social science research data, digital preservation, technology, etc. It also happens electronically via social media, a helpdesk where a real person answers the phone and emails, and feedback pages on the main web portal.
In some ways it feels as if this TRAC requirement is aimed at organizations that might be funded by one community, like a national government, but used by a very different community, such as research. In a scenario where the consumers and the payers are different, it is indeed critical that there be some mechanism to collect input, or the repository could enter a kind of "zombie" state where it ceases to serve its community effectively, but the funding organization continues to fund the repository nonetheless.
That said, I do think there is room for improvement in this area for ICPSR. In particular, I think there is a great opportunity to work more closely with the individual data producers, engaging them in the curation process, and making the workflow - from deposit through eventual release - more transparent.
The repository should be able to demonstrate that it is meeting explicit requirements, that it systematically and routinely seeks feedback from stakeholders to monitor expectations and results, and that it is responsive to the evolution of requirements.
Evidence: A policy that requires a feedback mechanism; a procedure that addresses how the repository seeks, captures, and documents responses to feedback; documentation of workflow for feedback (i.e., how feedback is used and managed); quality assurance records.
I think the market economy that keeps ICPSR in business is the very best evidence that the organization seeks input from its community, and applies that feedback to its operations, content selection, preservation strategies, and nearly every element of its business.
In practice we can see many different types of feedback mechanisms: contract renewals; annual membership renewals; biennial Organizational Representative meetings; regular ICPSR Council meetings; and, regular participation at all sorts of public forms about social science research data, digital preservation, technology, etc. It also happens electronically via social media, a helpdesk where a real person answers the phone and emails, and feedback pages on the main web portal.
In some ways it feels as if this TRAC requirement is aimed at organizations that might be funded by one community, like a national government, but used by a very different community, such as research. In a scenario where the consumers and the payers are different, it is indeed critical that there be some mechanism to collect input, or the repository could enter a kind of "zombie" state where it ceases to serve its community effectively, but the funding organization continues to fund the repository nonetheless.
That said, I do think there is room for improvement in this area for ICPSR. In particular, I think there is a great opportunity to work more closely with the individual data producers, engaging them in the curation process, and making the workflow - from deposit through eventual release - more transparent.
Friday, December 9, 2011
TRAC: A3.4: Formal, periodic review
A3.4 Repository is committed to formal, periodic review and assessment to ensure
responsiveness to technological developments and evolving requirements.
Long-term preservation is a shared and complex responsibility. A trusted digital repository contributes to and benefits from the breadth and depth of community-based standards and practice. Regular review is a requisite for ongoing and healthy development of the repository. The organizational context of the repository should determine the frequency of, extent of, and process for self-assessment. The repository must also be able to provide a specific set of requirements it has defined, is maintaining, and is striving to meet. (See also A3.9.)
Evidence: A self-assessment schedule, timetables for review and certification; results of self-assessment; evidence of implementation of review outcomes.
Steve Abrams from the California Digital Library gave an interesting talk earlier this year about the notion of applying a Neighborhood Watch metaphor to digital archives. You can find a PDF of the slideshow here.
This is a nice paradigm, and it fits well with some of the work ICPSR is doing with its Data-PASS partners. We're using the Stanford Lots of Copies Keep Stuff Safe (LOCKSS) software in a Private LOCKSS Network (PLN) to build a distributed archival storage network. And in addition to the PLN, we have also built tools to verify the integrity of the PLN and its content. We call this additional layer the SAFE-Archive, and the development has been led by the Odum Institute at the University of North Carolina.
I also see ICPSR periodically assess itself on a regular basis in response to opportunities to expand its reach thematically or technologically. For example, as ICPSR enters into the world of digital preservation for video as part of two recent grants from the Bill and Melinda Gates Foundation, this drives ICPSR to re-evaluate how it manages content.
I'm not sure that these types of activities are as formal as the TRAC requirement might like, and so the action item might look more like a documentation project rather than adding a new activity into ICPSR's standard operating procedures.
Long-term preservation is a shared and complex responsibility. A trusted digital repository contributes to and benefits from the breadth and depth of community-based standards and practice. Regular review is a requisite for ongoing and healthy development of the repository. The organizational context of the repository should determine the frequency of, extent of, and process for self-assessment. The repository must also be able to provide a specific set of requirements it has defined, is maintaining, and is striving to meet. (See also A3.9.)
Evidence: A self-assessment schedule, timetables for review and certification; results of self-assessment; evidence of implementation of review outcomes.
This is a nice paradigm, and it fits well with some of the work ICPSR is doing with its Data-PASS partners. We're using the Stanford Lots of Copies Keep Stuff Safe (LOCKSS) software in a Private LOCKSS Network (PLN) to build a distributed archival storage network. And in addition to the PLN, we have also built tools to verify the integrity of the PLN and its content. We call this additional layer the SAFE-Archive, and the development has been led by the Odum Institute at the University of North Carolina.
I also see ICPSR periodically assess itself on a regular basis in response to opportunities to expand its reach thematically or technologically. For example, as ICPSR enters into the world of digital preservation for video as part of two recent grants from the Bill and Melinda Gates Foundation, this drives ICPSR to re-evaluate how it manages content.
I'm not sure that these types of activities are as formal as the TRAC requirement might like, and so the action item might look more like a documentation project rather than adding a new activity into ICPSR's standard operating procedures.
Friday, December 2, 2011
TRAC: A3.3: Permission to preserve
A3.3 Repository maintains written policies that specify the nature of any legal permissions
required to preserve digital content over time, and repository can demonstrate that these
permissions have been acquired when needed.
Because the right to change or alter digital information is often restricted by law to the creator, it is important that digital repositories address the need to be able to work with and potentially modify digital objects to keep them accessible over time. Repositories should have written policies and agreements with depositors that specify and/or transfer certain rights to the repository enabling appropriate and necessary preservation actions to take place on the digital objects within the repository.
Because legal negotiations can take time, potentially slowing or preventing the ingest of digital objects at risk, a digital repository may take in or accept digital objects even with only minimal preservation rights using an open-ended agreement and address more detailed rights later. A repository’s rights must at least limit the repository’s liability or legal exposure that threatens the repository itself. A repository does not have sufficient control of the information if the repository itself is legally at risk.
Evidence: Deposit agreements; records schedule; digital preservation policies; records legislation and policies; service agreements.
ICPSR has a standard agreement that is uses for all deposits. This agreement grants ICPSR the non-exclusive right to replicate the content for preservation purposes and to deliver the content on our web site. This language resides inside of our Deposit Form web application.
This works very well for deposits that come from a known source, such as a government agency with whom we have an agreement to preserve and deliver content, or an individual researcher with whom we have been corresponding. In this case we have a good sense for who the depositor is, the role they play with regard to the data, and the mechanisms by which we can contact him/her.
Things become a bit messier what I will call a "drive-by deposit." This is an unsolicited, unexpected deposit, and in this case the depositor agrees to give us permission to make copies of the content for digital preservation purposes and to deliver the content via our web portal. That said, ICPSR does not require strong identities to execute a deposit, and so one could ask the question: How does ICPSR know that the depositor himself/herself has the authority to grant us rights to preserve and redistribute the content?
Because the right to change or alter digital information is often restricted by law to the creator, it is important that digital repositories address the need to be able to work with and potentially modify digital objects to keep them accessible over time. Repositories should have written policies and agreements with depositors that specify and/or transfer certain rights to the repository enabling appropriate and necessary preservation actions to take place on the digital objects within the repository.
Because legal negotiations can take time, potentially slowing or preventing the ingest of digital objects at risk, a digital repository may take in or accept digital objects even with only minimal preservation rights using an open-ended agreement and address more detailed rights later. A repository’s rights must at least limit the repository’s liability or legal exposure that threatens the repository itself. A repository does not have sufficient control of the information if the repository itself is legally at risk.
Evidence: Deposit agreements; records schedule; digital preservation policies; records legislation and policies; service agreements.
ICPSR has a standard agreement that is uses for all deposits. This agreement grants ICPSR the non-exclusive right to replicate the content for preservation purposes and to deliver the content on our web site. This language resides inside of our Deposit Form web application.
This works very well for deposits that come from a known source, such as a government agency with whom we have an agreement to preserve and deliver content, or an individual researcher with whom we have been corresponding. In this case we have a good sense for who the depositor is, the role they play with regard to the data, and the mechanisms by which we can contact him/her.
Things become a bit messier what I will call a "drive-by deposit." This is an unsolicited, unexpected deposit, and in this case the depositor agrees to give us permission to make copies of the content for digital preservation purposes and to deliver the content via our web portal. That said, ICPSR does not require strong identities to execute a deposit, and so one could ask the question: How does ICPSR know that the depositor himself/herself has the authority to grant us rights to preserve and redistribute the content?
Friday, November 25, 2011
TRAC: A3.2: Written policies and procedures
A3.2 Repository has procedures and policies in place, and mechanisms for their review,
update, and development as the repository grows and as technology and community
practice evolve.
The policies and procedures of the repository must be complete, written or available in a tangible form, remain current, and must evolve to reflect changes in requirements and practice. The repository must demonstrate that a policy and procedure audit and maintenance is in place and regularly applied. Policies and procedures should address core areas, including, for example, transfer requirements, submission, quality control, storage management, disaster planning, metadata management, access, rights management, preservation strategies, staffing, and security. High-level documents should make organizational commitments and intents clear. Lower-level documents should make day-to-day practice and procedure clear. Versions of these documents must be well managed by the repository (e.g., outdated versions are clearly identified or maintained offline) and qualified staff and peers must be involved in reviewing, updating, and extending these documents. The repository should be able to document the results of monitoring for relevant developments; responsiveness to prevailing standards and practice, emerging requirements, and standards that are specific to the domain, if appropriate; and similar developments. The repository should be able to demonstrate that it has defined "comprehensive documentation" for the repository. See Appendix 3: Minimum Required Documents for more information.
Evidence: Written documentation in the form of policies, procedures, protocols, rules, manuals, handbooks, and workflows; specification of review cycle for documentation; documentation detailing review, update, and development mechanisms. If documentation is embedded in system logic, functionality should demonstrate the implementation of policies and procedures.
A deliverable for a recent contract was something the client called an information system security plan. Our understanding was that in past contracts this was always understood to be a short document (2-3 pages) that summarized ICPSR's IT systems, and described the measures taken by ICPSR to protect them from unauthorized use. No big deal, right?
However, .....
In this most recent contract the security plan implementation details changed; rather than a brief summary document, the requirement was now two-fold.
The first deliverable consisted of a document showing the Federal Information Processing Standards (FIPS) categorization of the risks associated with our IT systems. This document was based on a standard known as FIPS Publication 199. It turns out that this methodology and level of documentation is relatively lightweight.
In brief, one asserts one of three levels (Low, Medium, High) of risk. There was never a question of asserting High risk, and so the choice as to select either Low or Medium. We worked with the University of Michigan's central IT security office, and based on the type of data preserved at ICPSR, they recommended that we select Low.
The second part required us to document the security controls defined by the National Institute of Standards and Technology related to a FIPS-199 categorization of Low risk. This standard is described in NIST Special Publication 800-53, and requires a very high level of documentation. (The standard is very heavy on policy and documentation, but very light on measurement and audit, and therefore some critics believe that this is a major flaw in the approach.)
Our NIST 800-53 security control documentation ran nearly 200 pages(!), and this page-count does not include documents which are required, but external, to 800-53. For example, if 800-53 requires one to assert that there is a policy on topic X, one does not need to include the policy within the 800-53 security controls documentation, but it does require one to write the policy on topic X (if it does not already exist). And so between the 800-53 controls and the external documents, our guess is that this ran well over 250 pages.
And so we are very well supplied with policies and procedures, and we even have the documentation to prove it now.
The policies and procedures of the repository must be complete, written or available in a tangible form, remain current, and must evolve to reflect changes in requirements and practice. The repository must demonstrate that a policy and procedure audit and maintenance is in place and regularly applied. Policies and procedures should address core areas, including, for example, transfer requirements, submission, quality control, storage management, disaster planning, metadata management, access, rights management, preservation strategies, staffing, and security. High-level documents should make organizational commitments and intents clear. Lower-level documents should make day-to-day practice and procedure clear. Versions of these documents must be well managed by the repository (e.g., outdated versions are clearly identified or maintained offline) and qualified staff and peers must be involved in reviewing, updating, and extending these documents. The repository should be able to document the results of monitoring for relevant developments; responsiveness to prevailing standards and practice, emerging requirements, and standards that are specific to the domain, if appropriate; and similar developments. The repository should be able to demonstrate that it has defined "comprehensive documentation" for the repository. See Appendix 3: Minimum Required Documents for more information.
Evidence: Written documentation in the form of policies, procedures, protocols, rules, manuals, handbooks, and workflows; specification of review cycle for documentation; documentation detailing review, update, and development mechanisms. If documentation is embedded in system logic, functionality should demonstrate the implementation of policies and procedures.
A deliverable for a recent contract was something the client called an information system security plan. Our understanding was that in past contracts this was always understood to be a short document (2-3 pages) that summarized ICPSR's IT systems, and described the measures taken by ICPSR to protect them from unauthorized use. No big deal, right?
However, .....
In this most recent contract the security plan implementation details changed; rather than a brief summary document, the requirement was now two-fold.
The first deliverable consisted of a document showing the Federal Information Processing Standards (FIPS) categorization of the risks associated with our IT systems. This document was based on a standard known as FIPS Publication 199. It turns out that this methodology and level of documentation is relatively lightweight.
In brief, one asserts one of three levels (Low, Medium, High) of risk. There was never a question of asserting High risk, and so the choice as to select either Low or Medium. We worked with the University of Michigan's central IT security office, and based on the type of data preserved at ICPSR, they recommended that we select Low.
The second part required us to document the security controls defined by the National Institute of Standards and Technology related to a FIPS-199 categorization of Low risk. This standard is described in NIST Special Publication 800-53, and requires a very high level of documentation. (The standard is very heavy on policy and documentation, but very light on measurement and audit, and therefore some critics believe that this is a major flaw in the approach.)
Our NIST 800-53 security control documentation ran nearly 200 pages(!), and this page-count does not include documents which are required, but external, to 800-53. For example, if 800-53 requires one to assert that there is a policy on topic X, one does not need to include the policy within the 800-53 security controls documentation, but it does require one to write the policy on topic X (if it does not already exist). And so between the 800-53 controls and the external documents, our guess is that this ran well over 250 pages.
And so we are very well supplied with policies and procedures, and we even have the documentation to prove it now.
Friday, November 18, 2011
TRAC: A3.1: Designated community
A3.1 Repository has defined its designated community(ies) and associated knowledge
base(s) and has publicly accessible definitions and policies in place to dictate how its
preservation service requirements will be met.
The definition of the designated community(ies) (producer and user community) is arrived at through the planning processes used to create the repository and define its services. The definition will be drawn from various sources ranging from market research to service-level agreements for producers to the mission or scope of the institution within which the repository is embedded.
Meeting the needs of the designated community—the expected understandability of the information, not just access to it—will affect the digital object management, as well as the technical infrastructure of the overall repository. For appropriate long-term planning, the repository or organization must understand and institute policies to support these needs.
For a given submission of information, the repository must make clear the operational definition of understandability that is associated with the corresponding designated community(ies). The designated community(ies) may vary from one submission to another, as may the definition of understandability that establishes the repository’s responsibility in this area. This may range from no responsibility, if bits are only to be preserved, to the maintenance of a particular level of use, if understanding by the members of the designated community(ies) is determined outside the repository, to a responsibility for ensuring a given level of designated community(ies) human understanding, requiring appropriate Representation Information.
The documentation of understandability will typically include a definition of the applications the designated community(ies) will use with the information, possibly after transformation by repository services. For example, if a designated community is defined as readers of English with access to widely available document rendering tools, and if this definition is clearly associated with a given set of Content Information and Preservation Description Information, then the requirement is met.
Examples of designated community definitions include:
Documentation for this TRAC requirement can be found in ICPSR's mission statement (published on our web portal) and in our deposit agreements.
The definition of the designated community(ies) (producer and user community) is arrived at through the planning processes used to create the repository and define its services. The definition will be drawn from various sources ranging from market research to service-level agreements for producers to the mission or scope of the institution within which the repository is embedded.
Meeting the needs of the designated community—the expected understandability of the information, not just access to it—will affect the digital object management, as well as the technical infrastructure of the overall repository. For appropriate long-term planning, the repository or organization must understand and institute policies to support these needs.
For a given submission of information, the repository must make clear the operational definition of understandability that is associated with the corresponding designated community(ies). The designated community(ies) may vary from one submission to another, as may the definition of understandability that establishes the repository’s responsibility in this area. This may range from no responsibility, if bits are only to be preserved, to the maintenance of a particular level of use, if understanding by the members of the designated community(ies) is determined outside the repository, to a responsibility for ensuring a given level of designated community(ies) human understanding, requiring appropriate Representation Information.
The documentation of understandability will typically include a definition of the applications the designated community(ies) will use with the information, possibly after transformation by repository services. For example, if a designated community is defined as readers of English with access to widely available document rendering tools, and if this definition is clearly associated with a given set of Content Information and Preservation Description Information, then the requirement is met.
Examples of designated community definitions include:
- General English-reading public educated to high school and above, with access to a Web Browser (HTML 4.0 capable).
- For GIS data: GIS researchers—undergraduates and above—having an understanding of the concepts of Geographic data and having access to current (2005, USA) GIS tools/computer software, e.g., ArcInfo (2005).
- Astronomer (undergraduate and above) with access to FITS software such as FITSIO, familiar with astronomical spectrographic instruments.
- Student of Middle English with an understanding of TEI encoding and access to an XML rendering environment.
- Variant 1: Cannot understand TEI
- Variant 2: Cannot understand TEI and no access to XML rendering environment
- Variant 3: No understanding of Middle English but does understand TEI and XML
- Two groups: the publishers of scholarly journals and their readers, each of whom have different rights to access material and different services offered to them.
Documentation for this TRAC requirement can be found in ICPSR's mission statement (published on our web portal) and in our deposit agreements.
Wednesday, November 16, 2011
DuraCloud Archiving and Preservation Webinar
Shameless self-promotion alert...
The nice folks at DuraSpace have published the audio and video from the recent webinar that Michele Kimpton (CEO, DuraSpace) and I gave on DuraCloud.
Michele spends the first 5-10 minutes talking about the business case behind DuraCloud, and then I spend about 30 minutes talking about ICPSR and how we came to use DuraCloud to store a copy of our archival holdings.
The nice folks at DuraSpace have published the audio and video from the recent webinar that Michele Kimpton (CEO, DuraSpace) and I gave on DuraCloud.
Michele spends the first 5-10 minutes talking about the business case behind DuraCloud, and then I spend about 30 minutes talking about ICPSR and how we came to use DuraCloud to store a copy of our archival holdings.
Friday, November 11, 2011
TRAC: A2.3: Keeping ahead of the curve
A2.3 Repository has an active professional development program in place that provides
staff with skills and expertise development opportunities.
Technology will continue to change, so the repository must ensure that its staff’s skill sets evolve, ideally through a lifelong learning approach to developing and retaining staff. As the requirements and expectations pertaining to each functional area evolve, the repository must demonstrate that staff are prepared to face new challenges.
Evidence: Professional development plans and reports; training requirements and training budgets, documentation of training expenditures (amount per staff); performance goals and documentation of staff assignments and achievements, copies of certificates awarded.
ICPSR does a very good job organization-wide at encouraging staff to develop professionally. This manifests itself in several different ways: attending workshops and seminars; taking courses to learn new skills and abilities; and, most impressively, funding continuing education for staff who want to pursue a degree (often a graduate degree). In the technology shop we take advantage of the generous professional development budget in many ways.
The operations team (managed by Asmat Noori) is always bringing new technology into ICPSR: new storage systems, new backup systems, new versions of software and operating systems, and more. And so there is a recurring need for people to attend training workshops so that they can manage new technologies effectively and efficiently. Further, some types of training - such as SANS security training - cuts across many different technologies, and needs to be renewed every few years, and the training budget also supports this type of activity.
The software development team (managed by Nathan Adams) also makes regular use of professional development. In this case the new technology is usually a software system or development system component, not a new type of hardware. And we will sometimes bring a trainer on-site to deliver a course to our entire team rather than sending people away to a class. Nathan also has one staff member taking advantage of the tuition package offered by ICPSR, and is enrolled in the University of Michigan's School of Information Master's degree program.
Technology will continue to change, so the repository must ensure that its staff’s skill sets evolve, ideally through a lifelong learning approach to developing and retaining staff. As the requirements and expectations pertaining to each functional area evolve, the repository must demonstrate that staff are prepared to face new challenges.
Evidence: Professional development plans and reports; training requirements and training budgets, documentation of training expenditures (amount per staff); performance goals and documentation of staff assignments and achievements, copies of certificates awarded.
ICPSR does a very good job organization-wide at encouraging staff to develop professionally. This manifests itself in several different ways: attending workshops and seminars; taking courses to learn new skills and abilities; and, most impressively, funding continuing education for staff who want to pursue a degree (often a graduate degree). In the technology shop we take advantage of the generous professional development budget in many ways.
The operations team (managed by Asmat Noori) is always bringing new technology into ICPSR: new storage systems, new backup systems, new versions of software and operating systems, and more. And so there is a recurring need for people to attend training workshops so that they can manage new technologies effectively and efficiently. Further, some types of training - such as SANS security training - cuts across many different technologies, and needs to be renewed every few years, and the training budget also supports this type of activity.
The software development team (managed by Nathan Adams) also makes regular use of professional development. In this case the new technology is usually a software system or development system component, not a new type of hardware. And we will sometimes bring a trainer on-site to deliver a course to our entire team rather than sending people away to a class. Nathan also has one staff member taking advantage of the tuition package offered by ICPSR, and is enrolled in the University of Michigan's School of Information Master's degree program.
Friday, November 4, 2011
TRAC: A2.2: The right quantity of staff and skills
A2.2 Repository has the appropriate number of staff to support all functions and services.
Staffing for the repository must be adequate for the scope and mission of the archiving program. The repository should be able to demonstrate an effort to determine the appropriate number and level of staff that corresponds to requirements and commitments. (These requirements are related to the core functionality covered by a certification process. Of particular interest to repository certification is whether the organization has appropriate staff to support activities related to the long-term preservation of the data.) The accumulated commitments of the repository can be identified in deposit agreements, service contracts, licenses, mission statements, work plans, priorities, goals, and objectives. Understaffing or a mismatch between commitments and staffing indicates that the repository cannot fulfill its agreements and requirements.
Evidence: Organizational charts; definitions of roles and responsibilities; comparison of staffing levels to commitments and estimates of required effort.
This is an interesting question for an organization like ICPSR. My colleague Nancy McGovern mentioned something the other day: She noted the difference between digital preservation (where ICPSR spends some of its resources, but not the lion's share) and data curation (where we spend a significant quantity of resources).
My sense is that the effort required to perform a base level of digital preservation on our content - plain text survey data and PDF- and XML-format documentation - is relatively small, and even if ICPSR found itself operating on a minimal budget without any of its topical archives or special projects, there would be an adequate number of staff to manage the archival holdings, review fixity reports, and execute migrations of content from format to format, or from location to location.
In our present configuration, we can see a close correlation between required effort and resources. This most often manifests itself as line-items in individual project budgets. But it also shows up on the organizational chart when one sees specific organizational units at ICPSR which have a clear digital preservation mission or component.
Staffing for the repository must be adequate for the scope and mission of the archiving program. The repository should be able to demonstrate an effort to determine the appropriate number and level of staff that corresponds to requirements and commitments. (These requirements are related to the core functionality covered by a certification process. Of particular interest to repository certification is whether the organization has appropriate staff to support activities related to the long-term preservation of the data.) The accumulated commitments of the repository can be identified in deposit agreements, service contracts, licenses, mission statements, work plans, priorities, goals, and objectives. Understaffing or a mismatch between commitments and staffing indicates that the repository cannot fulfill its agreements and requirements.
Evidence: Organizational charts; definitions of roles and responsibilities; comparison of staffing levels to commitments and estimates of required effort.
This is an interesting question for an organization like ICPSR. My colleague Nancy McGovern mentioned something the other day: She noted the difference between digital preservation (where ICPSR spends some of its resources, but not the lion's share) and data curation (where we spend a significant quantity of resources).
My sense is that the effort required to perform a base level of digital preservation on our content - plain text survey data and PDF- and XML-format documentation - is relatively small, and even if ICPSR found itself operating on a minimal budget without any of its topical archives or special projects, there would be an adequate number of staff to manage the archival holdings, review fixity reports, and execute migrations of content from format to format, or from location to location.
In our present configuration, we can see a close correlation between required effort and resources. This most often manifests itself as line-items in individual project budgets. But it also shows up on the organizational chart when one sees specific organizational units at ICPSR which have a clear digital preservation mission or component.
Friday, October 28, 2011
TRAC: A2.1: The right type of staff and skills
A2.1 Repository has identified and established the duties that it needs to perform and has
appointed staff with adequate skills and experience to fulfill these duties.
The repository must identify the competencies and skill sets required to operate the repository over time and demonstrate that the staff and consultants have the range of requisite skills—e.g., archival training, technical skills, and legal expertise.
Evidence: A staffing plan; competency definitions; job description; development plans; plus evidence that the repository review and maintains these documents as requirements evolve.
I see three main areas of evidence to support this requirement.
One is that ICPSR has been in operation for fifty years, and it continues to win contracts and grants to preserve and disseminate social science research data and documentation. No organization can operate for fifty years if it does not have a team capable of delivering success.
Another bit of evidence appears on ICPSR's organization chart. I don't believe we publish it for the world to see, but we do maintain a copy on our intranet site. The org chart shows the areas, teams, and people one needs to curate content successfully. Data managers? Check. Metadata specialists? Check. Technology? Check. Administrative functions? Check. And, of course, specialists in digital preservation policy and standards.
Finally, there is also evidence in the body of job descriptions one would find on the University of Michigan jobs site (if its content was preserved!). One can see how job titles, job functions, and skillsets have evolved over the years as technology, best practices, and types of content have also evolved.
The repository must identify the competencies and skill sets required to operate the repository over time and demonstrate that the staff and consultants have the range of requisite skills—e.g., archival training, technical skills, and legal expertise.
Evidence: A staffing plan; competency definitions; job description; development plans; plus evidence that the repository review and maintains these documents as requirements evolve.
I see three main areas of evidence to support this requirement.
One is that ICPSR has been in operation for fifty years, and it continues to win contracts and grants to preserve and disseminate social science research data and documentation. No organization can operate for fifty years if it does not have a team capable of delivering success.
Another bit of evidence appears on ICPSR's organization chart. I don't believe we publish it for the world to see, but we do maintain a copy on our intranet site. The org chart shows the areas, teams, and people one needs to curate content successfully. Data managers? Check. Metadata specialists? Check. Technology? Check. Administrative functions? Check. And, of course, specialists in digital preservation policy and standards.
Finally, there is also evidence in the body of job descriptions one would find on the University of Michigan jobs site (if its content was preserved!). One can see how job titles, job functions, and skillsets have evolved over the years as technology, best practices, and types of content have also evolved.
Wednesday, October 26, 2011
Using DuraCloud for Archiving and Preservation
I think sometimes people will view the cloud as an alternative to keeping and maintaining local copies, but at ICPSR we're using the cloud as an easy-to-manage storage location to supplement more conventional locations, such as local NAS storage and the University of Michigan's "Value Storage" service.
Here is a copy of the invite that went out via email:
| |
|
Subscribe to:
Posts (Atom)