B1.4 Repository’s ingest process verifies each submitted object (i.e., SIP) for completeness and correctness as specified in B1.2.
Information collected during the ingest process must be compared with information from some other source—the producer or the repository’s own expectations—to verify the correctness of the data transfer and ingest process. The extent to which a repository can determine correctness will depend on what it knows about the SIP and what tools are available for verifying correctness. It can mean simply checking that file formats are what they claim to be (TIFF files are valid TIFF format, for instance), or can imply checking the content. This might involve human checking in some cases, such as confirming that the description of a picture matches the image.
Repositories should have established procedures for handling incomplete SIPs. These can range from rejecting the transfer, to suspending processing until the missing information is received, to simply reporting the errors. Similarly, the definition of “completeness” should be appropriate to a repository’s activities. If an inventory of files was provided by a producer as part of pre-ingest negotiations, one would expect checks to be carried out against that inventory. But for some activities such as Web harvesting, “complete” may simply mean “whatever we could capture in the harvest session.” Whatever checks are carried out must be consistent with the repository’s own documented definition and understanding of completeness and correctness.
Evidence: Appropriate policy documents and system log files from system performing ingest procedure; formal or informal acquisitions register of files received during the transfer and ingest process; workflow, documentation of standard operating procedures, detailed procedures; definition of completeness and correctness, probably incorporated in policy documents.
For this post I am going to focus on the files that a depositor uploads to our web site. The other elements of the deposit, which are largely metadata, are collected automatically by the deposit system. It's only a title or name for the deposit that the depositor must provide.
As mentioned in an early post on deposits, we collect several pieces of information about each file. One item we collect is the MIME type, and we do this using the UNIX file utility, but where we have expanded the magic database to include information about common statistical packages and Microsoft Office file formats. For example, a vanilla, out-of-the-box version of file will report a DOCX format file as a Zip file, and while that is correct on some level, it wasn't the best description of the file.
After making this inventory, our system generates an auto-reply to the depositor enumerating the files and what we think they are. Note that we do not assign any higher level purpose (e.g., this is a data file; this is a codebook) programatically.
Assuming that the depositor does not find anything amiss with our inventory, a data manager will pick up the deposit, and will start preparing the materials for preservation at ICPSR, and for distribution on our web site.
A couple of things we are NOT doing, but which might be valuable in the future....
One, in addition to using file to report MIME types, we might also run a tool like JHOVE to inspect the correctness of the file formats. We do get a fair number of PDF format documents, and JHOVE does a nice job with those. However, we also get a lot of plain text documents, stats files, and MS Office files, and JHOVE doesn't do a very nice job with those. I've wondered if we might be able to write a small grant where ICPSR would promise to build plug-ins for JHOVE for the stats package formats.
Two, in addition to reporting file formats to our depositors, we might also report checksums for each file so that they would have the opportunity to inspect that the deposit went without error.
Showing posts with label certification. Show all posts
Showing posts with label certification. Show all posts
Friday, September 10, 2010
Friday, September 3, 2010
TRAC: B1.3: Authenticating the source
B1.3 Repository has mechanisms to authenticate the source of all materials.The repository’s written standard operating procedures and actual practices must ensure the digital objects are obtained from the expected source, that the appropriate provenance has been maintained, and that the objects are the expected objects. Confirmation can use various means including, but not limited to, digital processing and data verification and validation, and through exchange of appropriate instrument of ownership (e.g., submission agreements/deposit agreement/deed of gift).
Evidence: Submission agreements/deposit agreements/deeds of gift; workflow documents; evidence of appropriate technological measures; logs from procedures and authentications.
ICPSR's Deposit Form is the mechanism that ensures proper documentation and workflow for deposited materials. It ensures that we have a signature from the data depositor, and keeps track of what was deposited, who did it, and when it was done.
One feature of the Deposit Form is that it auto-emails the depositor with a list of the files it thinks have been deposited, and each file includes a one-line description regarding the file format (e.g., SPSS Portable File, PDF document, version 1.6, or Microsoft Word). If the deposited file is a Zip package or a tarball, we pull apart the bundle first, and then examine the individual files before generating this report.
The person who signs the deposit must have a valid MyData account, and we use this to track and record identity. Of course, this same information is also helpful during the data management process if ICPSR needs to contact the data producer or researcher.
Friday, August 27, 2010
TRAC: B1.2: Defining deposit requirements
B1.2 Repository clearly specifies the information that needs to be associated with digital material at the time of its deposit (i.e., SIP).
For most types of digital objects to be ingested, the repository should have written criteria, prepared by the repository on its own or in conjunction with other parties, that specify exactly what digital object(s) are transferred, what documentation is associated with the object(s), and any restrictions on access, whether technical, regulatory, or donor-imposed.
The level of precision in these specifications will vary with the nature of the repository’s collection policy and its relationship with creators. For instance, repositories engaged in Web harvesting, or those that rescue digital materials long after their creators have abandoned them, cannot impose conditions on the creators of material, since they are not “depositors” in the usual sense of the word. But Web harvesters can, for instance, decide which metadata elements from the HTTP transactions that captured a site are to be preserved along with the site’s files, and this still constitutes “information associated with the digital material.” They may also choose to record the information or decisions—whether taken by humans or by automated algorithms—that led to the site being captured.
Evidence: Transfer requirements; producer-archive agreements.
ICPSR is pretty flexible with regard to its deposit requirements. The business rules behind our Deposit Form web app allow one to submit content for deposit at ICPSR with little more than a working title for the deposit and a signature. Of course, the Deposit Form also has many, many fields for collecting additional descriptive metadata, but those fields are not required (although we do appreciate all of the descriptive metadata we can get).
On the back-end of the deposit our business rules require us to create, collect, and store the following information about each file in the deposit:
For most types of digital objects to be ingested, the repository should have written criteria, prepared by the repository on its own or in conjunction with other parties, that specify exactly what digital object(s) are transferred, what documentation is associated with the object(s), and any restrictions on access, whether technical, regulatory, or donor-imposed.
The level of precision in these specifications will vary with the nature of the repository’s collection policy and its relationship with creators. For instance, repositories engaged in Web harvesting, or those that rescue digital materials long after their creators have abandoned them, cannot impose conditions on the creators of material, since they are not “depositors” in the usual sense of the word. But Web harvesters can, for instance, decide which metadata elements from the HTTP transactions that captured a site are to be preserved along with the site’s files, and this still constitutes “information associated with the digital material.” They may also choose to record the information or decisions—whether taken by humans or by automated algorithms—that led to the site being captured.
Evidence: Transfer requirements; producer-archive agreements.
ICPSR is pretty flexible with regard to its deposit requirements. The business rules behind our Deposit Form web app allow one to submit content for deposit at ICPSR with little more than a working title for the deposit and a signature. Of course, the Deposit Form also has many, many fields for collecting additional descriptive metadata, but those fields are not required (although we do appreciate all of the descriptive metadata we can get).
On the back-end of the deposit our business rules require us to create, collect, and store the following information about each file in the deposit:
- Original name from the deposit
- MIME type
- Unique ID (that we create and assign)
- Fingerprint and the method used to collect the fingerprint (currently MD5 hashes)
- Date of deposit
- Location
- Identity of the depositor
Thursday, March 11, 2010
TRAC: B1.1: What we preserve
B1.1 Repository identifies properties it will preserve for digital objects.
This process begins in general with the repository’s mission statement and may be further specified in pre-accessioning agreements with producers or depositors (e.g., producer-archive agreements) and made very specific in deposit or transfer agreements for specific digital objects and their related documentation.
For example, one repository may only commit to preserving the textual content of a document and not its exact appearance on a screen. Another may wish to preserve the exact appearance and layout of textual documents, while others may choose to normalize the data during the ingest process.
Evidence: Mission statement; submission agreements/deposit agreements/deeds of gift; workflow and policy documents, including written definition of properties as agreed in the deposit agreement/deed of gift; written processing procedures; documentation of properties to be preserved.
My sense is that this TRAC requirement is more about policy than technology, but nonetheless I'll comment on it from a status quo, technological standpoint for the purposes of this post. That will give the reader a good sense for how things are working today, but leaves open the possibility (likelihood?) that ICPSR may change its mission statement, submission agreements, and/or other possibilities down the road, and then the underlying technology will change too.
The short version of the story is that ICPSR captures two essential items from each submission: the data (normalized into plain ASCII characters), and the technical documentation (normalized into DDI XML, PDF, and TIF images). And we keep the content in these formats until we need to migrate to a different format.
In our world of survey research data there is no requirement to preserve original formats (such as a SAS Portable file), look and feel (such as original technical documentation written in Word Perfect), or even low-level data format (such as numeric data coded in punched card format). We can take everything, pull out the essential intellectual content, and then normalize that content in quite durable formats.
That said, we may take these durable formats, such as DDI XML metadata and plain ASCII data, and feed them into tools that produce dissemination formats that are easier for our designated community to use. This includes common stat package formats, and also includes formats that feed into our on-line analysis system. However, these dissemination formats are not preserved, and they contain no essential properties not also contained in the preservation formats.
As I've been thinking about this particular TRAC requirement over the past week, I think we're probably missing one important piece of information for each of our data files: the character set. In practice almost all of our holdings are numeric data, and so our implicit character set ("us-ascii") is the correct one to use, but this seems like a good area for being more explicit. This would allow us to handle data files that contain non-numeric, non-US-ASCII characters in a more durable fashion.
This process begins in general with the repository’s mission statement and may be further specified in pre-accessioning agreements with producers or depositors (e.g., producer-archive agreements) and made very specific in deposit or transfer agreements for specific digital objects and their related documentation.
For example, one repository may only commit to preserving the textual content of a document and not its exact appearance on a screen. Another may wish to preserve the exact appearance and layout of textual documents, while others may choose to normalize the data during the ingest process.
Evidence: Mission statement; submission agreements/deposit agreements/deeds of gift; workflow and policy documents, including written definition of properties as agreed in the deposit agreement/deed of gift; written processing procedures; documentation of properties to be preserved.
My sense is that this TRAC requirement is more about policy than technology, but nonetheless I'll comment on it from a status quo, technological standpoint for the purposes of this post. That will give the reader a good sense for how things are working today, but leaves open the possibility (likelihood?) that ICPSR may change its mission statement, submission agreements, and/or other possibilities down the road, and then the underlying technology will change too.
The short version of the story is that ICPSR captures two essential items from each submission: the data (normalized into plain ASCII characters), and the technical documentation (normalized into DDI XML, PDF, and TIF images). And we keep the content in these formats until we need to migrate to a different format.
In our world of survey research data there is no requirement to preserve original formats (such as a SAS Portable file), look and feel (such as original technical documentation written in Word Perfect), or even low-level data format (such as numeric data coded in punched card format). We can take everything, pull out the essential intellectual content, and then normalize that content in quite durable formats.
That said, we may take these durable formats, such as DDI XML metadata and plain ASCII data, and feed them into tools that produce dissemination formats that are easier for our designated community to use. This includes common stat package formats, and also includes formats that feed into our on-line analysis system. However, these dissemination formats are not preserved, and they contain no essential properties not also contained in the preservation formats.
As I've been thinking about this particular TRAC requirement over the past week, I think we're probably missing one important piece of information for each of our data files: the character set. In practice almost all of our holdings are numeric data, and so our implicit character set ("us-ascii") is the correct one to use, but this seems like a good area for being more explicit. This would allow us to handle data files that contain non-numeric, non-US-ASCII characters in a more durable fashion.
Friday, March 5, 2010
TRAC: C3.4: Disaster preparedness
C3.4 Repository has suitable written disaster preparedness and recovery plan(s), including at least one off-site backup of all preserved information together with an off-site copy of the recovery plan(s).
The repository must have a written plan with some approval process for what happens in specific types of disaster (fire, flood, system compromise, etc.) and for who has responsibility for actions. The level of detail in a disaster plan, and the specific risks addressed need to be appropriate to the repository’s location and service expectations. Fire is an almost universal concern, but earthquakes may not require specific planning at all locations. The disaster plan must, however, deal with unspecified situations that would have specific consequences, such as lack of access to a building.
Evidence: ISO 17799 certification; disaster and recovery plans; information about and proof of at least one off-site copy of preserved information; service continuity plan; documentation linking roles with activities; local geological, geographical, or meteorological data or threat assessments.
Building and documenting systems and procedures for coping with a disaster has a scope well beyond just IT. But there are two key areas worth discussing that fall within the purview of IT.
One area is ensuring that ICPSR is able to deliver its content to its clients, members, and the public at all time. This is an area where we've made significant investments over the past twelve months, and where we also now have a good story to tell.
The main ICPSR delivery mechanism is its web site. The technological resources that power the primary instance of the web site reside at ICPSR itself on the campus of the University of Michigan. This consists of three mains systems: a reasonably powerful server running web applications; another powerful server running an Oracle database; and our storage appliance.
Our equipment resides in an eclectic machine room. On the plus side it has items one would expect to find like equipment cabinets, local air handlers providing chilled air, and UPS to protect us from power fluctuations. On the negative side there is no raised floor or cable trays, which makes for a messy machine room, and our connection to Ann Arbor's (not U-M's) electrical grid is somewhat precarious. An off-site network operations center monitors our gear 24 x 7 and notifies us via SMS, pager, and telephone if anything looks broken.
We maintain a replica of our web environment in Amazon's cloud, and we use a simple mechanism to trigger a failover to the replica: The oncall technician changes the DNS record for www.icpsr.umich.edu to point to the replica instead of the primary. The time-to-live on the record is low (five minutes), and so once the process has been followed, failover is quick. (And the change is made on a "stealth" DNS server that also lives in Amazon's cloud.)
And, finally, we synchronize the replica several times throughout the workday so that software and content is always fresh.
This setup doesn't create a web environment which promises "five nines" type of uptime (i.e., 99.999% availability), but it does give us the capability to avoid any long multi-day outage like we saw late in 2008, and it also gives us the capability to deliver content indefinitely from the replica should ICPSR stuffer a disaster.
The second main area where IT plays a key role is with archival storage. This is less about 24 x 7 availability, and more about ensuring that our archival holdings are available for long-term access in a robust storage fabric.
A post from November 2009 is still an accurate depiction of how we replicate our archival holdings so that we can be guaranteed to have a copy even if something catastrophic happens to our main location in Ann Arbor. I'm also interested in deploying copies outside the United States. We've had some very useful conversations with colleagues at the ANU Supercomputer Facility in Australia, and perhaps some sort of reciprocal storage arrangement might be worked out.
The repository must have a written plan with some approval process for what happens in specific types of disaster (fire, flood, system compromise, etc.) and for who has responsibility for actions. The level of detail in a disaster plan, and the specific risks addressed need to be appropriate to the repository’s location and service expectations. Fire is an almost universal concern, but earthquakes may not require specific planning at all locations. The disaster plan must, however, deal with unspecified situations that would have specific consequences, such as lack of access to a building.
Evidence: ISO 17799 certification; disaster and recovery plans; information about and proof of at least one off-site copy of preserved information; service continuity plan; documentation linking roles with activities; local geological, geographical, or meteorological data or threat assessments.
Building and documenting systems and procedures for coping with a disaster has a scope well beyond just IT. But there are two key areas worth discussing that fall within the purview of IT.
One area is ensuring that ICPSR is able to deliver its content to its clients, members, and the public at all time. This is an area where we've made significant investments over the past twelve months, and where we also now have a good story to tell.
The main ICPSR delivery mechanism is its web site. The technological resources that power the primary instance of the web site reside at ICPSR itself on the campus of the University of Michigan. This consists of three mains systems: a reasonably powerful server running web applications; another powerful server running an Oracle database; and our storage appliance.
Our equipment resides in an eclectic machine room. On the plus side it has items one would expect to find like equipment cabinets, local air handlers providing chilled air, and UPS to protect us from power fluctuations. On the negative side there is no raised floor or cable trays, which makes for a messy machine room, and our connection to Ann Arbor's (not U-M's) electrical grid is somewhat precarious. An off-site network operations center monitors our gear 24 x 7 and notifies us via SMS, pager, and telephone if anything looks broken.
We maintain a replica of our web environment in Amazon's cloud, and we use a simple mechanism to trigger a failover to the replica: The oncall technician changes the DNS record for www.icpsr.umich.edu to point to the replica instead of the primary. The time-to-live on the record is low (five minutes), and so once the process has been followed, failover is quick. (And the change is made on a "stealth" DNS server that also lives in Amazon's cloud.)
And, finally, we synchronize the replica several times throughout the workday so that software and content is always fresh.
This setup doesn't create a web environment which promises "five nines" type of uptime (i.e., 99.999% availability), but it does give us the capability to avoid any long multi-day outage like we saw late in 2008, and it also gives us the capability to deliver content indefinitely from the replica should ICPSR stuffer a disaster.
The second main area where IT plays a key role is with archival storage. This is less about 24 x 7 availability, and more about ensuring that our archival holdings are available for long-term access in a robust storage fabric.
A post from November 2009 is still an accurate depiction of how we replicate our archival holdings so that we can be guaranteed to have a copy even if something catastrophic happens to our main location in Ann Arbor. I'm also interested in deploying copies outside the United States. We've had some very useful conversations with colleagues at the ANU Supercomputer Facility in Australia, and perhaps some sort of reciprocal storage arrangement might be worked out.
Tuesday, February 23, 2010
TRAC: C3.3: Roles, responsibilities, and authorization
C3.3 Repository staff have delineated roles, responsibilities, and authorizations related to implementing changes within the system.
Authorizations are about who can do what—who can add users, who has access to change metadata, who can get at audit logs. It is important that authorizations are justified, that staff understand what they are authorized to do, and that there is a consistent view of this across the organization.
Evidence: ISO 17799 certification; organizational chart; system authorization documentation.
ICPSR is a very mature organization, and experiences very low staff turnover and very little of the reorganizations, mergers, and other organizational changes one finds in the for-profit world. Roles and responsibilities are known well, and the knowledge is common across work groups. We don't often hire completely new types of positions, and so when we do hire a new employee, they tend to slot easily into an existing category of employee (e.g., "data processor").
The combination of a category, or role, and the specific work group - such as, the new hire, Jane, is a Data Processor in SAMHDA - translate easily into a set of related technology containers such as Active Directory groups, LDAP groups, old school UNIX /etc/group entries, and even our own custom database that maps people to groups.
My sense is that the net effect is that we have roles that are well understood, but not crisply defined; and we have containers to hold the roles, but those containers are as idiosyncratic as the collection of technology we use to drive ICPSR.
Authorizations are about who can do what—who can add users, who has access to change metadata, who can get at audit logs. It is important that authorizations are justified, that staff understand what they are authorized to do, and that there is a consistent view of this across the organization.
Evidence: ISO 17799 certification; organizational chart; system authorization documentation.
ICPSR is a very mature organization, and experiences very low staff turnover and very little of the reorganizations, mergers, and other organizational changes one finds in the for-profit world. Roles and responsibilities are known well, and the knowledge is common across work groups. We don't often hire completely new types of positions, and so when we do hire a new employee, they tend to slot easily into an existing category of employee (e.g., "data processor").
The combination of a category, or role, and the specific work group - such as, the new hire, Jane, is a Data Processor in SAMHDA - translate easily into a set of related technology containers such as Active Directory groups, LDAP groups, old school UNIX /etc/group entries, and even our own custom database that maps people to groups.
My sense is that the net effect is that we have roles that are well understood, but not crisply defined; and we have containers to hold the roles, but those containers are as idiosyncratic as the collection of technology we use to drive ICPSR.
Friday, February 19, 2010
TRAC: C3.2: Security controls
C3.2 Repository has implemented controls to adequately address each of the defined security needs.
The repository must show how it has dealt with its security requirements. If some types of material are more likely to be attacked, the repository will need to provide more protection, for instance.
Evidence: ISO 17799 certification; system control list; risk, threat, or control analyses; addition of controls based on ongoing risk detection and assessment.
Broadly speaking ICPSR has only two types of objects to manage: (1) those that have been reviewed and vetted, and are available for wide dissemination, and (2) everything else.
The first category of objects require relatively modest security controls. Because this content is freely available via our web site (albeit some only to members) there are relatively few concerns about its security. It's important that we know that the objects haven't been corrupted (unintentionally or otherwise), and it's also important to ensure that member-only content is accessible only to members.
The second category has grown considerably in the past few years, particularly as ICPSR receives more confidential data through government agencies, partners, and unsolicited via its deposit system.
To meet the rising challenge of confidential data ICPSR will be changing the architecture of its network, and will also be deploying a Secure Data processing Environment. (More on the SDE in a subsequent post.) Both of these steps supplement existing security controls that we discussed in the post The ICPSR Web Site and Security from September 2009.
Our current network architecture is very flat. All network-connected equipment is part of a single virtual local area network (VLAN), and items are protected at the border between ICPSR and the outside world via Cisco access-list policies. Individual systems may also be protected via local firewalls, such as iptables on a Linux server. And equipment located in Amazon's cloud is protected via similar mechanisms using AWS EC2 Security Groups.
Our new network architecture will look very different, and will consist of four separate VLANs. Here is the overall picture:
This new architecture makes use of a new security service provided by the University of Michigan's central IT organization, ITS. The service is called Virtual Firewall, and is based on the Checkpoint firewall product.
This architecture divides ICPSR's network-connected resources into four VLANs:
The Semi-private VLAN hosts the bulk of ICPSR's current systems: desktop computers, printers, storage systems, and so on. This equipment needs robust outbound access to the Internet, but inbound access should be restricted closely. For example, there is no good reason why ICPSR's printers need to be accessible from any arbitrary host on the Internet.
The Private VLANs will host our most secure systems. This will include the Secure Data processing Environment (SDE) we're building, and also a future Virtual Data Enclave (VDE). Access between these VLANs and the Internet will be highly, highly restricted, and more general access will be permitted only from our Semi-private VLAN.
We've just started the process of building out this future network architecture with ITS, and expect that we'll finish the deployment and transition before the end of the summer.
The repository must show how it has dealt with its security requirements. If some types of material are more likely to be attacked, the repository will need to provide more protection, for instance.
Evidence: ISO 17799 certification; system control list; risk, threat, or control analyses; addition of controls based on ongoing risk detection and assessment.
Broadly speaking ICPSR has only two types of objects to manage: (1) those that have been reviewed and vetted, and are available for wide dissemination, and (2) everything else.
The first category of objects require relatively modest security controls. Because this content is freely available via our web site (albeit some only to members) there are relatively few concerns about its security. It's important that we know that the objects haven't been corrupted (unintentionally or otherwise), and it's also important to ensure that member-only content is accessible only to members.
The second category has grown considerably in the past few years, particularly as ICPSR receives more confidential data through government agencies, partners, and unsolicited via its deposit system.
To meet the rising challenge of confidential data ICPSR will be changing the architecture of its network, and will also be deploying a Secure Data processing Environment. (More on the SDE in a subsequent post.) Both of these steps supplement existing security controls that we discussed in the post The ICPSR Web Site and Security from September 2009.
Our current network architecture is very flat. All network-connected equipment is part of a single virtual local area network (VLAN), and items are protected at the border between ICPSR and the outside world via Cisco access-list policies. Individual systems may also be protected via local firewalls, such as iptables on a Linux server. And equipment located in Amazon's cloud is protected via similar mechanisms using AWS EC2 Security Groups.
Our new network architecture will look very different, and will consist of four separate VLANs. Here is the overall picture:
This new architecture makes use of a new security service provided by the University of Michigan's central IT organization, ITS. The service is called Virtual Firewall, and is based on the Checkpoint firewall product.
This architecture divides ICPSR's network-connected resources into four VLANs:
- Public
- Semi-private
- Private (local to ICPSR)
- Private (local to ITS)
The Semi-private VLAN hosts the bulk of ICPSR's current systems: desktop computers, printers, storage systems, and so on. This equipment needs robust outbound access to the Internet, but inbound access should be restricted closely. For example, there is no good reason why ICPSR's printers need to be accessible from any arbitrary host on the Internet.
The Private VLANs will host our most secure systems. This will include the Secure Data processing Environment (SDE) we're building, and also a future Virtual Data Enclave (VDE). Access between these VLANs and the Internet will be highly, highly restricted, and more general access will be permitted only from our Semi-private VLAN.
We've just started the process of building out this future network architecture with ITS, and expect that we'll finish the deployment and transition before the end of the summer.
Friday, February 12, 2010
TRAC: C3.1: Risk assessment
C3.1 Repository maintains a systematic analysis of such factors as data, systems, personnel, physical plant, and security needs.
Regular risk assessment should address external threats and denial of service attacks. These analyses are likely to be documented in several different places, and need not be comprehensively contained in a single document.
Evidence: ISO 17799 certification; documentation describing analysis and risk assessments undertaken and their outputs; logs from environmental recorders; confirmation of successful staff vetting.
ICPSR performs regular risk assessment using a variety of techniques.
One, on a purely technological level, ICPSR analyzes its security exposure through regular vulnerability scans that are conducted by the University of Michigan's Information Technology Security Services (ITSS) team. This includes participation in a mandatory, quarterly, campus-wide scan, and also an optional, monthly, detailed scan of our own network.
Two, ICPSR and its parent organization, the Institute for Social Research (ISR), undergo periodic Information Technology audits that are conducted by the University of Michigan's Office of University Audits. These are comprehensive assessments of physical and electronic security.
Three, ICPSR and its parent organization, ISR, conduct two risk evaluation projects each year to assess risks and vulnerabilities in IT infrastructure. The most recent one undertaken by ICPSR was in 2009, and the focus was on research data security.
Four, again, ICPSR along with ISR, participate in an annual review and refresh of an organization-wide IT security plan. This is a comprehensive evaluation of vulnerabilities, strengths, and resources related to security, and flows into a University of Michigan central repository of security plans, and is also reviewed by the leadership of ISR.
Five, ICPSR also participated in the Center for Research Libraries Auditing and Certification of Digital Archives Project "test audit" in 2005-2006. This audit reviewed a broad spectrum of functions and services at ICPSR (including IT), and produced a publicly available report. The report was very favorable to ICPSR:
Regular risk assessment should address external threats and denial of service attacks. These analyses are likely to be documented in several different places, and need not be comprehensively contained in a single document.
Evidence: ISO 17799 certification; documentation describing analysis and risk assessments undertaken and their outputs; logs from environmental recorders; confirmation of successful staff vetting.
ICPSR performs regular risk assessment using a variety of techniques.
One, on a purely technological level, ICPSR analyzes its security exposure through regular vulnerability scans that are conducted by the University of Michigan's Information Technology Security Services (ITSS) team. This includes participation in a mandatory, quarterly, campus-wide scan, and also an optional, monthly, detailed scan of our own network.
Two, ICPSR and its parent organization, the Institute for Social Research (ISR), undergo periodic Information Technology audits that are conducted by the University of Michigan's Office of University Audits. These are comprehensive assessments of physical and electronic security.
Three, ICPSR and its parent organization, ISR, conduct two risk evaluation projects each year to assess risks and vulnerabilities in IT infrastructure. The most recent one undertaken by ICPSR was in 2009, and the focus was on research data security.
Four, again, ICPSR along with ISR, participate in an annual review and refresh of an organization-wide IT security plan. This is a comprehensive evaluation of vulnerabilities, strengths, and resources related to security, and flows into a University of Michigan central repository of security plans, and is also reviewed by the leadership of ISR.
Five, ICPSR also participated in the Center for Research Libraries Auditing and Certification of Digital Archives Project "test audit" in 2005-2006. This audit reviewed a broad spectrum of functions and services at ICPSR (including IT), and produced a publicly available report. The report was very favorable to ICPSR:
Auditors found an organization with a mature, fully operational archive. ICPSR as an organization has a 44-year history of growing and managing a large (at the time of the audit 2.3 Terabytes) data archive of valuable content with a virtually unblemished record in data management and access. The future prospects of both the organization and the data appear favorable, due to a combination of sound financial management and planning, multiple sources of revenue, robust reporting and accountability mechanisms, and sound technical decisions related to processes, procedures, and formats.In summary, ICPSR conducts risk assessment - particularly technological risk assessment - in a variety of ways on a routine basis.
Monday, February 8, 2010
TRAC: C2.2: Appropriate software technologies
C2.2 Repository has software technologies appropriate to the services it provides to its designated community(ies) and has procedures in place to receive and monitor notifications, and evaluate when software technology changes are needed.
The repository needs to be aware of the types of access services expected by its designated community(ies), and to make sure its software capabilities can support these services. For example, it may need to add format translations to meet the needs of currently widely used application tools, or it may need to add a data subsetting service for very large data objects.
Evidence: Technology watch; documentation of procedures; designated community profiles; user needs evaluation; software inventory.
My sense is that ICPSR is very strong in this area, and is, in general, quite strong in areas that touch the Access portion of the OAIS framework.
Regardless of the format in which it receives content, and regardless of the internal format used to preserve the content, ICPSR routinely produces the "full product suite" of formats for delivery on the web site. At this time this includes platform-independent versions of SPSS, SAS, and Stata, as well as tab-separate values (for those that might want to use Excel or Access), and even plain ASCII data with setups.
ICPSR also prepares the content for use with our online analysis systems, SDA from UC Berkeley. This opens up access to the data for those users who lack statistical analysis software, and also allows one to subset the data prior to download.
Also, ICPSR spends a considerable amount of resources on data discovery and data browsing systems, such as our new faceted search, and the extensive metadata we prepare to document the data. These too might fall into the "appropriate software" portion of TRAC.
Does my sense of how we're doing match that of the communities we serve? How well do you think we're doing at providing the right types of access and appropriate formats?
The repository needs to be aware of the types of access services expected by its designated community(ies), and to make sure its software capabilities can support these services. For example, it may need to add format translations to meet the needs of currently widely used application tools, or it may need to add a data subsetting service for very large data objects.
Evidence: Technology watch; documentation of procedures; designated community profiles; user needs evaluation; software inventory.
My sense is that ICPSR is very strong in this area, and is, in general, quite strong in areas that touch the Access portion of the OAIS framework.
Regardless of the format in which it receives content, and regardless of the internal format used to preserve the content, ICPSR routinely produces the "full product suite" of formats for delivery on the web site. At this time this includes platform-independent versions of SPSS, SAS, and Stata, as well as tab-separate values (for those that might want to use Excel or Access), and even plain ASCII data with setups.
ICPSR also prepares the content for use with our online analysis systems, SDA from UC Berkeley. This opens up access to the data for those users who lack statistical analysis software, and also allows one to subset the data prior to download.
Also, ICPSR spends a considerable amount of resources on data discovery and data browsing systems, such as our new faceted search, and the extensive metadata we prepare to document the data. These too might fall into the "appropriate software" portion of TRAC.
Does my sense of how we're doing match that of the communities we serve? How well do you think we're doing at providing the right types of access and appropriate formats?
Friday, January 8, 2010
TRAC: C2.1: Appropriate hardware technologies
C2.1 Repository has hardware technologies appropriate to the services it provides to its designated community(ies) and has procedures in place to receive and monitor notifications, and evaluate when hardware technology changes are needed.
The repository needs to be aware of the types of access services expected by its designated community(ies), including, where applicable, the types of media to be delivered, and needs to make sure its hardware capabilities can support these services. For example, it may need to improve its networking bandwidth over time to meet growing access data volumes and expectations.
Evidence: Technology watch; documentation of procedures; designated community profiles; user needs evaluation; hardware inventory.
For the types of material that ICPSR preserves and delivers to its clients, hardware selection has been relatively unimportant since the birth and promulgation of the Web.
In ICPSR's early days, when it would send magnetic tapes to its member schools for use on their own campus, hardware selection, particularly in terms of media types, was a very important consideration. If most campuses were using, say, IBM labeled cartridge tapes routinely but ICPSR was sending ANSI labeled 9-track tapes, this would have imposed an additional burden on our designated community. However, in a world where the typical access is via a Web download, the underlying hardware selection - disk, server, router, switch - just doesn't matter all that much.
Also, because ICPSR's unit of delivery is relatively small - most of our content is on the order of megabytes - issues such as network bandwidth are not critical. In a typical day ICPSR might deliver 20GB of content to hundreds or even thousands of separate sessions, and so with even modest networking resources (1 Gb/s NIC, 1 Gb/s switch, 10 Gb/s campus backbone, etc), the "pipes" are more than sufficient to deliver good service.
One interesting area to consider with regard to hardware selection is virtualization. For example, in some cases it might make more sense for a depositor to leave ICPSR with an image of a complete computing system in addition to a dataset. In most cases the deposit contains simple rectangular data that can be normalized relatively easily into plain character data and accompanying "setups" to make it easy to use with the most common statistical packages. However, if the dataset were more complex - say a relational database - then normalizing the data could be prohibitively expensive, or damaging to the data, or both. In this case, having a machine image that would contain a host operating system, the database application, and the user database might be a useful addition both for preservation and delivery.
The repository needs to be aware of the types of access services expected by its designated community(ies), including, where applicable, the types of media to be delivered, and needs to make sure its hardware capabilities can support these services. For example, it may need to improve its networking bandwidth over time to meet growing access data volumes and expectations.
Evidence: Technology watch; documentation of procedures; designated community profiles; user needs evaluation; hardware inventory.
For the types of material that ICPSR preserves and delivers to its clients, hardware selection has been relatively unimportant since the birth and promulgation of the Web.
In ICPSR's early days, when it would send magnetic tapes to its member schools for use on their own campus, hardware selection, particularly in terms of media types, was a very important consideration. If most campuses were using, say, IBM labeled cartridge tapes routinely but ICPSR was sending ANSI labeled 9-track tapes, this would have imposed an additional burden on our designated community. However, in a world where the typical access is via a Web download, the underlying hardware selection - disk, server, router, switch - just doesn't matter all that much.
Also, because ICPSR's unit of delivery is relatively small - most of our content is on the order of megabytes - issues such as network bandwidth are not critical. In a typical day ICPSR might deliver 20GB of content to hundreds or even thousands of separate sessions, and so with even modest networking resources (1 Gb/s NIC, 1 Gb/s switch, 10 Gb/s campus backbone, etc), the "pipes" are more than sufficient to deliver good service.
One interesting area to consider with regard to hardware selection is virtualization. For example, in some cases it might make more sense for a depositor to leave ICPSR with an image of a complete computing system in addition to a dataset. In most cases the deposit contains simple rectangular data that can be normalized relatively easily into plain character data and accompanying "setups" to make it easy to use with the most common statistical packages. However, if the dataset were more complex - say a relational database - then normalizing the data could be prohibitively expensive, or damaging to the data, or both. In this case, having a machine image that would contain a host operating system, the database application, and the user database might be a useful addition both for preservation and delivery.
Tuesday, January 5, 2010
TRAC: C1.10: Patching and risk-benefit assessment
C1.10 Repository has a process to react to the availability of new software security updates based on a risk-benefit assessment.
Decisions to apply security updates are likely to be the outcome of a risk-benefit assessment; security patches are frequently responsible for upsetting alternative aspects of system functionality or performance. It may not be necessary for a repository to implement all software patches, and the application of any must be carefully considered. Each security update implemented by the repository must be documented with details was about how it is completed; both automated and manual updates are acceptable. Significant security updates might pertain to software other than core operating systems, such as database applications and Web servers, and these should also be documented.
Evidence: Risk register (list of all patches available and risk documentation analysis); evidence of update processes (e.g., server update manager daemon); documentation related to the update installations.
Like many organizations ICPSR uses an automated system for basic security patching, both on the Windows platform and on the Linux platform. In this context there is absolutely no risk assessment for an individual patch; they flow from a trusted source, and are installed automatically.
For our most sensitive components, such as the Apache web server, Oracle, and the kernel we run on our Linux systems, we install patches and/or upgrade versions manually. These changes take place less often, and at a minimum are announced well ahead of time to the entire group. For example, a systems administrator would announce an intention to upgrade to the newest release of the Apache web server on our staging server, see how things work for a week, and then announce the same upgrade on our production web server. This isn't captured in anything as formal as a risk register, but there is clear and frequent communication about the change.
Decisions to apply security updates are likely to be the outcome of a risk-benefit assessment; security patches are frequently responsible for upsetting alternative aspects of system functionality or performance. It may not be necessary for a repository to implement all software patches, and the application of any must be carefully considered. Each security update implemented by the repository must be documented with details was about how it is completed; both automated and manual updates are acceptable. Significant security updates might pertain to software other than core operating systems, such as database applications and Web servers, and these should also be documented.
Evidence: Risk register (list of all patches available and risk documentation analysis); evidence of update processes (e.g., server update manager daemon); documentation related to the update installations.
Like many organizations ICPSR uses an automated system for basic security patching, both on the Windows platform and on the Linux platform. In this context there is absolutely no risk assessment for an individual patch; they flow from a trusted source, and are installed automatically.
For our most sensitive components, such as the Apache web server, Oracle, and the kernel we run on our Linux systems, we install patches and/or upgrade versions manually. These changes take place less often, and at a minimum are announced well ahead of time to the entire group. For example, a systems administrator would announce an intention to upgrade to the newest release of the Apache web server on our staging server, see how things work for a week, and then announce the same upgrade on our production web server. This isn't captured in anything as formal as a risk register, but there is clear and frequent communication about the change.
Friday, December 25, 2009
TRAC: C1.9: Change testing
C1.9 Repository has a process for testing the effect of critical changes to the system.
Changes to critical systems should be, where possible, pre-tested separately, the expected behaviors documented, and roll-back procedures prepared. After changes, the systems should be monitored for unexpected and unacceptable behavior. If such behavior is discovered the changes and their consequences should be reversed.
Whole-system testing or unit testing can address this requirement; complex safety-type tests are not required. Testing can be very expensive, but there should be some recognition of the fact that a completely open regime where no changes are ever evaluated or tested will have problems.
Evidence: Documented testing procedures; documentation of results from prior tests and proof of changes made as a result of tests.
When I arrived at ICPSR in 2002 the same systems were used for both production services and for new development. In fact, the total number of systems at ICPSR was very small; for example, a single machine was our production database machine, our shared "time sharing" machine for UNIX-based data processing, our anonymous ftp server, and pretty much everything else.
Our model is quite different these days. Most of our software development occurs on the desktop of the programmer, and new code is rolled out to a staging server for testing and evaluation. The level and intensity of the testing varies widely at ICPSR; my sense is that public-facing systems get the most evaluation, and internal-only systems receive very little. Nonetheless, unless the evaluation reveals serious flaws, the software is then rolled into production on a fixed deployment schedule. Because all software resides in a repository, we can back out changes easily if needed.
The last six software developers we've hired have all worked in the Java environment, and we're in the process of moving our Perl/CGI team to Java as well. My sense is that getting all of the major systems in Java will make it easier to use unit-testing tools, like JUnit for example.
Changes to critical systems should be, where possible, pre-tested separately, the expected behaviors documented, and roll-back procedures prepared. After changes, the systems should be monitored for unexpected and unacceptable behavior. If such behavior is discovered the changes and their consequences should be reversed.
Whole-system testing or unit testing can address this requirement; complex safety-type tests are not required. Testing can be very expensive, but there should be some recognition of the fact that a completely open regime where no changes are ever evaluated or tested will have problems.
Evidence: Documented testing procedures; documentation of results from prior tests and proof of changes made as a result of tests.
When I arrived at ICPSR in 2002 the same systems were used for both production services and for new development. In fact, the total number of systems at ICPSR was very small; for example, a single machine was our production database machine, our shared "time sharing" machine for UNIX-based data processing, our anonymous ftp server, and pretty much everything else.
Our model is quite different these days. Most of our software development occurs on the desktop of the programmer, and new code is rolled out to a staging server for testing and evaluation. The level and intensity of the testing varies widely at ICPSR; my sense is that public-facing systems get the most evaluation, and internal-only systems receive very little. Nonetheless, unless the evaluation reveals serious flaws, the software is then rolled into production on a fixed deployment schedule. Because all software resides in a repository, we can back out changes easily if needed.
The last six software developers we've hired have all worked in the Java environment, and we're in the process of moving our Perl/CGI team to Java as well. My sense is that getting all of the major systems in Java will make it easier to use unit-testing tools, like JUnit for example.
Friday, December 11, 2009
TRAC: C1.8: Change management
C1.8 Repository has a documented change management process that identifies changes to critical processes that potentially affect the repository’s ability to comply with its mandatory responsibilities.
Examples of this would include changes in processes in data management, access, archival storage, ingest, and security. The really important thing is to be able to know what changes were made and when they were made. Traceability makes it possible to understand what was affected by particular changes to the systems.
Evidence: Documentation of change management process; comparison of logs of actual system changes to processes versus associated analyses of their impact and criticality.
Establishing and following a change management process is a lot like stretching before working out. We know we should do it, we feel better when we do, but, wow, is it ever hard to make a point to do it.
I've used a change management process in the past. At ANS Communications we would only make major configuration changes to the network on certain days and at certain times, and we would drive the changes from a central database. This was particularly important when we made a long and very painful transition away from the NSFnet-era equipment that had formed our backbone network to new equipment from a company known as Bay Networks.
At ICPSR I think we could use something fairly straight-forward. There are only a handful of critical software systems, and they don't change that often. We already track software-level changes in CVS, and we already announce feature-level changes to the designated community (e.g., ICPSR staff for internal systems), and so we might pull it all together by linking the announcements with the code changes in JIRA. I could also imagine a thread on our Intranet (which is Drupal-based) which could form a central summary of changes: what, when, how, and links to more details.
Examples of this would include changes in processes in data management, access, archival storage, ingest, and security. The really important thing is to be able to know what changes were made and when they were made. Traceability makes it possible to understand what was affected by particular changes to the systems.
Evidence: Documentation of change management process; comparison of logs of actual system changes to processes versus associated analyses of their impact and criticality.
Establishing and following a change management process is a lot like stretching before working out. We know we should do it, we feel better when we do, but, wow, is it ever hard to make a point to do it.
I've used a change management process in the past. At ANS Communications we would only make major configuration changes to the network on certain days and at certain times, and we would drive the changes from a central database. This was particularly important when we made a long and very painful transition away from the NSFnet-era equipment that had formed our backbone network to new equipment from a company known as Bay Networks.
At ICPSR I think we could use something fairly straight-forward. There are only a handful of critical software systems, and they don't change that often. We already track software-level changes in CVS, and we already announce feature-level changes to the designated community (e.g., ICPSR staff for internal systems), and so we might pull it all together by linking the announcements with the code changes in JIRA. I could also imagine a thread on our Intranet (which is Drupal-based) which could form a central summary of changes: what, when, how, and links to more details.
Friday, November 13, 2009
TRAC: C1.7: Refreshing/migrating content
C1.7 Repository has defined processes for storage media and/or hardware change (e.g., refreshing, migration).
The repository should have triggers for initiating action and understanding of how long it will take for storage media migration, or refreshing — copying between media without reformatting the bitstream. Will it finish before the media is dead, for instance? Copying large quantities of data can take a long time and can affect other system performance. It is important that the process includes a check that the copying has happened correctly.
Repositories should also consider the obsolescence of any/all hardware components within the repository system as potential trigger events for migration. Increasingly, long-term, appropriate support for system hardware components is difficult to obtain, exposing repositories to risks and liabilities should they chose to continue to operate the hardware beyond the manufacturer or third-party support.
Evidence: Documentation of processes; policies related to hardware support, maintenance, and replacement; documentation of hardware manufacturers’ expected support life cycles.
ICPSR's archival storage consumes less than 6 TB of storage today. Over the past month we've made copies in other locations, and the time to copy it across a network is anywhere from a day to a week, depending upon the speed of the network. So that's much shorter than the lifespan of the media. :-)
The master copy resides on an EMC Celera NAS. From time to time one of the SATA drives that underpins archival storage will fail, and the Celera will fail over to its hot spare, and make a phone call for EMC to schedule a replacement. And, so in some odd way, the media gets refreshed on an incremental basis slowly over time.
We bought our Celera in 2005, and my expectation is that we'll likely replace it with something else in 2010; 2011 at the very latest. And so it's timely to start thinking about a written procedure for moving the master copy of the content from the Celera to the next storage platform. I don't think it will be a complicated procedure, and putting it together might make for a good future post.
The repository should have triggers for initiating action and understanding of how long it will take for storage media migration, or refreshing — copying between media without reformatting the bitstream. Will it finish before the media is dead, for instance? Copying large quantities of data can take a long time and can affect other system performance. It is important that the process includes a check that the copying has happened correctly.
Repositories should also consider the obsolescence of any/all hardware components within the repository system as potential trigger events for migration. Increasingly, long-term, appropriate support for system hardware components is difficult to obtain, exposing repositories to risks and liabilities should they chose to continue to operate the hardware beyond the manufacturer or third-party support.
Evidence: Documentation of processes; policies related to hardware support, maintenance, and replacement; documentation of hardware manufacturers’ expected support life cycles.
ICPSR's archival storage consumes less than 6 TB of storage today. Over the past month we've made copies in other locations, and the time to copy it across a network is anywhere from a day to a week, depending upon the speed of the network. So that's much shorter than the lifespan of the media. :-)
The master copy resides on an EMC Celera NAS. From time to time one of the SATA drives that underpins archival storage will fail, and the Celera will fail over to its hot spare, and make a phone call for EMC to schedule a replacement. And, so in some odd way, the media gets refreshed on an incremental basis slowly over time.
We bought our Celera in 2005, and my expectation is that we'll likely replace it with something else in 2010; 2011 at the very latest. And so it's timely to start thinking about a written procedure for moving the master copy of the content from the Celera to the next storage platform. I don't think it will be a complicated procedure, and putting it together might make for a good future post.
Wednesday, November 4, 2009
TRAC: C1.6: Reporting and repairing loss
C1.6 Repository reports to its administration all incidents of data corruption or loss, and steps taken to repair/replace corrupt or lost data.
Having effective mechanisms to detect bit corruption and loss within a repository system is critical, but is only one important part of a larger process. As a whole, the repository must record, report, and repair as possible all violations of data integrity. This means the system should be able to notify system administrators of any logged problems. These incidents, recovery actions, and their results must be reported to administrators and should be available.
For example, the repository should document procedures to take when loss or corruption is detected, including standards for measuring the success of recoveries. Any actions taken to repair objects as part of these procedures must be recorded. The nature of this recording must be documented by the repository, and the information must be retrievable when required. This documentation plays a critical role in the measurement of the authenticity and integrity of the data held by the repository.
Evidence: Preservation metadata (e.g., PDI) records; comparison of error logs to reports to administration; escalation procedures related to data loss.
My sense is that this requirement is just about policy as it is process. Fortunately for our data holdings (but unfortunately for TRAC preparation), data loss or corruption is a very infrequent event, and therefore as one might expect, the set of policies and written processes is pretty small.
As a point of comparison, if we look at our policies and processes for handling loss with "working files" we will find a much richer set of policies and systems. We have established infrastructure (an EMC Network Attached Storage (NAS) storage applicance and associated Dell tape management solution); we have internal policies and processes that document how to retrieve lost content; we have external policies that describe which parts of the NAS are written to tape, and the schedule of tape backups; and, we exercise the system on a regular basis as people inadvertently delete or damage files with which they are working actively.
On the Archival Storage side - or even the Access side, where we also look for loss and corruption - the number of data loss or data corruption events is very, very low. Email reports come out on a regular basis, but they always (almost) say that everything is fine. And on that rare occasion where there is an issue, the remedy is quick.
Perhaps the right solution here is to use the small sample of issues that have arisen over the years as our baseline for writing up a process, and then posting that process on our internal web site. That would be easy to do. But then the concern is this: If a policy is used very, very infrequently, it is likely to fall into disrepair. It is also likely to become forgotten. Maybe the tool that examines for loss or corruption should also contain a link to the relevant policies and recovery processes?
What strategies have others used to address this TRAC requirement?
Having effective mechanisms to detect bit corruption and loss within a repository system is critical, but is only one important part of a larger process. As a whole, the repository must record, report, and repair as possible all violations of data integrity. This means the system should be able to notify system administrators of any logged problems. These incidents, recovery actions, and their results must be reported to administrators and should be available.
For example, the repository should document procedures to take when loss or corruption is detected, including standards for measuring the success of recoveries. Any actions taken to repair objects as part of these procedures must be recorded. The nature of this recording must be documented by the repository, and the information must be retrievable when required. This documentation plays a critical role in the measurement of the authenticity and integrity of the data held by the repository.
Evidence: Preservation metadata (e.g., PDI) records; comparison of error logs to reports to administration; escalation procedures related to data loss.
My sense is that this requirement is just about policy as it is process. Fortunately for our data holdings (but unfortunately for TRAC preparation), data loss or corruption is a very infrequent event, and therefore as one might expect, the set of policies and written processes is pretty small.
As a point of comparison, if we look at our policies and processes for handling loss with "working files" we will find a much richer set of policies and systems. We have established infrastructure (an EMC Network Attached Storage (NAS) storage applicance and associated Dell tape management solution); we have internal policies and processes that document how to retrieve lost content; we have external policies that describe which parts of the NAS are written to tape, and the schedule of tape backups; and, we exercise the system on a regular basis as people inadvertently delete or damage files with which they are working actively.
On the Archival Storage side - or even the Access side, where we also look for loss and corruption - the number of data loss or data corruption events is very, very low. Email reports come out on a regular basis, but they always (almost) say that everything is fine. And on that rare occasion where there is an issue, the remedy is quick.
Perhaps the right solution here is to use the small sample of issues that have arisen over the years as our baseline for writing up a process, and then posting that process on our internal web site. That would be easy to do. But then the concern is this: If a policy is used very, very infrequently, it is likely to fall into disrepair. It is also likely to become forgotten. Maybe the tool that examines for loss or corruption should also contain a link to the relevant policies and recovery processes?
What strategies have others used to address this TRAC requirement?
Wednesday, October 28, 2009
TRAC: C1.5: Detecting corruption and loss
C1.5 Repository has effective mechanisms to detect bit corruption or loss.
The repository must detect data loss accurately to ensure that any losses fall within the tolerances established by policy (see A3.6). Data losses must be detected and detectable regardless of the source of the loss. This applies to all forms and scope of data corruption, including missing objects and corrupt or incorrect or imposter objects, corruption within an object, and copying errors during data migration or synchronization of copies. Ideally, the repository will demonstrate that it has all the AIPs it is supposed to have and no others, and that they and their metadata are uncorrupted.
The approach must be documented and justified and include mechanisms for mitigating such common hazards as hardware failure, human error, and malicious action. Repositories that use well-recognized mechanisms such as MD5 signatures need only recognize their effectiveness and role within the overall approach. But to the extent the repository relies on homegrown schemes, it must provide convincing justification that data loss and corruption are detected within the tolerances established by policy.
Data losses must be detected promptly enough that routine systemic sources of failure, such as hardware failures, are unlikely to accumulate and cause data loss beyond the tolerances established by the repository’s policy or specified in any relevant deposit agreement. For example, consider a repository that maintains a collection on identical primary and backup copies with no other data redundancy mechanism. If the media of the two copies have a measured failure rate of 1% per year and failures are independent, then there is a 0.01% chance that both copies will fail in the same year. If a repository’s policy limits loss to no more than 0.001% of the collection per year, with a goal of course of losing 0%, then the repository would need to confirm media integrity at least every 72 days to achieve an average time to recover of 36 days, or about one tenth of a year. This simplified example illustrates the kind of issues a repository should consider, but the objective is a comprehensive treatment of the sources of data loss and their real-world complexity. Any data that is (temporarily) lost should be recoverable from backups.
Evidence: Documents that specify bit error detection and correction mechanisms used; risk analysis; error reports; threat analyses.
For each object in Archival Storage, ICPSR computes a MD5 hash. This "fingerprint" is then stored as metadata for each object.
Automated jobs "prowl" Archival Storage on a regular basis computing the current MD5 hash for an object, and comparing it to the stored version. In the case where the hashes differ, and exception is generated, and this information is reported to the appropriate staff for diagnosis and correction.
In practice we see very few exceptions such as these, and the most common cause is a blend of human-error and software failing to handle the error gracefully.
Recovery is quick. In the event the problem was caused by human-error, and the ctime (last modified) timestamp has changed, then any copies managed via rsync may also be damaged, and we instead need to fetch the original object from a different source (e.g., tape or a copy managed via SRB's Srsync). In the event the problem was caused without ctime also changing, then we also have the option of fetching an original copy from one of our rsync-managed copies.
The repository must detect data loss accurately to ensure that any losses fall within the tolerances established by policy (see A3.6). Data losses must be detected and detectable regardless of the source of the loss. This applies to all forms and scope of data corruption, including missing objects and corrupt or incorrect or imposter objects, corruption within an object, and copying errors during data migration or synchronization of copies. Ideally, the repository will demonstrate that it has all the AIPs it is supposed to have and no others, and that they and their metadata are uncorrupted.
The approach must be documented and justified and include mechanisms for mitigating such common hazards as hardware failure, human error, and malicious action. Repositories that use well-recognized mechanisms such as MD5 signatures need only recognize their effectiveness and role within the overall approach. But to the extent the repository relies on homegrown schemes, it must provide convincing justification that data loss and corruption are detected within the tolerances established by policy.
Data losses must be detected promptly enough that routine systemic sources of failure, such as hardware failures, are unlikely to accumulate and cause data loss beyond the tolerances established by the repository’s policy or specified in any relevant deposit agreement. For example, consider a repository that maintains a collection on identical primary and backup copies with no other data redundancy mechanism. If the media of the two copies have a measured failure rate of 1% per year and failures are independent, then there is a 0.01% chance that both copies will fail in the same year. If a repository’s policy limits loss to no more than 0.001% of the collection per year, with a goal of course of losing 0%, then the repository would need to confirm media integrity at least every 72 days to achieve an average time to recover of 36 days, or about one tenth of a year. This simplified example illustrates the kind of issues a repository should consider, but the objective is a comprehensive treatment of the sources of data loss and their real-world complexity. Any data that is (temporarily) lost should be recoverable from backups.
Evidence: Documents that specify bit error detection and correction mechanisms used; risk analysis; error reports; threat analyses.
For each object in Archival Storage, ICPSR computes a MD5 hash. This "fingerprint" is then stored as metadata for each object.
Automated jobs "prowl" Archival Storage on a regular basis computing the current MD5 hash for an object, and comparing it to the stored version. In the case where the hashes differ, and exception is generated, and this information is reported to the appropriate staff for diagnosis and correction.
In practice we see very few exceptions such as these, and the most common cause is a blend of human-error and software failing to handle the error gracefully.
Recovery is quick. In the event the problem was caused by human-error, and the ctime (last modified) timestamp has changed, then any copies managed via rsync may also be damaged, and we instead need to fetch the original object from a different source (e.g., tape or a copy managed via SRB's Srsync). In the event the problem was caused without ctime also changing, then we also have the option of fetching an original copy from one of our rsync-managed copies.
Wednesday, October 21, 2009
TRAC: C1.4: Synchronizing objects
C1.4 Repository has mechanisms in place to ensure any/multiple copies of digital objects are synchronized.
If multiple copies exist, there has to be some way to ensure that intentional changes to an object are propagated to all copies of the object. There must be an element of timeliness to this. It must be possible to know when the synchronization has completed, and ideally to have some estimate beforehand as to how long it will take. Depending whether it is automated or requires manual action (such as the retrieval of copies from off-site storage), the time involved may be seconds or weeks. The duration itself is immaterial—what is important is that there is understanding of how long it will take. There must also be something that addresses what happens while the synchronization is in progress. This has an impact on disaster recovery: what happens if a disaster and an update coincide? If one copy of an object is altered and a disaster occurs while other copies are being updated, it is essential to be able to ensure later that the update is successfully propagated.
Evidence: Workflows; system analysis of how long it takes for copies to synchronize; procedures/documentation of operating procedures related to updates and copy synchronization; procedures/documentation related to whether changes lead to the creation of new copies and how those copies are propagated and/or linked to previous versions.
I think we have a good story to tell.
As new objects enter Archival Storage at ICPSR, they reside in a well-known, special-purpose location. Automated, regularly scheduled system jobs synchronize those objects with remote locations using standard, established tools such as rsync and other, less common tools such as Storage Resource Broker (SRB) command-line utilities, such as Srsync.
The output of these system jobs is captured and delivered nightly to a shared electronic mailbox. The mailbox is reviewed on a daily basis; this task belongs to the member of the ICPSR IT team who is currently on-call. When a report is missing or when a report indicates an error, the problem is escalated to someone who can diagnose and correct the problem. One common problem, for example, occurs when an object larger than 2GB enters Archival Storage and the SRB Srsync utility faults. (SRB limits objects to 2GB.) We then remove this object from the list of items to be synchronized with SRB.
Because the synchronization process is incremental, it has a very short duration. However, if we were to need to synchronize ALL content, this takes on the order of days or even weeks. For example, we recently synchronized a copy of our Access holdings to a computing instance residing in Amazon's EC2 EU-West region, and we found it took approximately one week to copy about 500GB. As another example, we recently synchronized a copy of our Archival Storage (which is much larger than the Access collection) to a system, which like ICPSR and the University of Michigan, is connected to Internet2's Abilene network, and that took far less time.
If multiple copies exist, there has to be some way to ensure that intentional changes to an object are propagated to all copies of the object. There must be an element of timeliness to this. It must be possible to know when the synchronization has completed, and ideally to have some estimate beforehand as to how long it will take. Depending whether it is automated or requires manual action (such as the retrieval of copies from off-site storage), the time involved may be seconds or weeks. The duration itself is immaterial—what is important is that there is understanding of how long it will take. There must also be something that addresses what happens while the synchronization is in progress. This has an impact on disaster recovery: what happens if a disaster and an update coincide? If one copy of an object is altered and a disaster occurs while other copies are being updated, it is essential to be able to ensure later that the update is successfully propagated.
Evidence: Workflows; system analysis of how long it takes for copies to synchronize; procedures/documentation of operating procedures related to updates and copy synchronization; procedures/documentation related to whether changes lead to the creation of new copies and how those copies are propagated and/or linked to previous versions.
I think we have a good story to tell.
As new objects enter Archival Storage at ICPSR, they reside in a well-known, special-purpose location. Automated, regularly scheduled system jobs synchronize those objects with remote locations using standard, established tools such as rsync and other, less common tools such as Storage Resource Broker (SRB) command-line utilities, such as Srsync.
The output of these system jobs is captured and delivered nightly to a shared electronic mailbox. The mailbox is reviewed on a daily basis; this task belongs to the member of the ICPSR IT team who is currently on-call. When a report is missing or when a report indicates an error, the problem is escalated to someone who can diagnose and correct the problem. One common problem, for example, occurs when an object larger than 2GB enters Archival Storage and the SRB Srsync utility faults. (SRB limits objects to 2GB.) We then remove this object from the list of items to be synchronized with SRB.
Because the synchronization process is incremental, it has a very short duration. However, if we were to need to synchronize ALL content, this takes on the order of days or even weeks. For example, we recently synchronized a copy of our Access holdings to a computing instance residing in Amazon's EC2 EU-West region, and we found it took approximately one week to copy about 500GB. As another example, we recently synchronized a copy of our Archival Storage (which is much larger than the Access collection) to a system, which like ICPSR and the University of Michigan, is connected to Internet2's Abilene network, and that took far less time.
Wednesday, October 14, 2009
TRAC: C1.3: Managing all objects
C1.3 Repository manages the number and location of copies of all digital objects.
The repository system must be able to identify the number of copies of all stored digital objects, and the location of each object and their copies. This applies to what are intended to be identical copies, not versions of objects or copies. The location must be described such that the object can be located precisely, without ambiguity. It can be an absolute physical location or a logical location within a storage media or a storage subsystem. One way to test this would be to look at a particular object and ask how many copies there are, what they are stored on, and where they are. A repository can have different policies for different classes of objects, depending on factors such as the producer, the information type, or its value. Some repositories may have only one copy (excluding backups) of everything, stored in one place, though this is definitely not recommended. There may be additional identification requirements if the data integrity mechanisms use alternative copies to replace failed copies.
Evidence: random retrieval tests; system test; location register/log of digital objects compared to the expected number and location of copies of particular objects.
Our story here is a mixed bag of successes and barriers.
For the master copy of any object we can easily and quickly specify its location. And for the second (tape) copy, we too can easily specify the location as long as we're not too specific. For example, we can point to the tape library and say, "It's in there." And, of course, with a little more work, we can use our tape management system to point us to the specific tape, and the location on that tape. Maintaining this information outside of the tape management system would be expensive, and it's not clear if there would be any true benefit.
The location of other copies can be derived easily, but those specific locations are not recorded in a database. For example, let's say that the master copy of every original deposit we have is stored in a filesystem hierarchy like /archival-storage/deposits/deposit-id/. And let's say that on a daily basis we synchronize that content via rsync to an off-site location, say, remote-location.icpsr.umich.edu:/archival-storage/deposits/deposit-id/. And let's also say that someone reviews the output of the rsync run on a daily basis, and also performs a random spot-check on an irregular basis.
In this scenario we might have a large degree of confidence that we could find a copy of any given deposit on that off-site location. We know it's there because rsync told us it put it there. But we don't have a central catalog that says that deposit #1234 is stored under /archival-storage/deposits/1234, on tape, and at remote-site.icpsr.umich.edu/archival-storage/deposits/1234. One could build exactly such a catalog, of course, and then create the process to keep it up to date, but would it have much value? What if all we did was tell a wrapper around rsync to capture the output and update the catalog?
Probably not.
And so if we interpret the TRAC requirement to build a location register to mean that we should have a complete, enumerated list of each and every copy, then we don't do so well here. But if we interpret the requirement to mean that we can find a copy by looking on a list (i.e., the catalog proper) or look at a rule (i.e., if the master copy is in location x, then two other copies can be found by applying functions f(x) and g(x)), then we're doing pretty well after all.
Limitations in storage systems also add complexity. For instance, I was once looking at Amazon's S3 as a possible location for items in archival storage. But S3 doesn't let me have objects bigger than 5GB, and since I sometimes have very large files, this means that the record-keeping would be even more complicated. For an object with name X, you can find it in this S3 bucket, unless it is bigger than 5GB, in which case you need to look for N different objects and join them together. Ick.
The repository system must be able to identify the number of copies of all stored digital objects, and the location of each object and their copies. This applies to what are intended to be identical copies, not versions of objects or copies. The location must be described such that the object can be located precisely, without ambiguity. It can be an absolute physical location or a logical location within a storage media or a storage subsystem. One way to test this would be to look at a particular object and ask how many copies there are, what they are stored on, and where they are. A repository can have different policies for different classes of objects, depending on factors such as the producer, the information type, or its value. Some repositories may have only one copy (excluding backups) of everything, stored in one place, though this is definitely not recommended. There may be additional identification requirements if the data integrity mechanisms use alternative copies to replace failed copies.
Evidence: random retrieval tests; system test; location register/log of digital objects compared to the expected number and location of copies of particular objects.
Our story here is a mixed bag of successes and barriers.
For the master copy of any object we can easily and quickly specify its location. And for the second (tape) copy, we too can easily specify the location as long as we're not too specific. For example, we can point to the tape library and say, "It's in there." And, of course, with a little more work, we can use our tape management system to point us to the specific tape, and the location on that tape. Maintaining this information outside of the tape management system would be expensive, and it's not clear if there would be any true benefit.
The location of other copies can be derived easily, but those specific locations are not recorded in a database. For example, let's say that the master copy of every original deposit we have is stored in a filesystem hierarchy like /archival-storage/deposits/deposit-id/
In this scenario we might have a large degree of confidence that we could find a copy of any given deposit on that off-site location. We know it's there because rsync told us it put it there. But we don't have a central catalog that says that deposit #1234 is stored under /archival-storage/deposits/1234, on tape, and at remote-site.icpsr.umich.edu/archival-storage/deposits/1234. One could build exactly such a catalog, of course, and then create the process to keep it up to date, but would it have much value? What if all we did was tell a wrapper around rsync to capture the output and update the catalog?
Probably not.
And so if we interpret the TRAC requirement to build a location register to mean that we should have a complete, enumerated list of each and every copy, then we don't do so well here. But if we interpret the requirement to mean that we can find a copy by looking on a list (i.e., the catalog proper) or look at a rule (i.e., if the master copy is in location x, then two other copies can be found by applying functions f(x) and g(x)), then we're doing pretty well after all.
Limitations in storage systems also add complexity. For instance, I was once looking at Amazon's S3 as a possible location for items in archival storage. But S3 doesn't let me have objects bigger than 5GB, and since I sometimes have very large files, this means that the record-keeping would be even more complicated. For an object with name X, you can find it in this S3 bucket, unless it is bigger than 5GB, in which case you need to look for N different objects and join them together. Ick.
Wednesday, October 7, 2009
TRAC: C1.2: Backup infrastructure
C1.2 Repository ensures that it has adequate hardware and software support for backup functionality sufficient for the repository’s services and for the data held, e.g., metadata associated with access controls, repository main content.
The repository needs to be able to demonstrate the adequacy of the processes, hardware and software for its backup systems. Some will need much more elaborate backup plans than others.
Evidence: Documentation of what is being backed up and how often; audit log/inventory of backups; Trustworthy Repositories Audit & Certification: Criteria and Checklist validation of completed backups; disaster recovery plan—policy and documentation; “firedrills”—testing of backups; support contracts for hardware and software for backup mechanisms.
ICPSR has extensive documentation and infrastructure to support its core access functions even when a catastrophic failure disables its primary location in Ann Arbor, Michigan. The documentation - planning documents and instructions - reside in a Google Group, and all members of the IT team, and two of ICPSR's senior staff outside of IT are members of the group. The process has been used twice in 2009, once as a test, and once when the Ann Arbor site suffered a power failure.
ICPSR has a less well documented, but fairly prosaic, backup solution in place. All non-ephemeral content at ICPSR resides on a large Network Attached Storage (NAS) appliance. The IT team has configured the NAS to "checkpoint" each filesystem once per day, and each checkpoint is retained for 30 days. Checkpoints provide a read-only, self-serve backup system for those instances where a member of the staff has inadvertently damaged or destroyed non-archival content.
Further, we write all filesystems to a tape library, which is located in a different machine room than the NAS. Every two weeks tapes are removed from the tape library, and stored in yet a different building. We retain the last four weekly backups, and the last twelve monthly backups. The system is exercised on an infrequent, but regular basis when we restore files that were damaged or destroyed beyond the thirty day checkpoint window.
Finally, unlike "working" files where all copies reside locally, and where we retain only one year of content, our archival storage solution consists of copies in at least four locations. The master copy (1) is on the NAS; a copy (2) is written to tape each month; a copy (3) is synchronized daily with the San Diego Supercomputer Center's storage grid; and, a copy (4) is synchronized daily with the MATRIX Center at Michigan State University. Furthermore, archival content collected prior to 2009 has also been copied into the Chronopolis project storage grid, which adds two additional copies.
One area with room for improvement would be regular "fire drills" where we would attempt to retrieve a random number of random objects from an arbitrarily selected archival storage location.
Monday, September 28, 2009
Trustworthy Repositories Audit & Certification: Criteria and Checklist:
We're working through the Center for Research Libraries (CRL) Trustworthy Repositories
Audit & Certification (TRAC) Criteria and Checklist. My primary responsibility is to review Section C, which focuses on technology. Having looked through the TRAC document several times now, my sense is that ICPSR has a pretty good story to tell, and the main task is to take the time to tell the story. I thought it might be interesting to begin a series of blog posts, one per item from Section C in the TRAC checklist.
I have three goals for the posts:
And if you're working through TRAC yourself, I'd love to hear any advice you have (either as Comments here or via email).
Audit & Certification (TRAC) Criteria and Checklist. My primary responsibility is to review Section C, which focuses on technology. Having looked through the TRAC document several times now, my sense is that ICPSR has a pretty good story to tell, and the main task is to take the time to tell the story. I thought it might be interesting to begin a series of blog posts, one per item from Section C in the TRAC checklist.
I have three goals for the posts:
- Tell ICPSR's story for each item so that we have transparent evidence and documentation in each area
- Share ICPSR's evidence and documentation for others who are working through the TRAC document, or who are thinking about starting; I think it's often easier to work with something that someone has already begun v. starting from scratch
- Solicit input from others who have completed TRAC certification: What have we missed?
And if you're working through TRAC yourself, I'd love to hear any advice you have (either as Comments here or via email).
Subscribe to:
Posts (Atom)