B6.2 Repository has implemented a policy for recording all access actions (includes requests, orders etc.) that meet the requirements of the repository and information producers/depositors.
A repository need only record the actions that meet the requirements of the repository and its information producers/depositors. This may mean that little or no information is recorded about access. That is acceptable if the repository can demonstrate that it does not need to do more. Some repositories may want information about what is being accessed, but not about the users. Others may need much more detailed information about access. A policy should be established and implemented that relates to demonstrable needs. Are these figures being monitored? Are statistics produced and made available?
Evidence: Access policies; use statements.
ICPSR collects a considerable amount of information about each access: who, what, when, and where (in terms of via which of the properties within the portal was the source). This allows ICPSR to assist users who are having access problems, and to produce summary reports for a member's Organizational Representative or a government agency which relies upon ICPSR to provide access to its content.
ICPSR collects even more information if the content is part of a restricted-use collection. In this case, a research plan, CV, data protection plan, and more are required.
Because a single delivery platform is serving so many different masters (the consortium of members; government agencies; individual depositors; etc), a single policy may not be particularly workable, unless it is necessarily open-ended and broad (e.g., "save as much information as you can since you never know what report you'll need to produce").
Friday, July 15, 2011
Wednesday, July 13, 2011
ICPSR's Secure Data Environment (SDE)
ICPSR has designed, built, and deployed what we call the Secure Data-processing Environment (SDE) over the past twelve months. This is a tightly managed, highly controlled environment in which many members of the ICPSR staff perform their day-to-day data management (data processing) work.
The main business requirement behind the SDE is that it should be difficult, if not impossible, for content to leak out without a member of the staff taking an explicit action, such as running a program which formally releases content on the web site and commits it to archival storage. For example, it should not be possible for someone to upload a data file into a web form, or to attach it to a piece of email.
The design called for many changes to ICPSR's technology infrastructure. We separated our storage into two pools - Private (accessed within the SDE) and Semi-Private (which is more accessible). We separated our network into three main virtual LANs - Private, Semi-Private, and Public. We also updated many, many software systems so that they would operate properly within the SDE. And we also changed processes to conform to the new business requirements. For example, if one process required a data processor to send an email containing a data file to someone else at ICPSR, we changed the process so that email was not required.
I'll post a series of articles over the next few weeks with more details about the SDE and its technology. This will include posts about how we separated storage; how we segmented the network; how we used virtualization technology to solve certain problems; how we changed key software systems; and, how the SDE changed business processes at ICPSR, and how it continues to do so even today.
The main business requirement behind the SDE is that it should be difficult, if not impossible, for content to leak out without a member of the staff taking an explicit action, such as running a program which formally releases content on the web site and commits it to archival storage. For example, it should not be possible for someone to upload a data file into a web form, or to attach it to a piece of email.
The design called for many changes to ICPSR's technology infrastructure. We separated our storage into two pools - Private (accessed within the SDE) and Semi-Private (which is more accessible). We separated our network into three main virtual LANs - Private, Semi-Private, and Public. We also updated many, many software systems so that they would operate properly within the SDE. And we also changed processes to conform to the new business requirements. For example, if one process required a data processor to send an email containing a data file to someone else at ICPSR, we changed the process so that email was not required.
I'll post a series of articles over the next few weeks with more details about the SDE and its technology. This will include posts about how we separated storage; how we segmented the network; how we used virtualization technology to solve certain problems; how we changed key software systems; and, how the SDE changed business processes at ICPSR, and how it continues to do so even today.
Monday, July 11, 2011
June 2011 deposits at ICPSR
June 2011 was a very busy time for our deposit system. The number of deposits was pretty typical, but the number of files was enormous.
In addition to the usual suspects like plain ASCII, SAS, SPSS, MS Word, PDF, we also have some of the usual problems, such as files being reported by the automated checker as containing C or C++ source code, when the truth is that they are likely text/plain instead.
One interesting data point is the pair of deposits that contain video files, and lots of them. Upon further review these appear to be vintage SPSS files for the IBM PC. Here's a string that appears in all of the files:
SPSS/PC+ System File Written by Data Entry II
and here is another one:
PCSPSS SYSTEM FILE. IBM PC DOS, SPSS/PC+ V3.0
From a timestamp located nearby, it looks like these files were from 1994. Or maybe they were moved from a mainframe to a PC in 1994?
And there are a few others on the list above that would benefit from some human scrutiny as well.
| # of files | # of deposits | File format |
| 2 | 1 | application/msaccess |
| 23 | 1 | application/msoffice |
| 165 | 23 | application/msword |
| 698 | 4 | application/octet-stream |
| 266 | 28 | application/pdf |
| 144 | 11 | application/vnd.ms-excel |
| 1 | 1 | application/vnd.ms-powerpoint |
| 14 | 2 | application/vnd.wordperfect |
| 141 | 1 | application/x-123 |
| 4 | 1 | application/x-arc011lzw |
| 25 | 1 | application/x-dbase |
| 23 | 1 | application/x-dosexec |
| 1 | 1 | application/x-empty |
| 1 | 1 | application/x-rar |
| 19 | 5 | application/x-sas |
| 1307 | 21 | application/x-spss |
| 10 | 4 | application/x-stata |
| 3 | 3 | application/x-zip |
| 20 | 9 | message/rfc8220117bit |
| 8 | 7 | text/html |
| 12 | 6 | text/plain; charset=iso-8859-1 |
| 10 | 6 | text/plain; charset=unknown |
| 4386 | 46 | text/plain; charset=us-ascii |
| 2 | 1 | text/plain; charset=utf-8 |
| 11 | 4 | text/rtf |
| 7 | 2 | text/x-c++; charset=us-ascii |
| 1 | 1 | text/x-c; charset=us-ascii |
| 1 | 1 | text/x-mail; charset=us-ascii |
| 1 | 1 | text/xml |
| 153 | 2 | video/unknown |
In addition to the usual suspects like plain ASCII, SAS, SPSS, MS Word, PDF, we also have some of the usual problems, such as files being reported by the automated checker as containing C or C++ source code, when the truth is that they are likely text/plain instead.
One interesting data point is the pair of deposits that contain video files, and lots of them. Upon further review these appear to be vintage SPSS files for the IBM PC. Here's a string that appears in all of the files:
SPSS/PC+ System File Written by Data Entry II
and here is another one:
PCSPSS SYSTEM FILE. IBM PC DOS, SPSS/PC+ V3.0
From a timestamp located nearby, it looks like these files were from 1994. Or maybe they were moved from a mainframe to a PC in 1994?
And there are a few others on the list above that would benefit from some human scrutiny as well.
Friday, July 8, 2011
TRAC: B6.1: Access and delivery options
B6.1 Repository documents and communicates to its designated community(ies) what access and delivery options are available.
Repository policies should document the various aspects of access to and delivery of the preserved information. Generally, the designated community(ies) should know the policies or at least the consequences of them. The users should know what they can ask for, when, and how, and what it costs, among other things. [See Appendix 6: Understanding Digital Repositories & Access Functionality for an in-depth review of digital repository access requirements.]
Repositories might have to deal with a single, homogeneous community or with multiple or disparate communities. Different policies might be needed for different communities as well as for different collection types.
Evidence: Public versions of access policies; delivery policies; fee policies.
Access is one of ICPSR's strong suits. Evidence to support this TRAC requirement can be found across many different pages on ICPSR's public web portal.
For example, if we take study 2999 (Israeli Election Study, 1999 - the first hit when searching for 'election' on the portal), the home page displays a section called Access Notes which makes it clear what it is possible to do with the content.
If one clicks through the link to download content, the next display makes it clear what formats are available.
And there are also pages describing how to become a consortium member, how much things cost if you are not a member, etc.
And if there are versions of content available in both public-use and restricted-use versions, the site also makes that clear.
Repository policies should document the various aspects of access to and delivery of the preserved information. Generally, the designated community(ies) should know the policies or at least the consequences of them. The users should know what they can ask for, when, and how, and what it costs, among other things. [See Appendix 6: Understanding Digital Repositories & Access Functionality for an in-depth review of digital repository access requirements.]
Repositories might have to deal with a single, homogeneous community or with multiple or disparate communities. Different policies might be needed for different communities as well as for different collection types.
Evidence: Public versions of access policies; delivery policies; fee policies.
Access is one of ICPSR's strong suits. Evidence to support this TRAC requirement can be found across many different pages on ICPSR's public web portal.
For example, if we take study 2999 (Israeli Election Study, 1999 - the first hit when searching for 'election' on the portal), the home page displays a section called Access Notes which makes it clear what it is possible to do with the content.
If one clicks through the link to download content, the next display makes it clear what formats are available.
And there are also pages describing how to become a consortium member, how much things cost if you are not a member, etc.
And if there are versions of content available in both public-use and restricted-use versions, the site also makes that clear.
Wednesday, July 6, 2011
ICPSR web portal availability in 2010-2011
It's that time again: the end of another fiscal year. And that means it is also time for my annual summary of ICPSR web portal availability.
All in all it was a pretty good year for ICPSR's production web portal. Our web portal hosts many different sites (ICPSR proper, NACJD, NACDA, SAMHDA, DSDR, CCEERC, the ICPSR Summer Program, and many more sites). We were able to exceed 99.75% availability most months, and only had two months (January and June 2011) where our level was a bit lower.
The main culprit of downtime throughout fiscal year 2011 was due to defects in software. As we have been retooling our technology environment from Perl and CGI scripts to Java applications, we have been making greater use of systems like Hibernate and Lucene. My sense is that we're relying more and more on open source middleware, and while that has the advantages of making it easier to develop software quickly, it also means that a problem in the underlying middleware can affect our overall availability. Some of this is due to buggy software; some is due to our learning curve on how to use the software properly; and, some of this is due to getting our arms around the optimal configuration and operation of these packages.
The January 2011 availability level - our lowest month of availability- was due largely to two problems. One was that we scheduled a maintenance window in our server room so that University of Michigan electricians could wire up a new "whole room" uninterruptible power supply, and this, of course, took our production web systems off-line. The other problem was that our regular synchronization process between our production systems and our cloud-based replica had failed in an unusual way that was difficult to detect at first. The database export/import had failed, but only partially, and that produced very odd behavior with our web portal. It took a significant amount of time to isolate the problem, and by the time we had a workaround deployed, the electricians had finished their work, and the production systems were back on-line.
 |
| The leftmost month above is July 2010 and the leftmost is Jun 2011. The vertical axis shows availability for each month in terms of a percentage. Our goal is to hit or exceed 99% availability each month. |
The main culprit of downtime throughout fiscal year 2011 was due to defects in software. As we have been retooling our technology environment from Perl and CGI scripts to Java applications, we have been making greater use of systems like Hibernate and Lucene. My sense is that we're relying more and more on open source middleware, and while that has the advantages of making it easier to develop software quickly, it also means that a problem in the underlying middleware can affect our overall availability. Some of this is due to buggy software; some is due to our learning curve on how to use the software properly; and, some of this is due to getting our arms around the optimal configuration and operation of these packages.
The January 2011 availability level - our lowest month of availability- was due largely to two problems. One was that we scheduled a maintenance window in our server room so that University of Michigan electricians could wire up a new "whole room" uninterruptible power supply, and this, of course, took our production web systems off-line. The other problem was that our regular synchronization process between our production systems and our cloud-based replica had failed in an unusual way that was difficult to detect at first. The database export/import had failed, but only partially, and that produced very odd behavior with our web portal. It took a significant amount of time to isolate the problem, and by the time we had a workaround deployed, the electricians had finished their work, and the production systems were back on-line.
Friday, July 1, 2011
TRAC: B5.4: Maintaining referential integrity
B5.4 Repository can demonstrate that referential integrity is maintained between all
archived objects (i.e., AIPs) and associated descriptive information.
Particular attention must be paid to operations that affect AIPs and their identifiers and how integrity is maintained during these operations. There may be times, depending on system design, when the repository cannot demonstrate referential integrity because some system component is out of action. However, repositories, must implement procedures that let them know when referential integrity is temporarily broken and ensure that it can be restored.
Evidence: Log detailing ongoing monitoring/checking of referential integrity, especially following repair/modification of AIP; legacy descriptive metadata; persistence of identifier/locator; documented relationship between AIP and metadata; system documentation and technical architecture; process workflow documentation.
I've given this TRAC requirement considerable thought, and have searched the web for examples on how others have answered this requirement, but I still don't think I have a firm grasp on exactly what it means, and how I would demonstrate compliance.
It is certainly the case that we have a list of AIPs, and each item on this list contains both a pointer to the content which we're preserving in Archival Storage and metadata about the object. So is that referential integrity? Or is it necessary, but not sufficient, for referential integrity? I don't know.
In our case at ICPSR, we just don't modify or repair AIPs all that often. But if we did, would I need to maintain a log or ledger of the "before AIP" which maps it to the "after AIP"? And having that log would be my evidence of compliance?
I would be interested in hearing from others. How do you interpret this item? What is your evidence?
archived objects (i.e., AIPs) and associated descriptive information.
Particular attention must be paid to operations that affect AIPs and their identifiers and how integrity is maintained during these operations. There may be times, depending on system design, when the repository cannot demonstrate referential integrity because some system component is out of action. However, repositories, must implement procedures that let them know when referential integrity is temporarily broken and ensure that it can be restored.
Evidence: Log detailing ongoing monitoring/checking of referential integrity, especially following repair/modification of AIP; legacy descriptive metadata; persistence of identifier/locator; documented relationship between AIP and metadata; system documentation and technical architecture; process workflow documentation.
I've given this TRAC requirement considerable thought, and have searched the web for examples on how others have answered this requirement, but I still don't think I have a firm grasp on exactly what it means, and how I would demonstrate compliance.
It is certainly the case that we have a list of AIPs, and each item on this list contains both a pointer to the content which we're preserving in Archival Storage and metadata about the object. So is that referential integrity? Or is it necessary, but not sufficient, for referential integrity? I don't know.
In our case at ICPSR, we just don't modify or repair AIPs all that often. But if we did, would I need to maintain a log or ledger of the "before AIP" which maps it to the "after AIP"? And having that log would be my evidence of compliance?
I would be interested in hearing from others. How do you interpret this item? What is your evidence?
TRAC: B5.3: Creating referential integrity
B5.3 Repository can demonstrate that referential integrity is created between all archived objects (i.e., AIPs) and associated descriptive information.
Every AIP must have some descriptive information and all descriptive information must point to at least one AIP, such that the integrity can be validated. This should be an easy requirement to satisfy and is a prerequisite for the next one.
Evidence: Descriptive metadata; persistent identifier/locator associated with AIP; documented relationship between AIP and metadata; system documentation and technical architecture; process workflow documentation.
Our descriptive metadata resides in an Oracle database (and it also exported into DDI XML format).
We use one piece of this metadata (the fingerprint) on a regular basis to conduct fixity checks; this is how we validate integrity.
Every AIP must have some descriptive information and all descriptive information must point to at least one AIP, such that the integrity can be validated. This should be an easy requirement to satisfy and is a prerequisite for the next one.
Evidence: Descriptive metadata; persistent identifier/locator associated with AIP; documented relationship between AIP and metadata; system documentation and technical architecture; process workflow documentation.
Our descriptive metadata resides in an Oracle database (and it also exported into DDI XML format).
We use one piece of this metadata (the fingerprint) on a regular basis to conduct fixity checks; this is how we validate integrity.
Subscribe to:
Posts (Atom)