One of the most challenging aspects of building our Secure Data Environment (SDE) for managing social science research data was redesigning our workflow.
In the past our environment was quite open, and so the workflow did not need to be concerned with certain aspects of access control. For example, if the workflow required an ICPSR data manager to send an email to to the original depositor, and the data manager wanted to include some content that was cut and pasted from the dataset, they would have been able to do that at any point in the process without any special actions. But, in the SDE we have disabled email, and we tightly control how material leaves the SDE, so this sort of free access is no longer available. And so the workflow had to change.
Content arrives at ICPSR through our Deposit Form web application. In brief this is how a depositor transfers content to us and grants us non-exclusive access to manage and share the content. (They can also add descriptive metadata to the content too.) The Deposit Form runs on our public-facing web server, which, of course, does not reside within our SDE.
One change we made, therefore, was to encrypt all content as it arrives. This means that the content isn't available in the clear - even accidentally - on our public web server. Next, an automated job runs on a regular, frequent basis, "sweeping" content from the public web server to the SDE. Once it arrives within the SDE we decrypt the content so that the data manager has easy access to the materials.
Data managers use another web application called the Deposit Viewer to view and manage deposits, and while they can view metadata about the deposit from the desktop and the SDE, they can only download the deposited files from within the SDE. This gives them the convenience of checking on deposit status, for example, from either environment, but ensures that the files do not leave the secure environment accidentally.
All data management functions take place within the SDE. A data manager may move content from the SDE to the outside world, but the transfer takes place via a software airlock. The airlock tracks what has been moved, who has moved it, and requires a supervisor to approve the transfer.
Once the data manager has completed all data processing and quality control, s/he then uses a set of utilities to generate the "ready to go" formats that we distribute via our web site and to release the materials both to the web site and to our archival storage fabric. Like the airlock process above, this step tracks who did what and when and where, and also requires management approval. The archival copies remain within the SDE, and the public-use, ready-to-go files move to our web site. Ensuring that key software systems have access to push files out of the SDE, but ensuring that staff do not, also required a few changes to our workflow.
Showing posts with label sde. Show all posts
Showing posts with label sde. Show all posts
Monday, August 8, 2011
Monday, August 1, 2011
ICPSR's Secure Data Environment (SDE) - The Virtual Desktop
A key feature of our Secure Data Environment (SDE), the computing environment in which ICPSR staff manage and process research data and documentation, is that it doesn't exist. At least not in the material world.
The SDE makes use of a University of Michigan system called the Virtual Desktop Infrastructure (VDI) service. According to the U-M web site the service is geared to lowering support costs for departments, and we have found that it is indeed a bit easier to manage virtual machines than physical machines. However, the real selling point for us was that we could centralize access to confidential (and potentially confidential) data in a single space that we could secure, manage, grow, shrink, etc. easily.
We restrict network access to our portion of the VDI using the Virtual Firewall service that I described in an earlier post. That limits the number of potential intruders dramatically. (Anyone who runs a server which is accessible via ssh from the general Internet will know what I mean.) We use University of Michigan-assigned credentials to grant login access to our pool of virtual machines. And because U-M is able to provision credentials from colleagues and associates who are outside of the U-M, we can also grant access to others as needed.
We've found the VDI service itself to be pretty solid overall, and since most of our use falls during the typical workday, we do not find maintenance windows during off-hours or the weekend to be terribly troublesome. There are a few things we'd like to do that are not part of the existing VDI service, and we've found the U-M to be a partner willing to work with us. For example, for a certain class of access we would like to use two-factor authentication and require someone to also enter a one-time passcode from a key fob. That isn't built into today's VDI service, but it may be available in the future.
The SDE makes use of a University of Michigan system called the Virtual Desktop Infrastructure (VDI) service. According to the U-M web site the service is geared to lowering support costs for departments, and we have found that it is indeed a bit easier to manage virtual machines than physical machines. However, the real selling point for us was that we could centralize access to confidential (and potentially confidential) data in a single space that we could secure, manage, grow, shrink, etc. easily.
We restrict network access to our portion of the VDI using the Virtual Firewall service that I described in an earlier post. That limits the number of potential intruders dramatically. (Anyone who runs a server which is accessible via ssh from the general Internet will know what I mean.) We use University of Michigan-assigned credentials to grant login access to our pool of virtual machines. And because U-M is able to provision credentials from colleagues and associates who are outside of the U-M, we can also grant access to others as needed.
We've found the VDI service itself to be pretty solid overall, and since most of our use falls during the typical workday, we do not find maintenance windows during off-hours or the weekend to be terribly troublesome. There are a few things we'd like to do that are not part of the existing VDI service, and we've found the U-M to be a partner willing to work with us. For example, for a certain class of access we would like to use two-factor authentication and require someone to also enter a one-time passcode from a key fob. That isn't built into today's VDI service, but it may be available in the future.
Wednesday, July 27, 2011
ICPSR's Secure Data Environment (SDE) - The Storage
To implement its Secure Data Environment (SDE) ICPSR replaces an aging storage array with two newer systems. The idea was to use a physical separation between storage devices to help make our data management environment for secure.
In many ways the physical separation of systems is overkill. There isn't much to be gained at the level of the individual data manager or data handler using two separate storage arrays rather than a single array that has been partitioned into two logical arrays. However, the real value comes, I think, in protecting the IT team from itself. And I include myself in that statement too.
It would be easy to have a single physical storage array with multiple virtual storage servers. That is, one can easily create a chunk of storage -- say, a filesystem called /secretStuff -- and then make it available to one virtual storage server, but not another. And by using a firewall one could then ensure that people working within the SDE would be able to access /secretStuff, and people working outside the SDE would not.
The risk, however, is that someone creates a filesystem like /secretStuff, and then accidentally makes it available across ALL virtual storage servers. And therefore, not only are SDE systems able to read files in /secretStuff, but the content also becomes, inadvertently, available to the web server too. That's not good.
We therefore placed one of our physical storage arrays on our Private network. Since this network uses private IPv4 address space, this made the array largely invisible to much of the Internet. Further, the firewall rules for the Private network are very, very restrictive, and access is available only within the SDE, and to a small number of developer workstations (and then only for ssh access). We use this storage array for all of our content which is confidential and content which might be confidential.
Our second physical storage array resides on our Semi-Private network. This too uses private IPv4 address space, and therefore is only accessible to machines within the University of Michigan enterprise network. We allow access via protocols like NFS and CIFS to the storage array within the University of Michigan environment, and we further manage detailed access control lists for individual NFS exports. The array provides storage to our web server and other public-facing machines, and also serves as the storage back-end for desktop computers. For example, if you work at ICPSR, then your My Documents folder maps to this array.
The biggest hurdle in replacing one old storage array with two new systems was separating peoples' storage into two categories: public stuff that they would need to access from their desktop (e.g., stuff they may want to email to someone), and private stuff that they would need to access from within the SDE (e.g., data files and documentation). This required a significant investment of time from everyone at ICPSR, and especially the IT staff. I think I spent about 10-15 weekends in the office between February and May, moving content between systems, making sure that drive mappings still worked, double-checking checkpoint schedules and backups, etc.
The separation seems to have gone relatively smoothly, at least from the perspective of the IT team. There were no major snafus during the transition, and the number of trouble tickets was relatively low.
The separation did mean that we needed to create some new systems - and tweak existing systems - to create mechanisms so that content could move between the two systems, but in a controlled way that could be audited later. I'll describe changes we made to our deposit and release systems in my next post, and will also describe our new data airlock system.
In many ways the physical separation of systems is overkill. There isn't much to be gained at the level of the individual data manager or data handler using two separate storage arrays rather than a single array that has been partitioned into two logical arrays. However, the real value comes, I think, in protecting the IT team from itself. And I include myself in that statement too.
It would be easy to have a single physical storage array with multiple virtual storage servers. That is, one can easily create a chunk of storage -- say, a filesystem called /secretStuff -- and then make it available to one virtual storage server, but not another. And by using a firewall one could then ensure that people working within the SDE would be able to access /secretStuff, and people working outside the SDE would not.
The risk, however, is that someone creates a filesystem like /secretStuff, and then accidentally makes it available across ALL virtual storage servers. And therefore, not only are SDE systems able to read files in /secretStuff, but the content also becomes, inadvertently, available to the web server too. That's not good.
We therefore placed one of our physical storage arrays on our Private network. Since this network uses private IPv4 address space, this made the array largely invisible to much of the Internet. Further, the firewall rules for the Private network are very, very restrictive, and access is available only within the SDE, and to a small number of developer workstations (and then only for ssh access). We use this storage array for all of our content which is confidential and content which might be confidential.
Our second physical storage array resides on our Semi-Private network. This too uses private IPv4 address space, and therefore is only accessible to machines within the University of Michigan enterprise network. We allow access via protocols like NFS and CIFS to the storage array within the University of Michigan environment, and we further manage detailed access control lists for individual NFS exports. The array provides storage to our web server and other public-facing machines, and also serves as the storage back-end for desktop computers. For example, if you work at ICPSR, then your My Documents folder maps to this array.
The biggest hurdle in replacing one old storage array with two new systems was separating peoples' storage into two categories: public stuff that they would need to access from their desktop (e.g., stuff they may want to email to someone), and private stuff that they would need to access from within the SDE (e.g., data files and documentation). This required a significant investment of time from everyone at ICPSR, and especially the IT staff. I think I spent about 10-15 weekends in the office between February and May, moving content between systems, making sure that drive mappings still worked, double-checking checkpoint schedules and backups, etc.
The separation seems to have gone relatively smoothly, at least from the perspective of the IT team. There were no major snafus during the transition, and the number of trouble tickets was relatively low.
The separation did mean that we needed to create some new systems - and tweak existing systems - to create mechanisms so that content could move between the two systems, but in a controlled way that could be audited later. I'll describe changes we made to our deposit and release systems in my next post, and will also describe our new data airlock system.
Wednesday, July 20, 2011
ICPSR's Secure Data Environment (SDE) - The Network
By the end of 2009 ICPSR's data network looked very much like it had in 1999. It consisted on a single virtual local area network (VLAN) that was home to a handful of IPv4 address blocks. The number of blocks had grown over the decade as ICPSR hosted more equipment in its machine room, such as servers running Stanford's LOCKSS software and Harvard's DataVerse Network (DVN) system. Also, as the ICPSR Summer Program expanded, the number of guest machines and lab machines expanded, and this too drove the acquisition of more network blocks.
The blocks were in public IPv4 space, and therefore in principle, any machine on ICPSR's VLAN could reach any location on the Internet, and vice-versa. In practice some simple devices, such as printers and network switches, used private IPv4 address space, routed only within the University of Michigan. This is a fairly common practice, of course, to conserve IPv4 address space and to protect (somewhat) systems from network-based attacks.
At that time we also made use of simple Cisco access-list rules which acted as a very primitive firewall. The campus network administrators did this for us, but somewhat grudgingly since it was a non-standard practice for them, and made ICPSR's data networking equipment more difficult to manage. And it was also less than ideal for us too since we didn't have regular access to the data networking switches and routers, and so never knew exactly how they were configured at any given time.
So in a nutshell we have a very flat, very basic, and very open network.
This all changed in early 2010 when we started using a new product/service available from the campus network administrators called the Virtual Firewall (VFW). This is based upon a Checkpoint product which (I believe) is often used with commercial network providers who resell network blocks to smaller companies. Within the University of Michigan it is used by departments and organizations like ICPSR who would like all of the benefits of having a firewall, but who lack the resources and expertise to manage all of the infrastructure. In many ways it is the "cloud version" of a firewall, giving one access to the tools to manage access controls, but without the expense of managing the physical firewall itself. This has been an outstanding service.
In addition to using the new VFW we also partitioned our network into four (and later seven!) VLANs:
The Public VLAN uses public address space and is home to all of our public-facing infrastructure. For example, it is the home of our production web server, our authoritative DNS server, and special-purpose machines running LOCKSS, DVN, etc. Access into and out of this VLAN is relatively open, but we do restrict access to certain protocols for certain machines.
The Semi-Private VLAN uses private address space and is home to all of our non-public, but non-sensitive systems such as desktop computers, printers, and so on. We make relatively light use of the VFW for this VLAN, and outbound access uses NAT so that people can reach the Internet. One of our two EMC NS-120 NAS units also resides on this VLAN.
The Private VLAN also uses private address space, and it contains all of our internal data management and archival storage systems. Our second EMC NS-120 NAS holds this content. This VLAN is heavily controlled via the VFW, and both inbound and outbound access are heavily restricted.
Finally, we use a different VLAN for a pool of virtual workstations that our data managers use to "process" research data and documentation. Like the Private VLAN, this VLAN makes extensive use of the VFW for access control. In many ways the access controls of this VLAN are similar to the Private VLAN, but we have found it useful to use two different network segments, one for the individual virtual workstations and one for the back-end systems.
The blocks were in public IPv4 space, and therefore in principle, any machine on ICPSR's VLAN could reach any location on the Internet, and vice-versa. In practice some simple devices, such as printers and network switches, used private IPv4 address space, routed only within the University of Michigan. This is a fairly common practice, of course, to conserve IPv4 address space and to protect (somewhat) systems from network-based attacks.
At that time we also made use of simple Cisco access-list rules which acted as a very primitive firewall. The campus network administrators did this for us, but somewhat grudgingly since it was a non-standard practice for them, and made ICPSR's data networking equipment more difficult to manage. And it was also less than ideal for us too since we didn't have regular access to the data networking switches and routers, and so never knew exactly how they were configured at any given time.
So in a nutshell we have a very flat, very basic, and very open network.
This all changed in early 2010 when we started using a new product/service available from the campus network administrators called the Virtual Firewall (VFW). This is based upon a Checkpoint product which (I believe) is often used with commercial network providers who resell network blocks to smaller companies. Within the University of Michigan it is used by departments and organizations like ICPSR who would like all of the benefits of having a firewall, but who lack the resources and expertise to manage all of the infrastructure. In many ways it is the "cloud version" of a firewall, giving one access to the tools to manage access controls, but without the expense of managing the physical firewall itself. This has been an outstanding service.
In addition to using the new VFW we also partitioned our network into four (and later seven!) VLANs:
- Public
- Semi-Private
- Private
- Virtual desktops
- Virtual Summer Program
- Virtual Data Enclave
- Virtual Testing and Evaluation
The Public VLAN uses public address space and is home to all of our public-facing infrastructure. For example, it is the home of our production web server, our authoritative DNS server, and special-purpose machines running LOCKSS, DVN, etc. Access into and out of this VLAN is relatively open, but we do restrict access to certain protocols for certain machines.
The Semi-Private VLAN uses private address space and is home to all of our non-public, but non-sensitive systems such as desktop computers, printers, and so on. We make relatively light use of the VFW for this VLAN, and outbound access uses NAT so that people can reach the Internet. One of our two EMC NS-120 NAS units also resides on this VLAN.
The Private VLAN also uses private address space, and it contains all of our internal data management and archival storage systems. Our second EMC NS-120 NAS holds this content. This VLAN is heavily controlled via the VFW, and both inbound and outbound access are heavily restricted.
Finally, we use a different VLAN for a pool of virtual workstations that our data managers use to "process" research data and documentation. Like the Private VLAN, this VLAN makes extensive use of the VFW for access control. In many ways the access controls of this VLAN are similar to the Private VLAN, but we have found it useful to use two different network segments, one for the individual virtual workstations and one for the back-end systems.
Subscribe to:
Posts (Atom)