Qualys browser checker

Ever since the recent craziness with vulnerabilities in Java plugins, I've making a concious effort to use Qualys's browser checker - https://browsercheck.qualys.com/ - on a routine basis both at home and at the office.

Installing the tool in your browser is very easy, and the service is free and painless to use.  I have been using it to both to determine if my current browser and plugins are up to date, and also to identify plugins that are installed and enabled, but which I don't really need or use (e.g., Silverlight which I often disable for long stretches at a time).

Qualys generates a nice report

like the one above to let you know if everything is up-to-date.

FLAME, no, not that one

Asmat Noori, who leads the IT Operations team at ICPSR, pointed me to this the other day:

http://www.msnbc.msn.com/id/47590214/ns/technology_and_science-security/#.T8Tg67_wPx5

Unlike our FLAME, which we hope will be a benevolent collection of built and borrowed systems, this:

Flame appears poised to go down in history as the third major cyber weapon uncovered after Stuxnet and its data-stealing cousin Duqu, named after the Star Wars villain.

So I think it wins the award for the Dangerous-but-kind-of-cool Flame.

OpenSSL FIPS 140-2 and RHEL 5

Since I often use the Internet to find guidance and answers to questions, I thought I'd add my own small contribution back to the community.

We found ourselves needing to build a FIPS 140-2 compliant version of the OpenSSL openssl command-line utility recently.  A good starting point is the OpenSSL FIPS 140-2 User Guide.  It contains instructions for where to find OpenSSL source, and importantly, instructions for verifying the integrity of the distribution, and this is a necessary component of building a FIPS 140-2 openssl.

I worked through the guide through nearly page 23.  However, when I reached section 4.2.1, things started to go wrong for me.  I was able to run config with no problem, but the make failed with an error about not having a target for fipscannister.o.

I then found a very helpful bit of advice in a Google Group post about OpenSSL and FIPS 140-2.  But that advice didn't quite match our environment (it was Ubuntu, and we run RHEL).  So here are my directions for building an openssl utility on RHEL 5.

First, in our case we downloaded and verified the integrity of openss-1.2.3 using the directions from the guide above.  Here are the results in a little sandbox:

batch-bryan:; pwd

/tmp/openssl-fips

batch-bryan:; ls -R

.:

lib/  src/

./lib:

./src:

openssl-fips-1.2.3.tar

Now, we head into the src directory, and unpackage the tarball:

batch-bryan:; cd src

/tmp/openssl-fips/src

batch-bryan:; tar xf openssl-fips-1.2.3.tar

And then configure things to build the FIPS canister and utility:

batch-bryan:; cd openssl-fips-1.2.3

/tmp/openssl-fips/src/openssl-fips-1.2.3

batch-bryan:; ./config fipscanisterbuild --prefix="/tmp/openssl-fips"

Now to build and install:

batch-bryan:; make

batch-bryan:; make install

And there it is:

batch-bryan:; ls /tmp/openssl-fips/lib

engines/             fips_premain.c       libcrypto.so@        libssl.so@

fipscanister.o       fips_premain.c.sha1  libcrypto.so.0.9.8*  libssl.so.0.9.8*

fipscanister.o.sha1  libcrypto.a          libssl.a             pkgconfig/

And there is the utility:

batch-bryan:; cd /tmp/openssl-fips/bin

/tmp/openssl-fips/bin

batch-bryan:; ls

c_rehash*  openssl*

batch-bryan:; setenv OPENSSL_FIPS 1

batch-bryan:; ./openssl version

OpenSSL FIPS Object Module v1.2

But if we try the stock one:

batch-bryan:; /usr/bin/openssl version

13789:error:2D06C06E:FIPS routines:FIPS_mode_set:fingerprint does not match:fips.c:493:

batch-bryan:; unsetenv OPENSSL_FIPS

batch-bryan:; /usr/bin/openssl version

OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008

Not so much.

To build and install the FIPS 140-2 compliant version of OpenSSL in a more "real" location than a sandbox, just change the value used in the prefix variable used in the config invocation above.

Convergence

The always entertaining and highly informative Moxie Marlinspike gave a very interesting talk at BlackHat USA 2001 about SSL.  This is the technology that (in theory) secures our communication channels on the Internet, keeping information like credit card numbers out of the hands of the bad guys.  I've seen past talks by Moxie where he describes the many flaws with SSL, but in this talk he introduces a new solution called Convergence.

The talk is fascinating, and I highly recommend watching it.  (It's on YouTube.)  It's about 45 minutes long, so enjoy over lunch.

In brief, Moxie cites two problems with the current SSL model, which requires all of us to trust Certificate Authorities (CAs), which have been hacked with increasing frequency, and which have also demonstrated drunk and disorderly behavior at times.  One, we have to trust them forever.  Two, there is no reasonably way to change who you trust.  For example, if one decided that Comodo (one of the largest CAs) just could not be trusted any longer, one could deleted Comodo from his/her browser's "trust database."  But doing this would make a large number of Internet web sites (20%) unusable.

Convergence replaces CAs with one or more self-selected "notaries" each of which can use a different method to ascertain whether a certificate is valid, including a self-signed certificate.  One may also use a "bounce notary" to separate those that know who you are from those that know where you are browsing.

ICPSR's Secure Data Environment (SDE) - The Network

By the end of 2009 ICPSR's data network looked very much like it had in 1999.  It consisted on a single virtual local area network (VLAN) that was home to a handful of IPv4 address blocks.  The number of blocks had grown over the decade as ICPSR hosted more equipment in its machine room, such as servers running Stanford's LOCKSS software and Harvard's DataVerse Network (DVN) system.  Also, as the ICPSR Summer Program expanded, the number of guest machines and lab machines expanded, and this too drove the acquisition of more network blocks.

The blocks were in public IPv4 space, and therefore in principle, any machine on ICPSR's VLAN could reach any location on the Internet, and vice-versa.  In practice some simple devices, such as printers and network switches, used private IPv4 address space, routed only within the University of Michigan.  This is a fairly common practice, of course, to conserve IPv4 address space and to protect (somewhat) systems from network-based attacks.

At that time we also made use of simple Cisco access-list rules which acted as a very primitive firewall.  The campus network administrators did this for us, but somewhat grudgingly since it was a non-standard practice for them, and made ICPSR's data networking equipment more difficult to manage.  And it was also less than ideal for us too since we didn't have regular access to the data networking switches and routers, and so never knew exactly how they were configured at any given time.

So in a nutshell we have a very flat, very basic, and very open network.

This all changed in early 2010 when we started using a new product/service available from the campus network administrators called the Virtual Firewall (VFW).  This is based upon a Checkpoint product which (I believe) is often used with commercial network providers who resell network blocks to smaller companies.  Within the University of Michigan it is used by departments and organizations like ICPSR who would like all of the benefits of having a firewall, but who lack the resources and expertise to manage all of the infrastructure.  In many ways it is the "cloud version" of a firewall, giving one access to the tools to manage access controls, but without the expense of managing the physical firewall itself.  This has been an outstanding service.

In addition to using the new VFW we also partitioned our network into four (and later seven!) VLANs:

1. Public
2. Semi-Private
3. Private
4. Virtual desktops
5. Virtual Summer Program
6. Virtual Data Enclave
7. Virtual Testing and Evaluation

I'm going to skip discussion of the last three VLANs for now to focus on the first four.

The Public VLAN uses public address space and is home to all of our public-facing infrastructure.  For example, it is the home of our production web server, our authoritative DNS server, and special-purpose machines running LOCKSS, DVN, etc.  Access into and out of this VLAN is relatively open, but we do restrict access to certain protocols for certain machines.

The Semi-Private VLAN uses private address space and is home to all of our non-public, but non-sensitive systems such as desktop computers, printers, and so on.  We make relatively light use of the VFW for this VLAN, and outbound access uses NAT so that people can reach the Internet.  One of our two EMC NS-120 NAS units also resides on this VLAN.

The Private VLAN also uses private address space, and it contains all of our internal data management and archival storage systems.  Our second EMC NS-120 NAS holds this content.  This VLAN is heavily controlled via the VFW, and both inbound and outbound access are heavily restricted.

Finally, we use a different VLAN for a pool of virtual workstations that our data managers use to "process" research data and documentation.  Like the Private VLAN, this VLAN makes extensive use of the VFW for access control.  In many ways the access controls of this VLAN are similar to the Private VLAN, but we have found it useful to use two different network segments, one for the individual virtual workstations and one for the back-end systems.

Disaster Preparedness Planning - Zombie Apocalypse

Now this is the kind of disaster planning that I could really embrace:

Uncle Sam wants YOU to be prepared for a zombie apocalypse.

The Centers for Disease Control and Prevention, known best for stamping out health threats like Ebola and E. coli, is now advising people how to prepare for a zombie invasion.

A blog post by Assistant Surgeon General Ali Khan instructs readers to stock up on food and water, not to mention first aid supplies (“Although you’re a goner if a zombie bites you, you can use these supplies to treat basic cuts and lacerations that you might get during a tornado or hurricane,” the agency says).

From the WSJ Health Blog

Whenever we do this sort of thing, it's always "What if we lose power?" or "What if there is a really heavy snow?"

Harummph.

I say, "What if flesh-eating zombies attack, and they are looking for a side order of social science datasets?"

P3P - Platform for Privacy Preferences Project

I came across an interesting paper on P3P (P3P is the Platform for Privacy Preferences Project) which is a W3C standard for expressing the privacy policies of a web site. The paper is from the CMU CyLab, and can be found here (PDF format).

A primary user of P3P is the Internet Explorer browser. It uses a "short form" of the policy to make decisions about whether a web site meets the security criteria one may set in the browser. Since most people never bother to configure different security levels for different sites, in practice any P3P descriptions that match "Medium security" will pass the security check.

The brief summary of the paper is that many of the top sites do not use P3P. Or, if they do use it, they make mistakes in the policy which will confuse browsers. And worse still, some sites seem to use P3P to actively trick browsers into thinking the site gathers no private information when it in fact does.

The paper is long, but many pages are part of an appendix. The main section of the paper is relatively short, well written, and is an interesting read.

ICPSR increases security of its web transactions

We'll be making a small, but important, configuration change on our web server this week. For a long time we've allowed so-called "weak" ciphers to be used with HTTP connections over SSL (aka HTTPS). This was good for web site visitors who had very old browsers; so old that the browser did not support stronger SSL ciphers. But it is bad news for most of us who are running more recent software since it would allow one to use less robust encryption when exchanging content via HTTPS.

We've been running this newer configuration for many months on a web server we use for staging new content. The many browsers and platforms we use to test new web pages and software work well with this configuration, and so we've decided to move it into the production environment.

Wikipedia has a nice page that describes the technical details behind the various ciphers that are used with SSL (and its successor TLS).

Web sites, firewalls, and attackers

Some of you may have noticed that our web server was very slow, or even unresponsive, early in the evening (EDT) on Friday, August 6th. It was slow enough that I received a text from our automated monitoring system. (I was the on-call that week. Three of us take turns, and we hand off the responsibility on Fridays at noon.)

When I checked out our web server I noticed that the load was very high, and that we were receiving dozens of web requests per second from an address located in a broadband block in China. There's no way to know if it was a denial of service attack or something more innocent, like a misguided attempt to harvest some content from our web site. But whatever it was, it was not good.

We haven't invested much resource (yet) in understanding the ins and outs of Apache's modsecurity package, which is how some folks would guard against such attacks. The package has a reputation for complexity, and it isn't often that we find our web site having these sorts of problems. (I'd estimate that we see something like this once every two years or so since I arrived in 2002.)

We have, however, invested in new firewall technology. Once I knew the IP address of the "attacker" it was very easy to add a rule to our firewall blocking all traffic from that source. It took only a few minutes to create and deploy the rule, and the load on our web server immediately dropped off to its usual level. In the firewall logs I could see the "attack" continue for hours after I added the rule, but because of the new firewall rule, it caused no harm. Eventually the "attack" stopped, and the logs were clear by Monday.

ICPSR's network migration begins

Last month I wrote about ICPSR's new network topology in my TRAC series of posts. We started the move to the new network last week, and the early signs are good, and it is time for an update.

Here's a high-level architectural view of our future network on the left.

We're breaking the current network into three virtual networks, each with a series of firewall rules to protect the people, content, and systems behind the firewall.

Moving from left to right...

Our Public (green) network has the lowest level of security. It hosts our public-facing systems such as our public web server, www.icpsr.umich.edu, the authoritative DNS server for the icpsr.umich.edu domain (and many others!), a machine we use to mirror Web portals for partner organizations, and more. Machines on this network have publicly routable IP addresses, and there is generally free and open access between this network, its resources, and the Internet.

Our Semi-Private (purple) network has a higher level of security. It hosts the vast bulk of our network-attached devices: desktop workstations, laptops, printers, private-facing servers (like a DHCP server), and more. Machines on this network have addresses in private IP address space, and use NAT to reach the Internet. Since the addresses are not public, there is no inbound access to the network from outside the University of Michigan network (UMnet) except through Virtual Private Network technology. I'm typing this blog post on a machine connected to the Semi-Private network.

Our Private (red) network has the highest level of security. It hosts a very small number of systems, all of which are used by ICPSR data managers to process the data and documentation we receive from the research community and from government agencies. In general, there is no access between this network and any other network unless it has explicitly been granted by a firewall rule.

Access to this network flows largely through a Virtual Desktop Infrastructure (VDI) service (also red) that we are using. The VDI hosts a collection of virtual Windows 7 desktop systems which contain the same software used on physical desktop workstations at ICPSR. Once one has connected to the VDI, one may access resources on the Private network, such as newly deposited content, ICPSR archival holdings, etc. Access from the VDI to the desktop is also tightly controlled so that one may not, for example, cut-and-paste between the two. Essentially this is a virtual data enclave, but where the client is ICPSR staff, not external data analysts.