TRAC: C1.9: Change testing

C1.9 Repository has a process for testing the effect of critical changes to the system.

Changes to critical systems should be, where possible, pre-tested separately, the expected behaviors documented, and roll-back procedures prepared. After changes, the systems should be monitored for unexpected and unacceptable behavior. If such behavior is discovered the changes and their consequences should be reversed.

Whole-system testing or unit testing can address this requirement; complex safety-type tests are not required. Testing can be very expensive, but there should be some recognition of the fact that a completely open regime where no changes are ever evaluated or tested will have problems.

Evidence: Documented testing procedures; documentation of results from prior tests and proof of changes made as a result of tests.

---

When I arrived at ICPSR in 2002 the same systems were used for both production services and for new development. In fact, the total number of systems at ICPSR was very small; for example, a single machine was our production database machine, our shared "time sharing" machine for UNIX-based data processing, our anonymous ftp server, and pretty much everything else.

Our model is quite different these days. Most of our software development occurs on the desktop of the programmer, and new code is rolled out to a staging server for testing and evaluation. The level and intensity of the testing varies widely at ICPSR; my sense is that public-facing systems get the most evaluation, and internal-only systems receive very little. Nonetheless, unless the evaluation reveals serious flaws, the software is then rolled into production on a fixed deployment schedule. Because all software resides in a repository, we can back out changes easily if needed.

The last six software developers we've hired have all worked in the Java environment, and we're in the process of moving our Perl/CGI team to Java as well. My sense is that getting all of the major systems in Java will make it easier to use unit-testing tools, like JUnit for example.