Web sites, firewalls, and attackers

Some of you may have noticed that our web server was very slow, or even unresponsive, early in the evening (EDT) on Friday, August 6th. It was slow enough that I received a text from our automated monitoring system. (I was the on-call that week. Three of us take turns, and we hand off the responsibility on Fridays at noon.)

When I checked out our web server I noticed that the load was very high, and that we were receiving dozens of web requests per second from an address located in a broadband block in China. There's no way to know if it was a denial of service attack or something more innocent, like a misguided attempt to harvest some content from our web site. But whatever it was, it was not good.

We haven't invested much resource (yet) in understanding the ins and outs of Apache's modsecurity package, which is how some folks would guard against such attacks. The package has a reputation for complexity, and it isn't often that we find our web site having these sorts of problems. (I'd estimate that we see something like this once every two years or so since I arrived in 2002.)

We have, however, invested in new firewall technology. Once I knew the IP address of the "attacker" it was very easy to add a rule to our firewall blocking all traffic from that source. It took only a few minutes to create and deploy the rule, and the load on our web server immediately dropped off to its usual level. In the firewall logs I could see the "attack" continue for hours after I added the rule, but because of the new firewall rule, it caused no harm. Eventually the "attack" stopped, and the logs were clear by Monday.