DDOS post-mortem 12 Jan 2015

Chart

Today, shortly after noon (EST) ScrumDo experienced some intermittent slowness and even a few short spouts of downtime for some users. As we began to investigate the problem, we immediately noticed thousands of bogus HTTP requests per second coming from Chinese based IP addresses.

Through a combination of spinning up additional app servers to handle the load, and some filtering rules we were able to get the site available for everyone within a few minutes, and working full-speed within an hour.

At the time, we thought we were being targeted in a DDOS attack, but after some more investigation it looks as though we were bystanders that got hit. Apparently, for the past several weeks there is a DNS cache poisoning attack going on in China that points the DNS entry for several highly popular sites to a random IP address. Our number came up, so to speak.

API Users

One of the filters we put in place was to require a valid HTTP_HOST header on all incoming requests. Most client libraries will do this for you automatically and you shouldn’t notice anything.

If you suddenly can’t access the API through one of your scripts, you’ll need to modify the script to send the appropriate header. We’re sorry for the short-notice.

ScumDo API Info