September 11th, 1999, thousands of /. visitors crashed ThinkGeek.

It was definitely not a bandwidth issue. ThinkGeek was running on a PII 450 with 128 megs of SDRAM. Add that to a CGI intensive shopping cart, with loads of dynamic pages, an outdated version of MOD_PERL and you have the ingredients for a slashdot effect. (What happens when you get slashdotted? Well, in our case you simply get overwhelmed with requests and run out of memory and your server crashes. I think the uptime was hovering around 124 when it finally fell. Neither /sbin/shutdown or a three finger salute could muster enough life to save us from the inevitable fsck...)

Somewhere we must have failed as geeks. And we had to spoil a 70 day uptime as well. How humbling...

So, we created the following temp page and set out to conquer the effect which conquered us.

Jen created a bunch of static pages from dynamic ones. Scott went to grab another PII 450 and 128 megs of RAM from a server at my home. I compiled a new kernel with SMP support and upgraded our version of MOD_PERL. After Jon put the new memory and 2nd processor in the server, things went a bit smoother. We could actually keep the server up past the 10-15 minutes it was lasting previously...

An order came in. And then another. ;)

Time to sit back and watch some Jackie Chan after all...

(For the curious: Actual picture of a slashdotted server)