My Side Project Got Hit by a DDoS Attack. Here is What I Learned.

I have been working on a side project for a few months now. I was applying everything I’ve learned over the last few years and having fun until something I only ever expected to see in the movies happened to my site.

A few nights ago, around 7:00 pm, I noticed a massive spike in requests to the web application. The high volume traffic persisted for more than an hour. That was when I realized this was far more than a casual surge in traffic. While the load balancer was working as expected, the downstream tasks were completely overloaded. The swarm of requests slowed the app’s response times to a crawl, pushing the median latency to 20 seconds and the p95 latency to a staggering 45 seconds (basically, the site was down). The database was unharmed and available throughout the whole ordeal.

I was under a DDoS attack.

I started monitoring the web application firewall, only to find the application getting hit with a massive wave of traffic originating from South Africa. Shocked and wondering why this was happening, I began reviewing the firewall rules. First, to mitigate the situation, I blocked all traffic from that specific geolocation — only to watch another wave of requests surge from South Korea less than five minutes later.

Faced with a shifting attack, I spent the night — not having fun — reviewing our security measures and reconfiguring our setup. I shifted our rate limiting strategy from a generous global threshold to a strict per-IP burst limit.

Lessons Learned:

• Don't misinterpret a surge: If you didn't invite people over, don't get too excited when someone knocks on the door!
• Align your defenses: A rate limiter is only as good as its configuration; ours was active but wasn't properly aligned with our compute capacity and auto-scaling thresholds.
• Geo-blocking is a game of Whack-A-Mole: Blocking traffic by geolocation is a temporary band-aid, not a permanent fix. In a distributed attack, the bad actors will just rotate their proxies to a new country in minutes.
• The illusion of uptime: Just because your load balancer isn't failing doesn't mean your application is healthy. It will happily route toxic traffic straight to your backend if you let it.

Moving Forward:

To make sure the next attack drops before it ever touches my backend, I am already adding more items for better resiliency, like edge caching and WAF challenges.

If you want to follow along with how I build, secure, and scale this project, sign up for the Codex Brew newsletter to get updates on the next post!

For those who have been through the fire: how do you typically handle these kinds of situations? What are your go-to and must-have strategies for keeping a project safe without breaking the bank?