Election night at a small news site
A regional news site runs on four application servers behind one load balancer, in a data centre with a 10 Gbit/s link to the internet. A CDN caches its article pages; search, comments and subscriber login reach the servers every time. On the night of a local election its readership triples, as planned. Three other kinds of traffic arrive too, and none of them is a reader:
- A scanner sending searches such as
council' OR '1'='1and comments carrying script tags. - A flood of UDP packets at 80 Gbit/s, aimed at the load balancer's IP address.
- 30,000 searches a second for random pairs of words, from about 20,000 home internet connections, each a well-formed HTTPS request.
Each needs a different defence, all of them at the edge: the part of the system that receives traffic from the internet before the application does.
A filter that reads requests
The scanner's requests are ordinary HTTP with parameters shaped like attacks: SQL fragments, script tags, ../ sequences reaching for files outside the web root. A web application firewall (WAF) sits in front of the application, reads each request (method, path, headers, query string, body) and compares it against rules; a match can log the request, block it, or challenge the client. Most teams start from a managed rule set, the open-source OWASP Core Rule Set or their provider's, and add their own. To read requests the WAF must see them decrypted, so it terminates TLS, or sits behind whatever does; Secrets management & encryption follows TLS from there to the origin.
Against the scanner this works: its probes stop at the edge instead of landing in application code. A rule for a newly published library flaw can even go live the same day, a stopgap until the patch ships, since attackers vary the payload until something slips past.
What the filter cannot see
- The application's rules. A subscriber who changes
/account/invoices/1041to1042sends a perfectly normal request; whether they may read invoice 1042 is decided in the application (Authorization: roles, relationships & policies). - The flaw behind the pattern. If the search builds SQL by pasting in the reader's text, the WAF blocks the payloads it recognises and the hole stays open for the rest. That fix is made in the code, and Web application security: injection, XSS, CSRF & SSRF shows how.
- Whatever does not reach it, or that it does not read. A request that gets to the servers by another route never meets it; many WAFs inspect only the first part of a large body; an encoding the rules do not normalise can slip through.
- A login with a stolen password. Credential stuffing, trying username and password pairs leaked from other sites, sends valid requests at a modest rate per address. Edge bot detection can slow it; multi-factor authentication and the other defences that make it fail are in Authentication: passwords, MFA & identity providers.
Blocking readers by mistake
A WAF also costs real users. Readers commenting on an explainer about the council's vote-counting software quoted DROP TABLE and got a bare 403; so did a search for O'Connor or Ryan, whose apostrophe followed by or looked like the scanner's probe. Any rule set broad enough to matter produces such false positives, and stricter settings trade more of them for more caught attacks. So a new rule first runs in count mode (also called detection-only: it logs what it would block and blocks nothing) until its false positives are understood and, where needed, excluded for the path and parameter that trip them, and a refusal page with a reference number turns the rest into support tickets instead of readers who quietly leave. A rule left counting forever, or blocking with nobody reading its log, fails silently one way or the other.
A flood that never reaches HTTP
The UDP flood is not made of requests. Its packets are much larger replies from thousands of misconfigured servers (open DNS resolvers, NTP and memcached servers) to small queries sent with the news site's address forged as the sender, so a modest botnet produces 80 Gbit/s. This volumetric attack, built here from reflection and amplification, works at the network layer: the 10 Gbit/s link saturates, readers' requests are dropped in the same queue as the flood before any server can filter anything, and no WAF rule helps. It has to be dropped upstream.
That is the job of DDoS protection (distributed denial of service). A CDN or dedicated provider announces the site's addresses from many locations at once with anycast, so each attacking source's packets go to the location nearest it in network terms, the flood is spread across far more capacity than the site's own link, and each location drops what it recognises as attack traffic before passing the rest on.
It only works if the attacker cannot go around it. The load balancer's address had sat in public DNS for years, so the origin gives up that address and now sits behind a tunnel it opens out to the provider, accepting no inbound connections from the internet. An origin that must keep a public address can refuse requests that skipped the edge by checking a client certificate the edge presents, which keeps attackers out of the application but not off its link; an allow-list of the provider's address ranges is weaker, because every customer of that provider shares them. Network topology: VPCs, NAT, VPN & bastions keeps the origin private, and DNS, proxies & the network edge points the name at the edge.
A flood of perfect requests
With the pipe protected, the third wave is the hard one. ?q=harbour+lantern, ?q=quarry+velvet: each search comes from a real home connection with a browser's headers, the WAF finds nothing wrong, and the CDN has never seen the query. The search database handles about 1,500 queries a second, so a twentieth of the attack's rate already saturates it, from traffic well under 1 Gbit/s. This application-layer (layer 7) DDoS exhausts the most expensive thing one request can trigger, not the network.
A rate limit per IP address barely bites: 30,000 requests a second from 20,000 addresses is 1.5 a second each on average, and a cap tight enough to matter at that rate would also refuse busy readers and the many who share one address behind an office or mobile network. A per-address cap still stops a single loud source, and a total cap on search keeps the database alive at the price of refusing some real searches (Rate limiting has the algorithms). The rest makes each request cost the attacker more and the site less:
- Challenge what looks automated. The edge scores clients on signals such as whether the TLS handshake and headers match the browser claimed, and sends doubtful ones a JavaScript or CAPTCHA-style challenge (Cloudflare Turnstile, reCAPTCHA, hCaptcha). Simple bots fail; a bot driving a full browser can pass, and a visible challenge costs readers friction, most of all those using assistive technology.
- Answer repeated questions at the edge. Real readers mostly search the same few candidates' names, and caching those results for a minute serves them without the database (CDN & static delivery); random words still miss, so this keeps popular searches answered rather than stopping the attack.
- Refuse what the origin cannot serve. Whatever gets through, the application turns away the excess early and cheaply instead of slowing down for everyone: Load Shedding.
Layers that expect each other to fail
Scrubbing drops the flood, the WAF drops attack-shaped requests and challenges doubtful clients, the cache answers repeated searches, and the origin takes traffic only from the edge.
This is defence in depth: independent layers, each built on the assumption that the one before it will miss something. The WAF is there in case the code has an injection flaw nobody knows about, and the code is written as if there were no WAF. The origin refuses traffic that skipped the edge, in case someone finds its address. And the host that faces the internet should hold as little as possible, no database password included (the service behind it keeps that), which is the idea Gatekeeper makes a pattern of.
No Challenge yet exercises a WAF's request filtering. Global DNS System practises the volumetric half: it must stay available under DDoS, with edge caches answering most queries and no route to the authoritative servers around the shield. Its palette draws that shield with its WAF node; read the node as a stand-in for edge scrubbing only, since a WAF reads HTTP requests and ordinary DNS queries are not HTTP.
Layers also leave gaps. A month after election night the site found its mobile app's API hostname still pointing at a server with a public address on the same link, outside every layer in the diagram. The cheapest check in this article is a list: every hostname and port the internet can reach, and whether each one goes through the edge.