Changed, and still the old address
You change a DNS record, the provider's console shows the new address within seconds, and requests keep arriving at the old one. Where is the old answer coming from? A team running an invoicing service met that question while moving its API from a cloud region in Virginia to one in Frankfurt, for its European customers. Two hours after the record for api.invoicer.example changed, Frankfurt had 60% of the requests, Virginia still had 40%, and one customer's integration was sending every new invoice to the region due to be switched off that evening.
Nobody got the record wrong. DNS (the Domain Name System) resolves a hostname to an IP address: the lookup before a connection (cached for the record's TTL), and a place to do load distribution and failover. The answer is not fetched from the team's servers for every request; it is copied along a chain of caches, each keeping its copy for as long as it was told it may.
Who answers the question
Follow one lookup from the customer's integration server, on a machine that has not asked before:
- The HTTP library asks the operating system's stub resolver, the small DNS client built into the OS, which checks any local cache and forwards the question to the configured resolver.
- That recursive resolver, run by the customer's network, their cloud provider or a public service, checks its own cache. On a miss it asks a root server, which refers it to the servers of the top-level domain, which refer it to the zone's authoritative servers.
- An authoritative server, at the DNS provider where the invoicing team edits its records, answers with the record: an address, and a TTL (time to live) in seconds,
3600here. - The recursive resolver keeps the answer for up to the TTL and returns it, and only then does the connection start.
The resolvers remember, and the recursive one keeps the referrals too, so after a recent lookup under invoicer.example it goes straight to the authoritative servers, and most requests cost no lookup at all. The same memory is why a change reaches each client only when that client's copy runs out. The connection that follows has its own cost in round trips, which Network protocols: TCP, UDP, HTTP/2 & HTTP/3 counts.
Root and top-level-domain servers are left out; the resolver has their referrals cached.
How long a wrong answer may live
The team lowered the TTL from 3600 to 60 thirty minutes before the move, and that was not enough. A resolver that had fetched the record ten minutes before the TTL was lowered still had twenty minutes left on its copy when the address changed, and would not ask again, or learn about the new TTL, until they ran out. A lower TTL takes effect only once the copies cached under the old one have expired, so lower it at least one old TTL before the change; a day ahead is common.
Why not keep every TTL at 60? Because the TTL trades load against change speed. At 60, every busy resolver asks the authoritative servers about once a minute per name instead of once an hour, and the bill grows at a provider that charges per query. At 86400, a change can take a day to reach everyone, far too slow for a record you may have to repoint in an incident. Most teams use hours for records that rarely move and a minute or a few for the ones they fail over.
Failures are cached too. During testing, a script in the team's Lisbon office queried api-eu.invoicer.example before the record existed; the office's resolver got NXDOMAIN, the answer that the name does not exist, and kept giving it for most of an hour after the record was created. This is negative caching, and the zone sets how long it lasts: the smaller of the TTL of its SOA (start of authority) record and that record's minimum field (RFC 2308).
The clients that never ask again
Resolver caches explain at most the first half hour. The Virginia traffic left after two hours came from three kinds of client:
- Open connections. A client with a keep-alive connection to Virginia keeps sending requests down it, because DNS is consulted when a connection opens, not for each request. A connection pool that never closes its connections never looks again.
- Caches of their own. Some language runtimes and HTTP libraries cache lookups themselves, with defaults that can outlast the record's TTL or ignore it.
- Resolved once. The integration still writing to Virginia had resolved the name when its process started, eleven days earlier, and kept the address in memory.
None of these follows the TTL, so a move by DNS has a tail with no fixed end, and the old address must keep working until the tail is small enough to cut. The team changed the Virginia proxy to forward every request it received to Frankfurt, asked the integration's owners to restart it, and switched Virginia off a week later instead of that evening.
Different answers for different askers
With customers on both sides of the Atlantic, the team stops giving one address to everyone. A DNS provider can answer by who is asking: geo answers choose a region from the asker's location, latency-based answers choose the region with the lowest measured latency from the asker's network, and health-checked failover stops returning a region whose health checks fail. Without that last one, a dead region keeps receiving traffic until someone repoints the record by hand.
The asker, though, is not the user. The authoritative server sees the recursive resolver's address, so a customer in Madrid whose company sends all its DNS through a resolver in Virginia is steered by Virginia's location and handed the far region. EDNS Client Subnet lets a resolver pass on a shortened form of the client's address so the answer can be keyed on that instead; some resolvers send it and some do not, for privacy, so a geo answer remains a guess about where the user is. Failover has a lag of its own: after a health check fails, resolvers and clients keep using cached answers until the TTL expires, and the clients of the last section longer still, so records you fail over need short TTLs. Taking one dead server out within a region is the load balancer's job (Load balancing & horizontal scaling); deciding which region a user's data lives in is Regional routing & geo-partitioning.
Where the address leads
The address in the record is not an application server. In both regions it belongs to a reverse proxy, Nginx here, which accepts the clients' connections and opens its own to the application behind it. It terminates TLS, so the certificate for api.invoicer.example lives on the proxy instead of on every app server, and where the network behind it is not trusted it re-encrypts to the app servers with internal certificates or mTLS (Secrets management & encryption). It routes by path, /v2/invoices to the new service and /webhooks to the old one, so a service can move without its name changing; one such rule is how the Virginia proxy sent the tail on to Frankfurt. And it is the only part with a public address, while the app servers sit on the private network that Network topology: VPCs, NAT, VPN & bastions lays out.
How late is the last client?
The Global DNS System Challenge puts you on the authoritative side, and its brief asks for TTL caching, geo and latency answers, and fast record changes. Your design should answer one question with a number: after an operator edits a record, how long until no resolver that respects the TTL and can still reach your servers returns the old address, and how much of it is the propagation to your own servers and how much the TTL you chose? Ask it again for a name that answered NXDOMAIN before it existed. The routing requirement of Content Delivery Network raises the same question for an edge that fails its health checks.