One price, five thousand times a second
An online shop's most popular product page gets 5,000 views a second, and every view runs the same query against PostgreSQL: SELECT price FROM products WHERE id = 42. The price changes about once a day, so the database spends its day producing one answer over and over, and when a sale doubles the traffic, that query is what tips it over.
The answer is to keep a copy somewhere cheaper to reach. A cache holds a copy of data in faster storage, usually memory, so most reads never reach the database; the share of reads it answers is its hit rate. The real question is where the copy lives. A read can be answered in five places, from the database out to the customer's browser, and the layers differ in three ways: who shares the copy, how stale it can get, and how a change reaches it.
A shared cache beside the database
The first copy goes into Redis, an in-memory store running on its own server next to the app. The app asks Redis for product:42 and goes to PostgreSQL only on a miss. At a 95% hit rate, 5,000 reads a second become 250 queries. Memcached fills the same role with fewer features. Cache-Aside covers that read path, the delete on write, and a race that can bring an old price back.
Teams usually add this layer first because it is shared: the shop's 20 app instances read the same copy of each key, so deleting one key takes the old price away from all of them at once. The cost is a network round trip per lookup and one more system to run.
It also changes what the database is sized for. PostgreSQL now handles 250 queries a second, and if Redis goes down or restarts empty, all 5,000 come back at once. So plan for the cache being down or cold: fall back to the database, shed the load it cannot take, and warm the cache before sending traffic to it. A hot key expiring at peak is a smaller version of the same burst, and Hot keys & cache stampedes covers the defences.
A copy inside each app instance
A product page needs more than its price: the category tree, currency rates, a dozen feature flags. Twenty Redis lookups per page at 5,000 pages a second is 100,000 network calls a second, mostly for data that barely moves.
So each instance keeps the hottest of these, plus the few hundred most viewed products, in its own memory: an in-process cache, often just a map with a size limit and an expiry. A hit costs no network at all. What to throw out when the map is full is covered in Cache eviction: TTL, LRU, LFU.
The catch is 20 copies, one per instance, each filled separately and lost when its instance restarts or is redeployed. Deleting the Redis key does nothing to them. Either every instance hears about a change, for example through a pub/sub message, or each copy expires after a short TTL (time to live, the age after which an entry stops being used, so the next read reloads it). Until then, the load balancer can send one customer's two requests to two instances holding two different prices. Seeing $30 and then $40 is a consistency problem, which Consistency models: strong vs eventual (CAP) takes up.
A copy near the customer
Everything so far sits in the shop's data centre in Virginia, so a customer in Lisbon waits for a trip across the Atlantic, and the app still builds the same page 5,000 times a second. A CDN (content delivery network) keeps copies of whole HTTP responses on servers in many cities, called edges. The product image, the stylesheet and, as long as it is the same for every visitor, the product page itself can be answered by the edge in Lisbon without reaching the shop at all.
An edge copy is shared by everyone that edge serves, so one fill answers thousands of visitors. The shop's response headers set how long it is kept: Cache-Control: s-maxage=60 gives shared caches such as a CDN 60 seconds. That sharing is also the danger. A page that shows one customer's name or cart must never be cached at a shared edge, or the next visitor receives it.
Changing data here is harder again, because the copies sit on the provider's servers. You can wait for the TTL, ask the provider to purge the URL, or change the URL. Images and stylesheets usually take the third route: a file named after a hash of its contents, such as app.3f9a1c.css, never changes under that name, so it can be cached for a year, and a new version is simply a new file. CDN & static delivery covers edge locations, purging at scale and shielding the origin.
A copy in the customer's browser
The closest layer is the browser itself. The same Cache-Control header tells it how long it may reuse a response: with max-age=300, a returning visitor's browser shows the stored copy for five minutes without sending a request. Nothing is faster, and this copy serves exactly one person.
It is also the one layer the shop cannot reach. There is no purge for a million browsers, so every lever is pulled in advance: a short max-age; no-cache, which lets the browser keep a copy but makes it check with the server before each use (it binds the CDN too, so it would also cancel the edge's 60 seconds; cheap when the response carries a validator such as an ETag, because an unchanged response comes back as a short 304 Not Modified); or a versioned URL. That is why the shop sends the product page with Cache-Control: max-age=0, s-maxage=60, which makes browsers revalidate on every visit while the edge keeps its 60 seconds, and lets browsers keep the hashed images for a year. HTTP caching: Cache-Control, ETags & revalidation covers these directives and the revalidation exchange.
10:00, the price drops to $30
The diagram is the stack the shop now runs. A request goes only as far as it has to: the browser, the edge, the app's own memory, Redis, and PostgreSQL last.
A request stops at the first layer that has a fresh copy. Redis never talks to PostgreSQL; the app queries it after a Redis miss.
At 10:00 the price of product 42 drops from $40 to $30. The update commits in PostgreSQL and the app deletes product:42 from Redis. Now follow the old price outward:
- Redis: gone. The next lookup misses and loads $30.
- In-process caches: an instance that holds $40 keeps serving it until its copy expires or a change message arrives. With a 30-second TTL, some instances show $30 and others $40 for up to half a minute.
- CDN edges: every edge that cached the page keeps $40 for the rest of its 60 seconds, unless the shop purges the URL.
- Browsers: they revalidate the page, so they get whatever the edge now holds. Had the page gone out with
max-age=300, some customers would see $40 for up to five minutes, and the shop could not shorten that.
The staleness can also add up. An instance can build the page from a copy that is 29 seconds old, and the edge then keeps that page for its own 60 seconds, because it cannot see how old the data inside was: about 90 seconds of $40 in the worst case, not 60. HTTP caches pass a response's age along in the Age header, so a browser counts the time a copy spent at the edge; the caches inside your app are invisible to that clock. Treat how stale the data may get as a budget for the whole path, and split it across the layers' TTLs.
Each layer also needs a decision about what a write does to its copy. Pick a policy deliberately, a TTL, invalidate-on-write (delete the key when the row changes), write-through (write the new value to the cache along with the database), or a combination, and accept the consistency trade-off that comes with it. Write-around keeps rarely read writes out of the cache, but it still needs invalidation or a TTL, or a copy that was already cached stays stale. Cache-Aside covers invalidate-on-write, and Cache read and write strategies covers write-through and write-around.
Which layers does this data deserve?
A layer earns its place when it saves something real and you can live with what a change costs there. Ask the same three questions of the data:
- Who may share the copy? An answer that is the same for everyone can sit in any layer, the CDN included. One that belongs to one user, like a cart or an account page, stays out of shared edges: the browser with
Cache-Control: private, and server-side caches keyed by that user. - How stale can it get? That is the budget for the whole path, which the layers share.
- How does a change reach every copy? If the honest answer for a layer is that it cannot, as with browsers, give that layer only data you can tolerate stale for its full lifetime, or a URL that changes when the content does.
For the shop: hashed images and stylesheets sit in browsers and edges for a year. The product page gets 60 seconds at the edge and browser revalidation, with the price behind it in Redis. Stock levels change every second and decide whether a purchase goes through, so they get a few seconds in Redis at most, and checkout reads the database. The cart lives in Redis keyed by session and is never cached at an edge. Data that is rarely read twice, such as a one-off export, deserves no layer at all: every request pays for the lookup and the fill, and almost none of them hit.
Layer a weather API
Open the Weather Service Challenge. Its forecasts are a clean case for layering: the same answer for everyone asking about one city, refreshed upstream on a schedule, and costly to fetch from the providers. For each response, decide which layers may hold it and with what TTL, and check that the TTLs together, plus the time between scheduled refreshes, stay within how old a forecast may be. Then add the scheduled refresh and say how a new forecast reaches copies that already exist, at the edge as well as in Redis.
Two Challenges take single layers further. Distributed Session Store asks you to invalidate a session on logout everywhere, including the copies app nodes keep locally. Content Delivery Network is the edge layer on its own: TTLs, purge on demand and shielding the origin.