Redis is down. Is this replica healthy?
A code review on a geocoding API, which turns addresses into coordinates, has stalled on one handler. Twelve replicas run behind a load balancer. Each holds a 3 GB address index in memory, and all twelve share one Redis that counts every API key's requests against its quota. The new /health handler pings Redis and returns 503 if Redis does not answer. Half the reviewers say that is what a health check is for; the other half say it will take the whole API down the first time Redis blinks.
Both halves are right about something. Health Endpoint Monitoring means each service exposes endpoints that something outside the process calls to decide what to do with it. What the endpoint should test depends on who is calling, and what that caller does with a failure.
The orchestrator asks: should I restart you?
One morning replica 7 is running, its port accepts connections, and it answers nothing. Every request thread is waiting on a lock that will never be released. Nothing crashed, so nothing restarts it. A liveness check exists for this case: the orchestrator calls it every few seconds and, after a few failures in a row, kills the process and starts a fresh one.
To catch replica 7, the check has to go through the same machinery requests use. A handler on a separate thread that always returns 200 will keep reporting a deadlocked process as alive.
Now put the Redis ping in liveness. Redis stops answering for forty seconds, every replica fails, and the orchestrator restarts all twelve. Each needs about ninety seconds to load its index again, so a forty-second blip in a dependency becomes minutes of total outage, and the restarts fixed nothing, because nothing was wrong with the replicas. Keep liveness local to the process, never a dependency check. A timeout that is too tight does the same on a smaller scale: a garbage-collection pause under load makes a healthy replica miss its check, and it is restarted just when capacity is scarce.
The load balancer asks: should I send you traffic?
A replica can be perfectly alive and still not want requests: it is loading its index, it is draining connections before a shutdown, or its own pool of worker threads is full. Readiness answers that. A failed readiness check takes the replica out of rotation and a passing one puts it back; nothing is restarted. The same instinct drives a service registry that hands out addresses, which Service Registry covers.
This is why liveness and readiness are separate checks. Merged into one, a replica that is merely busy fails the check and gets restarted, throwing away its warm index, when all it needed was a few seconds out of the load balancer.
Liveness looks only at the process and its failure means a restart. Readiness looks at what is particular to this replica and its failure means no new traffic. Neither fails because the shared quota cache does.
Back to the review. With the Redis ping in readiness, the forty-second blip goes like this:
- Redis stops answering.
- Every replica's readiness check pings Redis and fails.
- After its failure threshold, the load balancer marks all twelve replicas unhealthy.
- No replica is left in rotation, and every geocoding request fails until Redis returns and the checks pass again. Some load balancers fail open here and route to all twelve anyway, so the check bought nothing.
Moving traffic between replicas cannot help when what failed is shared by all of them. So readiness checks what is particular to this replica, including a dependency only this replica relies on, such as a sidecar running beside it. A dependency every replica shares stays out of it, so that its failure cannot eject the whole fleet at once. Even a replica that can do nothing useful without Redis loses nothing by failing those requests fast itself, and it serves again the moment Redis does, not a threshold later. How the replica copes with a dead quota cache is a decision for its own code, such as counting quotas locally for a while, accepting some overage, or refusing requests it cannot account for, often behind a Circuit Breaker that stops it waiting on Redis. Redis's own health still matters, so it goes to a separate deep-health endpoint that feeds alerts, not routing.
Shallow, deep, and who can call them
A shallow check confirms the process can answer; a deep check exercises dependencies, running a query or calling a downstream service. Deep checks suit alerting and cost more than they look, since every prober calls every replica every few seconds and a deep check that calls other services' deep checks multiplies that, so cache the result for a few seconds. Keep the detail private, too: publicly a status code is enough, because a public deep check lets anyone load your database by calling it in a loop, and versions or hostnames in the answer tell an attacker what you run.
Thresholds, and the replica that keeps coming back
Every check runs on an interval and acts after a threshold, and the product of the two roughly sets how long a dead replica keeps receiving requests: five seconds times three failures is about fifteen seconds of failed requests, which callers can ride out with the timeouts and retries Load balancing & horizontal scaling describes. Shorten either number and a slow moment gets a healthy replica removed.
Overload produces the worst version. Replica 3 is saturated, fails readiness and is removed; rested, it passes, comes back to its full share at once and fails again, while the eleven that took its traffic grow hotter. Requiring more consecutive passes to re-add a replica than failures to remove it damps that flapping, but the cure for overload is refusing excess work early, never the health checks themselves, as Load Shedding explains.
The first ninety seconds
Each new replica spends about ninety seconds loading its index. A liveness check that gives up after thirty seconds kills it before it finishes, again and again: a crash loop, in a deploy that would have worked. A fixed initial delay is a guess that breaks the day the index grows. A startup check, as Kubernetes defines it, fixes it: it passes once the index has loaded, liveness and readiness are not run until then, and it has its own generous budget, say thirty failures five seconds apart. Once it passes, liveness takes over with its tight threshold. Readiness keeps failing until the index is in memory, so no request reaches a half-loaded replica.
From outside, a different question
All twelve replicas can be ready while no customer reaches them, behind an expired TLS certificate or a broken DNS record that no check inside the network sees. Synthetic monitoring answers that question with a probe that calls the public URL from several regions the way a client would, confirming an outage from a second region before anyone is paged. Turning that result, and the deep-health endpoint, into alerts on what users feel is the subject of Observability: metrics, logs & alerting.
Evicting the right instances
In Service Discovery, the requirement this article answers is the third one: health checks evicting dead instances from the registry. Put a Health Endpoint Monitoring node between the Service Registry and the services, and label the edge with what the check tests, the process and its own readiness and nothing they all share, so a shared outage cannot empty the registry. Failover & Request Hedging asks the same question about a primary, where a probe cut off from a healthy primary is not proof that it is dead, and Uptime Monitoring & Status Page is the outside view, built as a product.