Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Service discovery & the mesh

Who turns a service name into the address of a live instance, and where: a library in the caller, something in the middle, or a mesh proxy, and what each placement costs.

One word in a URL

A parcel carrier's label service makes one call for every parcel it prints: GET http://rates/quote, to price the shipment. At that moment the rates service runs as 14 instances, three of them started ten minutes ago for the afternoon peak and one about to be replaced by a deploy. The word rates has to become the IP address and port of one of the instances that is up right now.

The first version skipped the question: its config held the addresses of the four rates servers the company ran then, so every new instance meant editing that list and redeploying every caller. Service discovery is the general answer: callers name the service they want, and something resolves that name to a live instance at call time, from a list the fleet keeps as current as it can. Keeping that list true, with instances registering, renewing and expiring, belongs to Service Registry. This article is about the question a new engineer asked in a design review: who does the resolving, and where? The three common answers differ in what the label service has to know.

Inside the caller

In client-side discovery, the label service carries a discovery library. The library fetches the list of rates instances from the registry, keeps a copy in memory that it refreshes every few seconds, and for each request picks an instance itself and connects straight to it. Netflix's Eureka, paired with a client-side load balancer such as Ribbon, is the well-known example.

The call takes no extra hop, and the caller can balance cleverly: it sees its own response times from each instance and can steer away from a slow one before any health check notices. What it asks in return is the library: every service that calls rates needs that logic, usually as a library for each language the company uses, and a change to balancing that lives in that code means redeploying the callers. A script written in an afternoon has to do the lookup too, through the library or its own calls to the registry, or it falls back to a hardcoded address, the thing discovery was meant to retire.

Somewhere in the middle

In server-side discovery, the label service calls one stable address and something on the path picks the instance. The classic form is a load balancer in front of the rates fleet, kept in sync with the registry, so callers need only a name and a plain HTTP client (the balancer's algorithms are the subject of Load balancing & horizontal scaling). In exchange, with a dedicated balancer, each call takes one more network hop, through a component that must itself run redundantly.

Kubernetes builds this into the platform: a Service named rates gets a stable virtual IP and a DNS name, rates.shipping.svc.cluster.local, and kube-proxy on every node programs rules that send each new connection to that IP to one of its ready Pods (how a Pod gets onto that list is told in Service Registry). The name always resolves to the same virtual IP, so DNS never sees the churn, and the Pod is chosen per connection. That bites with long-lived connections: a gRPC client by default sends every request down one HTTP/2 connection, so all its traffic goes to the Pod that connection reached, and Pods added for the peak get none of it.

When DNS hands out the instances

A lighter form uses DNS itself: the name resolves straight to the instances' addresses, one record each, and the caller picks one and connects to it. Kubernetes does this for a headless Service, and Consul answers DNS queries for registered services the same way. Nothing new sits on the request path.

The trouble is that DNS was built for answers that change rarely, and these change with every instance that starts or stops. Resolvers cache each answer for its TTL, and many clients cache it again: the JVM, for one, keeps successful lookups for as long as its networkaddress.cache.ttl setting says, whatever the record's TTL. And a client resolves when it opens a connection, then reuses it for as long as it lives. So the label service can keep sending quotes to an address scaled away minutes ago, failing each time, and ignore the three that just started. For why answers outlive their TTL, read DNS, proxies & the network edge; for discovery, the rule is short TTLs, clients that honour them, and connections recycled now and then so new instances get traffic.

Beside every service

A service mesh, in its common sidecar form, moves the resolving out of the application and out of the middle, into a proxy that runs next to each instance (the deployment shape that Sidecar explains). The label service still calls http://rates/quote. Its outbound traffic is redirected to its own local proxy, and that proxy does the rest:

  1. The mesh's control plane watches the platform's list of ready Pods and pushes the current rates endpoints to every proxy.
  2. The label service calls rates as before, and its local proxy receives the request.
  3. The proxy picks a rates instance from the list it holds, per request rather than per connection, once it sees the traffic as HTTP or gRPC.
  4. The proxy connects to the proxy beside that instance, each side proving its identity with a certificate the mesh issued: mutual TLS, which Istio and Linkerd turn on between their proxies by default.
  5. The rates proxy can check the caller's identity against an access policy, then hands the request to the rates process.
The proxies resolve

Neither service knows where the other runs; the control plane keeps each proxy's endpoint list current.

Istio works this way, with Envoy as the proxy; Linkerd uses a proxy of its own. When the proxy picks per request, the gRPC connection is no longer pinned: its new RPCs spread across the instances, while a stream already open stays on its instance until it ends. And with every call already passing through a proxy, the mesh hands it retries and timeouts per route, identity between services, and the same metrics for every call, whatever the service's language.

What the proxies cost

Every call now crosses two extra proxies, one at each end. Each adds a little latency, most visible in the tail, and every instance carries a proxy's memory and CPU. The control plane is one more system to upgrade and keep healthy, and a failed call has one more suspect: the code, the proxy, or its configuration.

Two mistakes make the bill larger. One is retrying in two places: if the label service makes up to three attempts and its proxy makes up to three for each, a quote that keeps failing becomes up to nine attempts, so pick one layer to retry and give it a budget, as Retry explains. The other is business routing in the proxy, such as sending premium customers' quotes to a separate pool: the proxy's job is transport, and a product rule kept in mesh configuration ends up owned by whoever runs the mesh instead of the team that owns the rule.

Choosing for the label service

For the label service, the platform's server-side discovery is enough while a few teams, mostly in one language, can keep encryption, retries and metrics in a shared library; a mesh earns its proxies once many teams in many languages need them between services, and someone is staffed to run it. Whichever you pick resolves from a list, and keeping dead or unready instances off that list is work for the registry and for health checks (Health Endpoint Monitoring); a cluster that learns its membership from peers, through Gossip Protocol, still has to detect the dead ones.

The Service Discovery Challenge's brief describes the client-side route, so make that choice visible: the Client fetches the list from the Service Registry and then calls three Service instances directly, with its edges labelled by how often it refreshes its copy and how it picks among healthy instances. Five or six nodes cover it, inside the Challenge's eight. Keep the registry off the request path between the Client and an instance; it hands out addresses, not traffic.

Practice this