Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Sidecar

Run a helper process in the same deployment unit as a service, sharing its lifecycle, its network and its volumes: what moving a job out of process buys, what it costs on every instance, and why start and stop order matter.

One audit rule, three languages

The rule from compliance has two parts. Every log line a service writes must carry the claim number it concerns and reach the central audit log within a minute. And every service must present a TLS certificate from the internal certificate authority, renewed every 24 hours. It applies to all nine services of an insurer's claims platform: intake, documents, payouts and six more, written over the years in Java, Python and Go.

The first attempt was a shared library. The platform team wrote it in Java, then ported it to Python; the Go port lagged a release behind. When the audit log changed its format, all nine services had to rebuild and redeploy, and the payouts team, in a release freeze for the quarter's close, shipped two weeks late. Nothing in the rule was about handling claims, yet every claims team had to schedule work to follow it.

The second attempt moved both jobs out of the services' code. Next to each instance of each service now run two small processes: a log shipper that reads the service's log files and sends them on, and a certificate reloader that fetches a fresh certificate and writes it to a file the service re-reads when it changes. Each is a Sidecar: a helper process deployed in the same unit as the service it serves, whether that unit is a Kubernetes pod, an ECS task or a VM. It is scheduled with the service and started and stopped with it. In a Kubernetes pod, as in an ECS task using the awsvpc network mode, it shares the service's network namespace, so the two reach each other on localhost, and it can mount the same volumes, so they can share files.

Two helpers beside claims intake

All three run in the same pod. Intake writes plain log files, each line carrying its claim number, and reads a certificate file; it knows nothing about the audit log or the certificate authority.

The helpers are written once, in whatever language suits them, and serve all nine services whatever theirs. A new shipper image needs no change to the payouts code and no build of it. A library still wins in some places: calling it is a function call, while reaching a sidecar crosses a process boundary, so a job that needs the request's own data mid-handling, or a small fleet in one language, is cheaper served in process.

The bill comes per instance

A sidecar runs beside every instance, so its cost is multiplied by the fleet. The claims services run about 360 pods at the end of the month. If the log shipper reserves 50 MiB of memory in each, the fleet sets aside nearly 18 GiB for log shipping alone, before the certificate reloader takes its share. A helper that looks tiny in one pod is a line item across the cluster.

A sidecar that proxies the service's traffic also adds a local hop, through a second process, to every call it proxies. A proxy beside every service, run by a control plane, is a service mesh, covered in Service discovery & the mesh; a proxy for the service's outbound calls alone is an Ambassador.

Who starts first, who leaves last

Sharing a lifecycle does not mean starting and stopping at the same instant, and the order is where sidecars bite. At start-up, intake expects a certificate file that the reloader has not written yet; it fails its TLS set-up and restarts, sometimes several times, until the reloader catches up. A service that calls out through a proxy sidecar has the same problem in another form: its first requests get connection refused from a proxy that is not listening yet.

Shutdown is worse, because the loss is silent. A deploy replaces an intake pod:

  1. Kubernetes signals every container in the pod, all of them ordinary containers here, to stop, with no order between them.
  2. The log shipper sends what it has already read and exits within a second.
  3. Intake spends another 20 seconds finishing the claims it had in flight, writing a log line for each.
  4. The pod is deleted, and its emptyDir volume, holding those last log lines, goes with it.

The audit log now has a gap of 20 seconds for every pod replaced, exactly where claims were being submitted during a deploy. Kubernetes added a native answer: a container listed under initContainers with restartPolicy: Always is a sidecar container, available since v1.28, on by default since v1.29 and stable since v1.33. The kubelet starts it before the app's containers and, when the pod terminates, waits until the app's containers have stopped before stopping the sidecars, in the reverse of their order in the spec, all within the pod's termination grace period. The app's containers start once the sidecar is reported started, which, if the sidecar has a startup probe, means the probe has passed, so the probe should test what the app needs: here, that the certificate file exists. On older clusters and other platforms, the equivalents are a dependency setting where the platform offers one, a short delay before the helper exits, and a service that tolerates its helper not being there yet.

When a helper dies

A helper can also crash on its own. The kubelet restarts it inside the same pod, and what happens meanwhile depends on the helper. While the log shipper is down, intake keeps writing to its file, and a shipper that records how far it had read catches up when it returns; resuming avoids losing lines, but it does not by itself meet the one-minute deadline. The reloader can be down until the certificate it last wrote expires, so its margin is whatever validity that certificate has left. A proxy sidecar is different: while it is down, every call through it fails. And in Kubernetes a pod is ready only while all of its containers are, so a helper that is down, or failing its readiness probe, takes the whole pod out of its Service's traffic. Give a helper a readiness probe only when the service truly cannot serve without it: a CA outage that fails the reloader's probe would take every intake pod out of service, with valid certificates still in place.

Upgrading a helper usually means replacing every pod of every service that carries it, so for a while two versions run side by side and whatever receives their output must accept both. Where a mesh's admission hook injects the sidecar, a new version reaches a pod only when the pod is recreated.

When the helper learns about claims

Once the shipper works, requests arrive for it. The payouts team asks whether it could also flag claims above 10,000 for a second review, since it already reads every line. It could, and then a change to that threshold, or to the claim fields it reads, would have to ship in step with payouts, and the shipper would no longer be the same helper in all nine services.

Watch outCommon mistake: packing business logic into the sidecar instead of pure cross-cutting concerns, so the app and its sidecar become inseparably coupled again.

In the Sidecar & Ambassador Challenge, give the Sidecar node one cross-cutting job, logging or TLS, say on the nodes or a connection label that it shares the app's pod, and connect the two by something local, a file or localhost. Then apply the test the payouts request failed: if your sidecar has to know claims rules, or change when they change, business logic has crept in.

Practice this