Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Containers & orchestration

Package a service with everything above the kernel into an image, name it by digest, and let an orchestrator keep the declared number of copies running: placed by requests, capped by limits, checked by probes, replaced by rolling updates.

Same commit, different invoice

An accounting product's invoices started leaving production in a fallback font, the totals column a few pixels out of line. The commit was the same one that rendered correctly on every laptop. The cause: the PDF renderer calls a font library installed on the machine, and the production servers, set up by hand two years earlier, carried an older version of it. The code was identical; everything around it was not.

A container fixes that by shipping the surrounding software with the code, as an image. An orchestrator then takes that image and keeps the right number of copies of it running across a fleet. The two belong together: the image is the unit the orchestrator runs.

An image carries everything above the kernel

A container image is a file system plus the instructions to start it: the operating system's userland (libraries, often a shell and a package manager), the language runtime, the system packages, the font library among them, and the service's code with its start command. Docker builds one from a Dockerfile, a list of steps, and a container is a process started from that image. The invoice service then runs the same way on any host with a compatible kernel and CPU architecture. Both halves of that condition matter: the image has no kernel of its own, and an image built for x86-64 does not run on an ARM server without emulation.

Each build step that changes files produces a layer, a read-only set of file changes stacked on the ones below: base system, runtime, font and PDF packages, code. Layers are named by a hash of their bytes (Content-addressed storage & deduplication explains why that stores identical layers once), so ordering them from what changes least to what changes most means a code change rebuilds only the top layer, and hosts that hold the lower ones download only the new one.

A running container adds a thin writable layer, lost when the container is replaced, which an orchestrator does routinely; finished invoices go to object storage or a database. Two things stay out of the image. Secrets, because anyone who can pull it can read every layer, and a file deleted in a later step is still inside the layer that added it. And environment-specific settings: one image should run in staging and production, with the differences injected when it starts, from the places described in Secrets management & encryption and Configuration management.

Tags move, digests do not

After the build, the pipeline pushes the image to a container registry, where hosts pull it from. Each image has a digest, sha256: and a hash of its manifest, that identifies exactly those bytes, and usually tags, readable names such as invoice:1.8 or latest. A tag is a pointer anyone with push rights can move, unless the registry makes tags immutable. Rebuild and push invoice:1.8 again, and hosts that already pulled it may keep the old bytes while new hosts get the new ones: two copies of "version 1.8" render invoices differently. Deploy by digest, and every host of a given platform runs the same bytes, so "what is running?" has one answer. Whether your pipeline built those bytes is a separate check, a signature verified before the image runs, covered in Software supply chain: signing, provenance & scanning.

Walls from the kernel, not a separate machine

On the host, a container is an ordinary process to which the Linux kernel shows a restricted world. Namespaces give it its own view: its own process list, where the invoice service is usually process 1 and sees no process outside the container; its own file system, built from the image; its own network interface and ports, so two containers can both listen on 8080.

Control groups (cgroups) cap what it consumes, CPU and memory among others, and the caps are what make sharing a host workable. An invoice batch with a memory leak and no limit grows until the host runs short, and the kernel then kills some process to free memory, possibly a neighbour's. With a 1 GiB limit, the leak stops at 1 GiB: the kernel first reclaims what it can, such as cached file pages, then kills a process in the invoice container. That keeps this leak inside the container's own cgroup; it does not protect the node from memory pressure with other causes, such as limits that add up to more memory than the node has.

Every container on a host shares that host's kernel, so a container is not a virtual machine, and it is not a strong enough boundary for code you do not trust: that is Sandboxing untrusted code: isolation, limits & egress. Running it like a small VM also hurts trusted code: add a cron daemon and a log shipper to the invoice container, and the runtime tracks only the main process, so the daemon can die unnoticed. A helper that must run beside the service goes in its own container next to it, as in the Sidecar pattern.

Three copies, twenty machines

The invoice service now runs as three copies, on a cluster of twenty machines, or nodes, shared with other services. An orchestrator places each copy, notices when one dies and replaces it, Kubernetes being the common one, and it works from a declaration, not commands. The team writes the desired state, usually in files kept in Git (Infrastructure as code): three replicas of the image with this digest, each requesting half a CPU and 512 MiB, limited to 1 GiB. Each copy runs as a pod, one or more containers scheduled together on one node, sharing one IP address and its ports. Controllers compare the desired state with what is running and act on the difference, over and over. That loop is reconciliation:

  1. A node fails, and the invoice pod on it stops answering.
  2. The control plane marks the node unreachable and, after a grace period of a few minutes by default, removes its pods.
  3. The controller sees two replicas where three are wanted and creates a replacement pod.
  4. The scheduler picks a node with enough unreserved capacity for half a CPU and 512 MiB and assigns the pod to it.
  5. The kubelet, the agent on that node, pulls the image by digest and starts the container.
From build to three pods

The registry holds the image under its digest, Argo CD applies the desired state kept in Git, and Kubernetes keeps three pods running.

The scheduler places pods by their requests, the capacity each reserves, not by what they use right now; the limit is the ceiling the kernel enforces once the pod runs.

Running is not the same as working

One day the PDF renderer in a pod hangs on a malformed invoice. The process is alive and its port open, so the pod looks healthy, and a third of invoice requests time out until someone deletes it. Probes let the orchestrator ask the service itself, usually through a health endpoint like those in Health Endpoint Monitoring:

  • A liveness probe, such as GET /healthz every 10 seconds, restarts the container after several failures in a row. Keep it to whether the process can make progress: one that also queries the database restarts every pod at once when the database has a slow minute.
  • A readiness probe takes the pod out of the set that receives traffic while it fails, without restarting it, for instance while it warms up.

Swapping versions under traffic

Version 1.9 fixes the font. Changing the digest in the desired state starts a rolling update: with three replicas and a Deployment's default settings, Kubernetes creates a new pod, waits for the readiness probe the team configured to pass, removes an old one and repeats. Old and new serve side by side for a while, so an API or schema change has to work with both. If the 1.9 pods never become ready, the rollout stops with the remaining 1.8 pods serving; Kubernetes marks it failed after a deadline but does not roll back on its own, so rolling back means setting the previous digest again. Canaries and metric-driven rollback belong to Progressive delivery: blue-green, canary & rollback, and the pipeline that builds and ships the image to CI/CD pipelines: integrate, build once, promote.

TipOrchestration is the supporting cast Microservices need — it provides scaling, self-healing, and rollouts the gateway and service mesh build on. Whether to split a system into services at all is Microservices vs monolith.

Ship 1.9 on a Grid

Take the invoice service through the Code Deployment Platform Challenge, with Argo CD syncing a manifest that pins the image digest rather than a tag. The Container & Package Registry Challenge stays with the registry, where a tag must resolve tomorrow to what it resolved today.

Practice this