Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Ambassador

A sidecar that proxies an app's outbound calls: the app talks plain HTTP to a local port, and the ambassador adds TLS, timeouts, retries and metrics on the way out. Strongest when the app cannot be changed, and full of traps when its retries meet the app's own.

A program nobody can recompile

A pharmacy chain runs its stock system on a server in each of its regions. The system is a vendor's packaged application: when a medicine runs low it sends a reorder to the wholesaler's API, over plain HTTP, to a URL set in its configuration file. The chain can change that URL and nothing else, and the vendor's next release is nine months away.

Then the wholesaler writes to its customers. In three months its API will accept only mutual TLS, each customer presenting a client certificate. It also admits that at its morning peak a few percent of calls fail with 503 and succeed a moment later, and too many of those end as failed reorders for a pharmacist to resubmit by hand. The stock system cannot present a certificate, and it records nothing about how long the wholesaler takes to answer.

Change the address, not the program

The one thing the chain controls is the URL, so it points it at http://localhost:9000. On that port, on the same server and started with the stock system, runs a small proxy that receives each call and makes it to the wholesaler on the stock system's behalf. That proxy is an Ambassador: a helper that handles an app's outbound calls to a remote service, out of the app's process, so connectivity concerns live in the proxy and the app keeps calling a local address. It is a specialised sidecar, and the deployment shape, its shared lifecycle, its resource cost and the order in which it must start and stop, is the subject of Sidecar.

The stock system calls out through its ambassador

The stock system still speaks plain HTTP, now to a local port. Everything the wholesaler demands happens between the ambassador and the API.

Follow one reorder through it:

  1. The stock system sends POST /orders for 40 packs of a blood-pressure medicine to localhost:9000.
  2. The ambassador starts a 3-second timer for this attempt, opens a TLS connection to the wholesaler and presents the chain's client certificate.
  3. The ambassador forwards the order.
  4. The wholesaler answers 503; the ambassador waits about half a second, then sends the order again.
  5. The second attempt succeeds; the ambassador records both attempts, their latency and the outcome, and returns the 201 to the stock system.

The stock system saw one call and one answer. The chain also got something the vendor never shipped: a count of failed attempts and a latency for every call to the wholesaler, from every region, in the same form. When the wholesaler later moves its API to a new host, the ambassador's configuration changes and the stock system's does not. If the wholesaler stops answering for minutes, the ambassador can open a Circuit Breaker and answer the stock system at once with an error instead of making it wait out every timeout.

This is the strongest case for an ambassador: an app you cannot change, a packaged product or a legacy binary, that has to meet new demands on the network. When you own the code, a well-maintained client library is a fair alternative, and the ambassador has to earn its extra process. Its usual argument is the same as any sidecar's: one proxy, configured one way, for apps in several languages.

Where the retries pile up

The ambassador's retries were added to an app that already had opinions, and that is where it goes wrong. The vendor's documentation, read more carefully, shows that the stock system waits 10 seconds for an answer and, when none comes or the answer is an error, resubmits the reorder, up to twice. Three traps follow.

Retries multiply first. The original call and two resubmits, each going through an ambassador that tries up to three times, make up to nine calls for one reorder. At the wholesaler's morning peak, the moment calls fail most, every region now sends up to nine times its normal load of failing reorders. One layer should retry, with a limited budget; Retry covers how to bound it. The stock system's resubmits cannot be switched off here, so the ambassador's attempts are the ones to cut, and the last trap says by how much.

Next, a retried order can arrive twice. A GET for a stock level can be repeated safely. A POST /orders cannot: if the first attempt timed out after the wholesaler had created the order, the retry can create a second one, and then 80 packs arrive instead of 40. The ambassador should retry only calls that are safe to repeat. A POST qualifies when the wholesaler accepts a key that identifies the order, as Idempotency Key explains, and the ambassador can attach one, generated once per incoming call and reused on each of its own attempts. That protects against the ambassador's retries only: a resubmit from the stock system arrives as a new call and gets a new key, unless the key is derived from something the stock system sends every time, such as its own order number.

Last, the app can give up before the ambassador does. The stock system waits those 10 seconds. The ambassador allows 3 seconds per attempt, three attempts, and waits half a second, then a second, between them: up to 10.5 seconds in all. When the wholesaler is slow, the stock system abandons a call and resubmits while the ambassador's third attempt is still in flight, and the wholesaler receives the same order from two directions at once. The whole retry budget, every attempt's timeout plus every pause, has to fit inside the caller's own timeout, with room to return the answer; Timeout covers choosing those values. Here the stock system's resubmits stay on, so the ambassador makes one attempt of 3 seconds per incoming call, well inside the 10. Two attempts with a half-second pause, 6.5 seconds, would also fit, but only if the stock system stopped resubmitting and the order were safe to retry.

One more thing that can fail

If the ambassador crashes or is still starting, the stock system cannot reorder anything, and its error names neither the ambassador nor its configuration; keeping a helper running, restarted and started before the app is the business of Sidecar. One ambassador per server keeps that failure inside one region, where a single shared one would put every region's reorders behind it. A service mesh applies the same idea to the services that join it, which Service discovery & the mesh covers.

A proxy, not a second app

A month in, someone proposes that the ambassador also convert the wholesaler's product codes into the chain's own, since every order passes through it anyway. Converting another company's model into your own is real work, with its own tests and owners; it belongs in an Anti-Corruption Layer, in code the chain owns, not in a proxy nobody would look in for the chain's rules. Kept thin, the ambassador could be replaced by an off-the-shelf proxy such as Envoy configured for the same job, and the stock system would never know.

Watch outCommon mistake: letting the ambassador grow business logic instead of pure connectivity concerns. It should stay a thin, swappable proxy for retries, TLS and timeouts, not a second app.

Practice: what the stock system never sees

The Sidecar & Ambassador Challenge asks for an app that only calls its ambassador locally, with retries and timeouts living in the ambassador. Put the ambassador between the Service and the External System, and write its numbers on its edge to the remote: attempts, timeout per attempt, which calls it retries. Then write the app's own timeout on the local edge, and check three things: only one layer retries, every retried call is safe to repeat, and the ambassador's whole budget fits inside the app's timeout. If it does not fit, the app can give up while an attempt is still in flight, and if it resubmits, the wholesaler gets the same call twice on a slow day.

Practice this