Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Webhooks & third-party integration

A webhook is an HTTP request another system sends you when something happens. Receiving one means verifying its signature, answering fast and processing it so that a second copy or a late arrival does no harm; sending them means signing, retrying with backoff, and keeping one dead endpoint from delaying everyone else's events.

A request nobody on this side made

At 09:14 the order server of a small coffee roaster, which sells bean subscriptions online, receives a POST to /hooks/payments. Nothing on the roaster's side asked for it. The JSON body says that event evt_7Q2, of type payment.succeeded, concerns order R-2291 for €38. A header, Provider-Signature, carries a timestamp and a long hexadecimal string. The sender is the roaster's payment provider, reporting a card payment that went through, and it expects an answer within seconds.

That request is a webhook: an HTTP call one system makes to another when something happens, to a URL the receiver registered in advance. Without it the roaster would poll, asking every few seconds about each pending payment and mostly hearing "not yet". Payment providers are not the only callers: an SMS provider reporting that a text reached a phone calls back the same way (Notifications & transactional messaging).

The price is a public endpoint, called by a system the roaster does not control, on that system's schedule. What follows takes evt_7Q2 from both ends: the roaster receiving it, then the provider sending it.

Is it really from the provider?

Anyone who learns the URL can POST a forged payment.succeeded and have coffee shipped for free, so the provider signs each request. At registration the two sides got a shared secret; the provider computes an HMAC, a keyed hash, over the timestamp and the exact bytes of the body, and the roaster recomputes it with its copy of the secret and rejects any request whose result differs. Three details decide whether the check protects anything:

  • Hash the raw body. Parsing the JSON and serialising it again can change spacing or key order, and genuine requests stop matching; a framework that parses the body early is a common cause.
  • Check the timestamp. It is part of the signed content, so a genuine request captured and sent again an hour later still carries a valid signature, but an old time. Rejecting anything older than a few minutes limits that replay to a short window, and inside it the event-id check described below keeps a copy from repeating its effects; a sender's own late retries pass this check only if it signs each attempt with a fresh timestamp, as schemes built this way do.
  • Compare in constant time. A comparison that stops at the first differing byte can leak, through its timing, how much of a guessed signature was right. Crypto libraries provide a constant-time compare for this.

The secret is a credential like a database password, and it is stored the way those are (Secrets management & encryption). Some providers sign with a private key and publish the public key instead, so the receiver holds no secret at all; which bytes are signed, and whether a timestamp is among them, then follow that provider's documented scheme.

Answer first, work later

A genuine payment.succeeded means real work: mark the order paid, reserve beans, buy a shipping label, email the customer. The roaster's first handler did it all before answering. One morning the label service took twelve seconds, the provider stopped waiting after ten, recorded the delivery as failed and sent the event again a few minutes later. The beans were reserved twice.

So the handler now does three things: verify the signature, write the event somewhere durable (a queue, or a table of received events), and answer 200. A worker takes it from there, and the provider gets its answer in milliseconds whatever the label service is doing; the worker side of that hand-off is in Message queues & async workers.

Watch outStore the event before you answer. A handler that returns 200 first and crashes before the event is saved has told the provider it was delivered, and the provider will not send it again.
Verify, store, answer, then process

The ingress only checks the signature and enqueues. The worker applies the event and records its id in the same transaction, so a second copy changes nothing in Orders.

Twice, late, or not at all

Answering fast makes duplicates rarer, not impossible. The roaster's 200 can be lost on the way back, or arrive just after the provider stopped waiting, and the provider cannot tell either case from a failure, so it sends again. A sender that retries this way delivers at least once, provided one attempt gets through before its retries run out, so the same event can arrive more than once, and the receiver makes a second copy harmless: the worker records each event id in the same database transaction as the event's effect, under a unique constraint on that id, so the database rejects a second copy even when two workers handle it at the same moment. That makes the database changes happen once; the shipping label and the customer email are calls to other systems, outside that transaction, so each needs its own guard against running twice. Idempotency & exactly-once illusions explains both.

Order is not promised either. A customer pays, and an hour later the roaster refunds a split bag. If the first delivery of payment.succeeded failed and its retry lands after payment.refunded, a handler that applies events as they arrive marks the refunded order paid again. Two defences, best combined: treat each event as a hint and fetch the payment's current state from the provider's API before acting, and apply the result through a state machine that refuses to move backwards, checked in the same database update that writes the new state, so a refunded payment never becomes paid even when two workers' fetches land in the wrong order. Payments & billing integration covers why the provider's webhook, not the charge response, settles a payment, and the Payment Processing System challenge asks for webhook handling that survives retries and events out of order.

Some events never arrive, because an outage on the roaster's side outlasted the provider's retries; a periodic job that compares the provider's payments with what was processed, over a window reaching back past the outage, catches those.

The other end: forty thousand URLs

Now the provider's side. Its delivery system serves forty thousand merchants and POSTs each event to the endpoints that event's merchant subscribed to its type, any of which may be slow, down or misconfigured. Each endpoint gets its own signing secret, so a secret leaked by one merchant forges nothing for the others, and each event is written in the same transaction as the payment change it reports, as in Transactional Outbox, so a crash cannot lose it before delivery starts.

Per-endpoint queues, retries on the side

Each endpoint has its own queue and a cap on deliveries in flight. A failed attempt waits in the retry scheduler, attempts that are used up go to dead letters, and every attempt is logged.

A dead endpoint is everyone's problem

When an endpoint times out or answers with a server error, the sender tries again later, with growing, randomised waits that can stretch over a day or more (Retry); retrying at once and inline turns a wave of failing endpoints into a storm of repeat calls, the starting point of the Diagnose: Webhook Delivery Retry Storm challenge.

Retries leave a trap of their own. Suppose all deliveries share one queue and a pool of 200 workers, with a 30-second timeout. A large merchant's endpoint starts hanging. Its events are a tenth of the traffic, but each holds a worker for 30 seconds where a healthy delivery takes 100 milliseconds, so once there is enough traffic to keep the pool busy, nearly every worker soon waits on that one merchant while everyone else's events queue behind. That is head-of-line blocking. The fix is per-endpoint isolation: each endpoint gets its own queue and a cap on deliveries to it in flight, small next to the pool, and the dispatcher hands a worker only deliveries for endpoints below their cap, so no worker waits on a full endpoint and a dead one ties up only its own share. A short timeout helps, and an endpoint that has failed for long enough is paused, and its owner told. The general shape, one slow dependency draining a shared pool, is described in Cascading failures: how one dependency takes down its callers.

The URLs are typed in by customers, too, and one pointing at an internal address would have the provider's own servers make requests inside its network; delivery workers check the address they actually connect to, the defence against server-side request forgery set out in Web application security: injection, XSS, CSRF & SSRF.

Dead letters, logs and replay

After the last attempt, a delivery moves to a dead-letter store with its last error instead of disappearing (Dead Letter Channel). Every attempt is logged with its time, response code and duration, which is how a merchant finds out why orders stopped updating, and the same log drives manual replay of one event or everything since a given time once the endpoint is fixed. Replay resends events the receiver may already have, one more reason the roaster's worker skips ids it has seen.

The Webhook Delivery System challenge asks for this sending side, head-of-line blocking included. Once your design is on the Grid, suppose one subscriber endpoint starts hanging and trace it through: check that another endpoint's events still leave on time, and that the hanging endpoint's events end up delivered or in dead letters, never silently dropped.

Practice this