Durable workflows: orchestration that survives restarts

The last line the job wrote

warehouse: deleting rows for 58123 is the last thing erase_account.py logged. The job erases a listener's account from a music-streaming service, which under GDPR has a month to do it, extendable by two more for complex requests. In order, it reads the account's email address, the key two outside processors file it under; deletes the playlists from the search index, then the account row, then the listening history in the warehouse; and asks the marketing service and the CRM to erase the address, the CRM confirming by webhook days later. During the warehouse step, a deploy restarted the host.

The job's progress lived in the memory of a process that no longer exists. Rerunning it fails: the first step finds no account, and the address the processors need went with the row.

Writing the place down

Progress has to live somewhere that outlives the process. A first version does it by hand: a status row per request, one queue message per step, each worker marking its step done and enqueuing the next. This job soon needs more: retries per store, a ten-day wait for the CRM that someone must notice running out, and a screen showing where each of 300 open requests stands. Built by hand, those are the parts of a workflow engine.

Durable execution is what such an engine provides: a multi-step process whose progress is persisted outside the process running it, so a crash, a deploy or a ten-day wait loses none of the recorded progress. Steps that touch the outside world run on workers; the engine records each step's outcome before the process moves on, and when a worker dies, another picks the run up from the last recorded point.

How the engine knows where it was

Replay-based engines, such as Temporal and Azure Durable Functions, let you write the workflow as ordinary code that calls each step through the engine's SDK, email = await lookupEmail(id), then await eraseSearch(id), and keep an event history per run. This is run erasure-58123 when the host restarts:

1. Run started: account 58123, deadline 14 November.
2. lookupEmail completed, with the address as its result.
3. eraseSearch completed.
4. eraseAccount completed.
5. eraseWarehouse scheduled, no result yet.

A worker that takes the run over executes the workflow function again from the top. A step whose result is in the history is not executed again: the call returns the recorded result, so lookupEmail hands back the address although the row is gone. The code reaches eraseWarehouse and waits, and the engine, seeing no result once that step's timeout expires, gives it to a live worker. Other engines, such as AWS Step Functions and Airflow, define the process as a state machine or a graph of tasks and store each step's state; recovery reads that state instead of replaying a workflow function.

The delete that ran twice

That retry is where the guarantee needs careful reading:

1. A worker asks the warehouse to delete the rows for 58123.
2. The warehouse deletes them and answers.
3. The worker dies before reporting the result to the engine.
4. The step's timeout expires, and the engine schedules it again.
5. Another worker sends the same delete.

With retries enabled, a step can run more than once: the engine starts new attempts until one's result is recorded or the policy gives up. With retries disabled the engine does not start it again, and step 3 surfaces as a failure your code must handle, although the rows were deleted. The history holds one result per step; the warehouse can still see the call twice. So steps must be idempotent: deleting rows already gone is harmless, and the marketing step should treat a not found on a second attempt as success. The step that emails the listener "your data has been erased" is not safe by nature; it needs a key the email provider deduplicates on, one of the tools Idempotency & exactly-once illusions compares for calls into systems you do not control.

Retries per step, and after the last one

Each step gets its own retry policy. The rate-limited CRM step backs off exponentially, 10 seconds, then 20, then 40, for up to a day, plus the jitter Retry adds against synchronised retries; the warehouse gets five quick attempts. A 403 from expired credentials will not heal, so it is marked non-retryable. A step also needs a timeout, because a call that hangs never fails on its own. When the last attempt fails, the error reaches the workflow code, which decides: here, park that store for a person to erase by hand, a Dead Letter Channel of one step, and carry on with the others, holding the listener's confirmation email until that store is done.

Ten days without a process

The CRM step waits for its webhook, or 10 days, whichever comes first. No thread sleeps through that. The engine stores a durable timer, a record with a fire time, beside the run's state, and the workers can be redeployed fifty times meanwhile. When the API receives the webhook, it sends the run an event, which Temporal calls a signal; the engine appends it to the history and resumes the run on any free worker. If the timer fires first, the run opens a ticket instead. A legal reviewer's approval, for invoices kept for tax purposes, is one more signal, a day or three weeks later. A durable timer waits inside a run that already exists; starting a run at 02:00 every night is a scheduler's job (Scheduled jobs & cron).

Code that must decide the same way twice

A function that runs again over its history must make the same decisions each time: it must be deterministic. Suppose it reads the host's clock directly: if (systemClock() > deadline) escalate(). Run a week before the deadline, it does not escalate; replayed after the deadline, it takes the other branch and asks for a step the history never recorded. Temporal detects the mismatch, and the run makes no progress until the code is fixed. So workflow code takes the time and random values from the engine, which returns the recorded ones on replay, and leaves every read and write of the outside world to steps.

Deploys get delicate too. Forty runs are waiting on the CRM when the team adds a step after the account delete; replaying an old history, the new code expects that step where the history shows the warehouse. Replay engines answer with versioning: a marker in the code that lets a run replaying past the change keep the path its history recorded, or old workers kept running until their runs finish. Engines that do not replay code escape this rule, not the care a changed definition needs.

Asking a run where it is

The history answers what nobody could for the script: each run's status, current step, and every step's input, output and attempts, searchable by status. Here it is the per-store record the company shows a regulator, and also a copy of the data being erased: the address lookupEmail returned stays in it for the retention period you configure, so keep personal data out of step results where an id will do, and encrypt the payloads that must carry it, like this one.

When a queue and a table are enough

An engine is another stateful system to operate or pay for, and every step is a round trip through it that persists its result, so a run of thousands of tiny steps is slow. For a job of a few steps, finished in minutes, with no waits and safe to rerun from the start, a queue and a status table are simpler and enough.

This run never has to undo anything, since a deleted row stays deleted; a process whose later failure must reverse earlier steps is a Saga, often run on an engine with each undo as one more step (Compensating Action). Without an engine, the coordination is hand-built, as a Process Manager reacting to messages or a Scheduler Agent Supervisor sweeping a state store for stuck steps, and the persistence, retries and timers above are yours to write.

Stop a worker halfway

The GDPR Erasure Pipeline Challenge asks for this very run (what it must find and erase, store by store, is in Deleting a person's data: erasure across every copy); Workflow Automation Platform and AI Agent Platform ask the same of user-built automations and of an agent's long task. Put the engine between the API and the per-store workers, stop a worker halfway through the deletes, and find the record that tells the next one where the run stopped, and where the ten-day wait lives. Then take each step that touches a store or a processor and say what happens when it runs twice: the engine cannot answer that one for you.