A laptop, a badge, and no login
A new engineer starts on Monday at a company of 3,000 people. Her badge opens the doors and her laptop is on the desk, but she cannot log in to anything. Onboarding a hire is one operation made of four steps against four systems the company does not run: create the account in the identity provider, add the hire to payroll, order a laptop from the supplier, and enable the building badge. The onboarding service started all four on Wednesday. Three answered. The identity provider replied 202 Accepted and promised a callback once the account existed, and the callback never came. Nothing in the onboarding service was waiting for it with a deadline, so nothing noticed.
Scheduler Agent Supervisor is a cloud design pattern for operations shaped like this one: several steps, each a call to a remote service that can fail, hang or go quiet. It splits the work three ways, and each part exists because of one way Wednesday went wrong.
A deadline written before the call
The Scheduler drives the operation. Before it starts a step, it writes a row for it in a durable state store: which step, its status (pending, done or failed), how many times it has failed, and a deadline by which it should be complete. Only then does it send the request. The order is the point: a step that has been sent always has a row, so a reply that never arrives leaves something behind, create-account pending since 09:02 with a deadline of 09:12, and the same row is there for a Scheduler that restarts.
The Scheduler does not call the identity provider itself. Each remote service sits behind an Agent, a small component that knows that one service: its API, its credentials, its rate limits, and how to retry a transient error with a timeout on each attempt, as Retry and Timeout describe. The agent reports the outcome to the Scheduler, which updates the step's row. The Scheduler knows the order of the steps and the agents know the remote systems, so adding a fifth step, say a seat in the chat tool, means one new agent and one new row in the plan.
Something has to read the deadlines
A deadline in a table helps nobody until something reads it, and on Wednesday no reply was ever going to arrive to trigger anything. The Supervisor is that reader. Periodically, every minute for instance, it sweeps the state store for steps that failed or are still pending past their deadline. It does not repair them and it does not call the agent. It asks the Scheduler to retry the step, or to compensate when retrying has stopped making sense, so that only the Scheduler ever changes the course of an operation.
The Scheduler writes each step and its deadline before calling. The Supervisor finds the overdue step in the store and asks the Scheduler, not the agent, to act.
With the three parts in place, Wednesday runs differently:
- The Scheduler records
create-account: pending, deadline 09:12and sends the step to the account agent. - The account agent calls the identity provider, which answers
202 Accepted. - The provider creates the account, but its callback is lost. The row stays pending.
- At 09:13 the Supervisor's sweep finds the step past its deadline and asks the Scheduler to retry.
- The Scheduler counts one failure, sets a new deadline and sends the step to the agent again.
A retry that finds the work done
In that replay the retry asks for an account that already exists. That is an ordinary outcome, not an edge case: a step that went quiet may well have succeeded, with only the news of it lost. So each step has to be safe to run twice. The account agent sends the hire's employee id with the request and treats "an account already exists for this employee" as success. Some duplicates cost money: a repeated laptop order is a second laptop, so that order carries a request id, the same on every retry, that the supplier deduplicates on. Idempotency & exactly-once illusions explains why no retry scheme makes this unnecessary.
Deadlines decide how often that happens. If the agent may spend three attempts of 20 seconds each plus its backoff on a step, a 30-second deadline sends the Supervisor after steps the agent is still working on, and repeats become routine instead of rare. Set each step's deadline longer than its agent's whole retry budget plus the longest the remote side normally takes to finish and call back, so an overdue step usually means something has really gone wrong.
When retrying stops helping
Some failures do not go away on the fifth attempt. The payroll provider keeps rejecting the hire's tax number because it was typed wrong, and only a person can fix that. The failure count in the store, or a rejection the agent reported as permanent, is what tells the Supervisor to stop asking for retries. It then asks the Scheduler either to park the operation and alert someone, which is usually right for onboarding, or to compensate, undoing the steps that already succeeded. Compensation earns its place when an operation must not stay half done: if the offer is withdrawn on Friday, the Scheduler disables the new account and cancels the laptop order, with the order of the undo steps, and the steps that cannot be undone, handled as Compensating Action and Saga describe.
Only one supervisor sweeping
Run two Supervisors so that a dead one does not leave overdue steps waiting, and both sweeps find the step pending past 09:12 and ask for a retry. Either one Supervisor is active at a time, chosen by Leader Election, or each sweep claims a step with a conditional update in the store before acting. Idempotent steps make an occasional double retry harmless, but they are the backstop, not the plan.
Who else does this job
A durable workflow engine keeps step progress, timers and retries in its own store and resumes after a crash: this shape, built for you (Durable workflows: orchestration that survives restarts). Nearby, a Process Manager routes on events, and a Saga undoes completed steps.
Put the sweeper on the Grid
The Distributed Job Scheduler challenge puts this pattern's recovery loop at system scale: jobs run at least once on many workers, with retries. Draw the job table as the state store, each job claimed by a worker with a deadline, and a sweeper that finds jobs whose worker went quiet past the deadline and hands them back to the scheduler, not straight to another worker. Then ask of your drawing what happens when two copies of the sweeper run.