Four teams, one event
Getting a new engineer ready for day one at a 900-person software company takes work from four teams, and no service in the company runs that work. It starts when the People team marks a candidate as hired and the HR system publishes an event called EmployeeHired, carrying the new employee's id, start date, team and office.
Three services subscribed to that event, and each does its own part when it arrives:
- IT creates the email account, orders a laptop, and publishes AccountCreated.
- Facilities assigns a desk in the right office and queues a badge to print.
- Payroll opens a pay record from the start date.
The list has no numbers because these reactions do not wait on each other, and no single component holds the steps of onboarding. The work moves forward because each service reacts to facts that other services announce, and that way of running a multi-service workflow is called Choreography: no coordinator, just services listening for their cue.
No arrow tells a service what to do. Each one subscribes to a fact and decides its own reaction; Security training hangs off IT's event, not HR's.
How an event differs from a command, and what a publisher gives up by announcing to an audience it cannot see, is ground that Publish / Subscribe owns. Choreography is what you get when a whole multi-step workflow is built out of those reactions, and the rest of this article is about what that does to the workflow.
The training nobody had to approve
Security wants every new hire to finish phishing training in their first week. The training service needs an email address to enrol someone, so the Security team hangs the new step off AccountCreated rather than EmployeeHired. No central definition of onboarding has to change, and no other team is in the meeting: the workflow gained a step owned by the team that wanted it.
That is the appeal for a workflow: each team owns its step and ships it on its own schedule, and no central component has to learn every team's rules. The same freedom works in reverse. The step now sits on IT's event, so if IT stops publishing AccountCreated, the workflow loses its training step without anyone deciding to remove it.
Ask where onboarding is defined
Six months later, a new engineer asks to see the onboarding workflow. There is no single file to open: the workflow is the sum of the subscriptions in five codebases, and the wiki diagram from the spring has no training step. In a choreography the flow is implicit: you can read each service's reaction, but no one place shows the sequence they add up to.
That bites hardest when one instance goes wrong. Dana starts on Monday, and on Friday her manager writes that she still has no laptop. Did IT receive EmployeeHired? Did the laptop order fail, or never reach the vendor? Each service's logs answer for its own step, and only if you can find Dana in them. Two habits make that search possible:
- One id on every event. Every event about Dana carries her employee id, including the events published in reaction to other events, so each service's logs and failed messages can be searched by it. That id is a Correlation Identifier.
- Trace context carried through the broker. If each service copies the trace context from the event it consumed onto the events it publishes, a trace can show EmployeeHired and the reactions that followed it, the account, the laptop order and the training enrolment, as one tree with timings. How that context travels, and which traces are worth keeping, is explained in Distributed tracing.
Both show what happened. Neither shows what did not. If IT never consumed the event, there is no span to find and no log line with Dana's id, and since no component is waiting for the laptop, nothing times out. In a choreography, a missing reaction is silent unless you build something to notice it. Teams that run one for long often add a tracker that only watches: one more subscriber that listens to every onboarding event, keeps a row per hire (account created, desk assigned, laptop shipped), and raises an alert when someone starting within three days still has a gap. It sends no commands, so the flow is still choreographed; it just gives the question "where is Dana?" one place to be asked.
An update that comes back around
Event chains can also close into loops, each link reasonable on its own. HR publishes EmployeeUpdated whenever a profile changes; IT copies the name onto the account and, after every account write, publishes AccountUpdated. HR subscribes to AccountUpdated to catch accounts that drift: when the name there differs from the profile, it publishes EmployeeUpdated again with the profile's spelling, to put the account right.
Dana fixes the accent in her surname on her HR profile. IT strips accents to suit an old mail system, so its AccountUpdated carries the name without one; HR sees a mismatch and sends its spelling again; IT strips it again, and each correction triggers the next for as long as both keep running, filling the broker with events. Nobody wrote a loop; it is the sum of two subscriptions in two codebases, so it tends to be found in production, not in review. The usual defences: publish only real changes, so IT, whose stored name did not change, stays quiet; give each field one owning service and have the others copy it without correcting it back; and have every event in the chain carry the origin of the change that started it, copied from the event it reacts to, so HR can recognise an AccountUpdated that its own update caused and ignore it.
The rule that belongs to nobody
Then legal adds a rule: no badge until the background check has passed and the contract is signed. Two different services publish those facts, in either order, and the check can take two days or not come back at all. Facilities could subscribe to both, store per hire which ones have arrived, and print when both are in. It works, and Facilities now holds the onboarding's state and enforces a rule that is not about desks. Add a deadline, such as escalating when the check is not back by the Thursday before the start date, and Facilities has become a coordinator in all but name, hidden where nobody would look for one.
That is the point at which this part of the flow is worth moving to a coordinator: a rule that joins several services' results or waits against a deadline, or an event chain so long that nobody can trace the end-to-end workflow without reading every service's code. A Process Manager can keep one instance per hire for the badge gate, and the move is partial: Payroll and Security training go on reacting to their events as before. When the coordinated part must wait for days and survive restarts, Durable workflows: orchestration that survives restarts describes the engines built for that. Undoing work after a withdrawn offer pulls the same way, and that design belongs to Saga, which can run either way: by events between the services or by a central Process Manager sending commands.
Practise the contrast
No Challenge exercises choreography on its own yet. The closest is the Process Manager Challenge, which practises the coordinator and asks you to make its intent clear against event choreography; its palette includes a Choreography node. Before you draw, list the parts of your flow that no participant would naturally own in a choreographed version, such as a join of two results or a deadline, and make those the Process Manager's job. Then use the Choreography node to mark one reaction you would leave choreographed, a step that only needs to hear the event, and note why it needs no coordinator.