Where is application A-2291?
A landlord calls a letting agency on Thursday. Someone applied for her flat on Monday, the listing still says "under offer", and she wants to know what is holding it up. The agency's platform handles applications as a chain of reactions:
- The application service publishes ApplicationSubmitted.
- The identity service hears it, checks the applicant's ID and publishes IdentityVerified.
- The referencing service hears that, contacts the previous landlord and the employer, and publishes ReferencesPassed once both have answered.
- The lease service hears that and sends the lease out for e-signature.
A developer searches four services' logs and finds IdentityVerified at 10:14 on Monday, then nothing. Did the referencing service miss the event, or is the employer not replying? No service can say, because none owns the application as a whole. That style is Choreography, and for a short flow it is fine. Then the business adds a rule: applicants under 21 need a guarantor before the lease goes out. The rule is about the process, not about identity, references or leases, and in this design the process has nowhere to live.
One instance per application
A Process Manager, a pattern from the Enterprise Integration Patterns catalogue, gives the process a home. It is a central component that keeps the state of each running process, receives the events the participants publish, and decides from that state which command to send next. When ApplicationSubmitted arrives, it creates a process instance for A-2291: a record with fields such as identity pending, references pending, guarantor not yet known, lease not sent. Every running application has its own.
The participants get simpler: the identity service executes VerifyIdentity and replies with an event, without knowing what comes next. The flow now lives in one place, explicit enough to read and to test, and the landlord's question is answered by reading one row.
The instance's state decides the command: verified, references passed, and 19 years old means a guarantor first.
Which instance is this for?
A ReferencesPassed event is useless until the process manager knows which application it belongs to. Every command it sends carries the application id, every participant echoes that id on the events it publishes, and the process manager loads the instance by it: the application id is the process's Correlation Identifier. When an outside party cannot echo your id (the referencing agency files the request as case RF-88310), the instance records that number on acknowledgement, so a reply carrying only RF-88310 still finds A-2291.
Two awkward cases follow. An event for an instance that does not exist is logged and set aside, not acted on; only the starting event creates one. And when the broker delivers ReferencesPassed a second time, the copy finds references already passed and changes nothing, so, as long as commands go out only when the state changes, it sends nothing. In this flow, where no step runs twice, the instance's state does the job of an Idempotent Consumer; a step that can be repeated also needs the message ids recorded.
Deciding what happens next
Here is A-2291 in the new design, event by event:
- ApplicationSubmitted arrives. The process manager creates the instance and sends VerifyIdentity and RequestReferences together, since neither depends on the other.
- ReferencesPassed arrives first. The instance now reads references passed, identity pending. There is nothing to send yet.
- IdentityVerified arrives, with a date of birth that makes the applicant 19. Both checks are done and a guarantor is needed, so the process manager sends RequestGuarantor instead of SendLease.
- GuarantorApproved arrives. The process manager sends SendLease.
Each decision is a function of two things: the instance's state and the event that just arrived. Because it waits for state rather than a fixed sequence, the references may come back before the ID check; and because the decision is one function, a test can feed it events and assert on the commands that come out. A Routing Slip fixes the list of steps when the message starts out; a process manager is for routes that depend on intermediate results, like the guarantor branch.
Notice that nothing here undoes anything. A process manager routes any multi-step flow. When a flow must also reverse steps that already committed, the compensations and the half-finished state belong to Saga; an orchestrated saga is usually built as a process manager, while a saga driven purely by events is choreography.
One home for the flow has a price: participants' rules tend to creep into it. Keep it to sequencing; the referencing service decides whether references pass, and the process manager decides what happens because they did.
Two writes and a race
Kept only in memory, the instances vanish with the next redeploy, and the process manager no longer knows which step any application had reached. The instance is saved each time an event is handled, so a restart resumes from the last recorded step instead of replaying from scratch; surviving a crash in general is the ground of Durable workflows: orchestration that survives restarts. Two details belong to the process manager itself:
- Saving and sending are two writes. A crash after saving "guarantor requested" and before sending RequestGuarantor leaves the instance waiting for a reply to a command nobody sent. Writing the command to a Transactional Outbox in the same transaction as the state closes that gap.
- Two events for one instance can race. If two workers handle IdentityVerified and ReferencesPassed for A-2291 at once, each loads both checks pending and concludes the other is still outstanding, so nobody sends RequestGuarantor (depending on how they save, one half may even overwrite the other). Handle one instance's events one at a time, for example by routing on the application id, or save with a version check and retry the event that loses.
The signature that never comes
The lease went out on Tuesday at 17:00, and the applicant has gone quiet. Nothing will ever tell the process manager about an event that did not happen, so it has to set its own deadline. When it sends SendLease, it records a deadline 72 hours ahead and asks the scheduler, the machinery Scheduled jobs & cron explains, to deliver a LeaseDeadline message for A-2291 at that moment; that request goes through the outbox like any command. When it arrives, the instance decides: if LeaseSigned is already recorded, the deadline is stale and ignored; if not, the process manager sends ReleaseOffer, so the flat goes back to the next applicant, and notifies the agent.
Every wait gets a deadline and a plan for when it passes. References not back in five working days might not cancel anything; they open a task for a person, which is escalation rather than failure. And the deadline races the signature like any other event: if the signature lands a few seconds after the release has been processed, the instance must say what happens to a lease signed for an offer that has already lapsed, rather than silently sending it on. Instead of one timer per wait, a Scheduler Agent Supervisor sweeps the state store for instances stuck past their deadlines.
Leave a deadline in the drawing
Draw a flow with a branch and a wait in the Process Manager challenge: one Process Manager node holding the state of a single long-running process, participant services that only execute commands and emit events, an arrow from each event to the command it triggers, the instance persisted in a Data Store, and a path for the expected event that never arrives. Before you submit, ask the landlord's question of your drawing. If answering it means following arrows through several services, you have drawn choreography again.