One index, every clinic
A practice-management SaaS for veterinary clinics ran a single deployment for all of its customers: about 3,000 clinics, one fleet of API servers, one PostgreSQL cluster, one queue for appointment reminders. A routine release added an index to the appointments table, and the migration built it without PostgreSQL's CONCURRENTLY option, so the build blocked writes to the table until the index was finished. For forty minutes no clinic anywhere could book, move or cancel an appointment.
That migration is easy to fix. The shape behind it is not: everything is shared, so a mistake anywhere can reach everyone. And the same cluster was approaching a ceiling of its own. A stack can only grow so far: the largest database instance on offer, the connections one primary accepts, the jobs one queue drains. Each step towards that ceiling costs more, and all 3,000 clinics ride on every one of them.
Copies of the whole thing
A Deployment Stamp, also called a scale unit or, in cell-based architectures, a cell, is a complete, self-contained copy of the application and its data: its own API servers, its own database, its own queue and workers. Each stamp serves one group of tenants and nobody else. The clinic company splits its customers into stamps of at most 500 clinics, a size it has load-tested, and from then on it adds capacity by adding stamps rather than by making one stack bigger.
The next migration that locks a table locks it in one stamp, so 500 clinics wait instead of 3,000. A configuration pushed with a typo, a runaway batch job or a database that falls over stays inside its stamp. The set of customers a failure can reach is its blast radius, and stamps cap it at one stamp, on two conditions: the failure lives inside the stamp rather than in something the stamps share, and releases reach the stamps one at a time, with the rollout halted at the first stamp that shows the fault. A Bulkhead applies the same idea to one service's threads or connections; stamps put the walls around the entire stack.
Which stamp is clinic 2417 on?
Every clinic still uses the same web address, so something has to send each request to the stamp that holds that clinic's data, from a source of truth: the tenant directory, a small table that maps each tenant to its stamp. A routing layer in front of the stamps resolves the tenant from the request (a subdomain, or a claim in the session token), looks it up in the directory and forwards the request. Every request needs that answer, so the routers keep recent entries in memory for a short time.
The stamp router asks the tenant directory which stamp holds each clinic and forwards the request there. The large chain has a stamp to itself, built from the same code as the others.
Why a directory, and not a formula such as the tenant id modulo the number of stamps? A formula reassigns tenants whenever the number of stamps changes, and it cannot put one particular tenant on a chosen stamp: the new chain that needs a stamp of its own, or the clinic being moved off a crowded stamp. With a directory, placement is a decision made per tenant. New clinics go to whichever stamp has room, and a tenant can be moved later. Routing tenants with no mapping layer at all, by hard-coding a stamp per customer or per DNS name, leaves no clean way to move a tenant to a healthier or less loaded stamp. Splitting one database by a key is Sharding & partitioning; stamps make the same cut through the whole stack, with the tenant as the key.
How big a stamp is, and when to add one
The 500 comes from a load test, not a guess: one stamp was driven at rising load until it missed its latency targets, and the limit was set well below that point. When the open stamps near their limit, the next one is created by the same templates and pipeline that built the others, so no stamp drifts from its siblings. Tenants are not equal: a chain that signs up as one tenant with 400 clinics would fill most of a stamp, so it gets a stamp to itself, running the same code and the same release as every other stamp. Inside a shared stamp the clinics still share everything, and one clinic's bulk import can still slow its 499 neighbours; data isolation and the defences against a noisy neighbour belong to Multi-tenancy.
Moving a clinic is a data migration
Stamp 2 fills faster than planned because a few of its clinics grew. Moving clinic 2417 to Stamp 4 looks like a one-line change to the directory, and that line alone would send its requests to a stamp that holds none of its appointments. The data moves first:
- The migration tool copies clinic 2417's rows from a consistent snapshot of Stamp 2, noting its position in Stamp 2's change log, while Stamp 2 keeps serving the clinic.
- Stamp 4 applies the changes the clinic made after that position, read from Stamp 2's change log, until it is seconds behind.
- Stamp 2 blocks writes for clinic 2417 on every path, workers included, waits for transactions in flight to finish, and enforces the block itself, whatever a router's cache says.
- Stamp 4 applies the last changes, and the tool compares row counts and checksums on both sides.
- The operator changes the clinic's directory entry to Stamp 4.
- Stamp 2 refuses any request for the clinic that still arrives through a stale cached entry, until those entries expire.
- Stamp 2 deletes its copy after a grace period.
Each step has a price. The copy takes hours for a large tenant and loads the stamp it is leaving. The clinic cannot write between steps 3 and 5, for seconds or minutes. Files in object storage, queued jobs, reminders scheduled for next week and search indexes have to follow too, or the clinic arrives missing pieces. And a write that gets past the block in step 3, from a job that missed it or a request through a stale entry, makes the two copies diverge. Moves are possible and should be rare, which is why placing a tenant well when it signs up matters.
Releasing one stamp at a time
Stamps give every release a natural order, a stamp of the company's own test clinics first and the rest in waves, so the index migration from the first paragraph would have stopped at the first stamp. For hours or days two versions run side by side, so anything that talks to more than one stamp must accept both; when a wave may proceed and how to roll one back belongs to Progressive delivery: blue-green, canary & rollback.
What stays above the stamps
A few things cannot live inside a stamp: the tenant directory; global identity, since a vet typing an email address on the login page has not yet said which clinic, let alone which stamp; and billing, which adds up usage across all of them. That layer sits on or beside the path of every request, so a failure there can reach every stamp and put the blast radius back at all 3,000 clinics. So it is kept small: little logic, few changes, released on its own and more carefully than any stamp, and built so that the stamps can ride out a short outage of it: routers keep serving from the directory entries they already hold, and a signed session token can be checked inside the stamp without calling the identity service.
Give the biggest customer its own stamp
Two requirements of the Multi-Tenant SaaS Platform Challenge are stamp questions: give a large tenant dedicated capacity without forking the deployment, and stop one tenant's spike from degrading everyone else. Draw the API Gateway asking a directory (a Config Service works) for the stamp, then a shared stamp and a dedicated one, each an application service with its own database, both built from the same release. A stamp contains a spike only for the tenants outside it, so the shared stamp still needs a per-tenant Rate Limiter.