A balance column cannot explain itself
The support ticket says Ana's balance is 30.00 short. The first version of the wallet keeps one row per user with a balance column, and a transfer subtracts from one row and adds to another. The column holds a number and no history: nothing to recompute it from, nothing that says which update was wrong. A mutable balance leaves you guessing, and at this wallet's volume a wrong number is a matter of when.
The volume: 50 million people topping up from a card, paying each other and paying merchants, say 100 million transfers a day. That is about 1,200 a second on average, 10× at peaks; at two ledger lines and around 400 bytes per transfer, about 40 GB a day and ~15 TB a year, real but split by account and mostly cold. A relational database handles that throughput comfortably. The hard part is that every unit of money must be traceable and none may be created or lost, while clients retry, transfers race each other, and a handful of accounts, the merchants and the fee account, receive a large share of all writes.
Write down movements, not balances
So the truth is an append-only ledger: a journal of transfers and an entries table where each transfer posts at least two lines that sum to zero, debit Ana 500, credit Kofi 500, in integer minor units. Never floats: 0.1 + 0.2 is not 0.3 in binary floating point, so store integer cents and a currency, and round in one documented place. Nothing is updated or deleted; a mistake is corrected by a new reversing entry. The balance is a derived view: by definition, the sum of an account's entries. Summing on every read is too slow, so a balances table is maintained in the same transaction as the entries, a cache that can always be rebuilt from the ledger, and periodically is, to prove it matches.
One transfer, one short transaction
Ana pays Kofi 5.00, which the ledger stores as 500. The wallet service runs one transaction:
- It inserts the journal row, with the client's idempotency key in a unique column.
- It inserts the two entries, −500 for Ana and +500 for Kofi.
- It inserts an outbox row for the transfer completed event.
- It runs
UPDATE balances SET amount = amount − 500 WHERE account = ana AND amount ≥ 500, and if that updates zero rows, rolls back with insufficient funds. - It adds 500 to Kofi's balance row and commits.
Ana's phone loses the response and retries. The retry's insert hits the unique constraint, and the service returns the stored result of the first attempt instead of making a second transfer: the ledger sees one transfer. Without the key, a lost response followed by a retry is a double payment. What matters is that the key is enforced inside the transaction that moves the money. A Redis cache of recent results can answer an obvious retry sooner, but it is only a hint, since an entry can be missing or evicted; the unique column is what makes a retry harmless.
The overdraft check is a predicate on the update itself, not a read followed by a write. Read first, and two transfers of 500 from Ana's 700 can both see 700, both decide 500 fits, and both debit, leaving her at −300. With the predicate on the update, the second waits for the first's row lock and then finds 200, which fails amount ≥ 500; Transactions & isolation levels explains why the check must live there. The outbox row makes the event part of the same commit, and a relay publishes it afterwards, so a crash cannot leave a transfer that nothing downstream hears about (Transactional Outbox).
Top-ups follow the same shape, with one rule: charge the card first, outside the transaction, then post the entries, crediting the user and debiting a clearing account for the processor. A call to the card processor inside the ledger transaction holds row locks on the hottest table you own for as long as a third party takes to answer. Talking to the processor itself is the ground of Payments & billing integration.
The account everyone pays
A café chain's merchant account receives 200 payments a second at lunchtime. Each of those transactions updates the same balances row, so every payer waits on the same lock: with each transaction holding the lock for a few milliseconds, strict per-account serialisation caps a single account at a few hundred transfers a second. The way out starts from a difference between credits and debits.
A credit can never fail an overdraft check, so credits to a hot account do not need the synchronous balance update at all: a late credit only leaves the row below the real balance. Post the entries, and let a consumer of the ledger stream fold credits into the balance asynchronously. It must do so idempotently, recording each applied entry id under a unique constraint in the same transaction as the balance update, because the stream delivers at least once and a replayed batch must not credit twice. Idempotency & exactly-once illusions covers both guards, the key on the way in and this one on the way out.
When debits are hot as well, as on a fee account paying out, split the balance into sub-accounts: N rows that sum to the available balance. Each debit tries one at random and moves on to another when that row is short; before declining, it checks the account's total across the slices and, when no single row covers the amount, takes from several, the check and the debit in one transaction that holds the account's slice rows locked, so a decline reflects the available balance, not the slice it happened to hit. A credit still waiting to be folded in is not yet spendable. A rebalancer keeps the slices topped up so that this fallback stays rare.
Async credits buy throughput with a balance that is momentarily behind its ledger, a lag you expose deliberately, as pending, rather than hide; sub-accounts keep the sum exact, at the price of a rebalancer. A third way, giving the account one owner that applies its transfers in sequence, is the design of Single-writer sequencing & matching engines; a wallet seldom needs it, since no rule here makes the earlier payer win.
Proving nothing leaked
Reconciliation runs against the ledger tables themselves, not the stream, so an event that was never published cannot hide a gap. It asks three questions:
- Does every account's
balancesrow equal the sum of its entries minus the credits not yet folded in, and is any credit pending too long? - Does the sum of all entries equal zero in each currency?
- Do our card top-ups match the processor's settlement file?
Any mismatch is an incident, and the ledger, not the balance, is what you trust to resolve it. The third question exists for one failure in particular: a card is charged, and the ledger write that should follow fails. The money is in limbo until reconciliation matches the processor's settlement against the missing entry and posts it.
Failover is the other quiet leak: if the replica promoted after a crash never received a transfer the payer was already told went through, that transfer disappears from the live ledger; Replication & read replicas explains synchronous and asynchronous copies.
The wallet Challenge
The Digital Wallet & Ledger Challenge is this system, from top-up to daily reconciliation. The edge that decides the design is the one from the service to the balances: a good run shows one transaction with the key's unique column inside it and the predicate written on that edge, and a reconciliation job reading the ledger tables rather than the stream. Payment Processing System has the same idempotency and reconciliation core, with the external processor at its centre instead of the ledger.