Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Audit logs & tamper evidence

What an audit record must hold, capturing it so no change skips it, storage the audited people cannot write to, hash chains anchored outside the store and what they prove, retention against erasure, and answering an auditor years later.

Who changed it, and can you prove it?

Who changed Marta Okafor's bank account on the 27th, and can you prove nobody has edited that record since? The question comes from the compliance team of a logistics company that runs its payroll on a SaaS platform. Marta, a warehouse supervisor, was paid 3,412 euros on the 28th, into an account she has never held. She says she changed nothing.

The platform's application log answers neither half. Whatever line it holds for the change, if it survived sampling and the two-week expiry, names the session's user, who may not be the person who acted, and anyone with write access to the log store could have edited it yesterday. Those logs serve the engineers running the system, as Observability: metrics, logs & alerting describes. An audit log is for someone outside it: a compliance team, a regulator, a court, or you during an incident. It has to be complete rather than sampled, kept for years rather than weeks, and hard for an administrator to edit without it showing, because a record anyone with production access can rewrite proves nothing.

A record built for the first half

Here is the record the payroll service wrote for that change:

  • Actor: agent:dmitri.v, of the payroll company's own support team.
  • On behalf of: employee:8812, Marta. Dmitri was in the support console's act as user mode, so to the application the change looked like Marta's own.
  • Authority: support ticket SUP-7730, opened after a phone call from someone who gave Marta's name and date of birth.
  • Action and target: bank_account.update on employee 8812 of tenant t-0417, with the last four digits of the old and new accounts, not the account numbers.
  • When and from where: the 27th at 16:52:07 UTC by the server's clock, never the client's; the console session's id and the agent's network address.
  • Outcome: allowed, and the change committed. Two attempts an hour earlier through the employee app are recorded too, as denied: they failed the second-factor check.

Record only Marta's identity, the one the session was using, and the trail would blame her, as the application log did; drop on behalf of and it would lose that Dmitri was acting as her. Without authority, nobody could say why an agent changed a bank account on a phone call; whether the role should permit that at all belongs to Authorization: roles, relationships & policies. Failed attempts are often the more interesting half: the denials make this look like a campaign, not a slip. The fields are structured, because Admin updated settings names no admin, no setting and no values, so nothing can filter on it or join it to anything else. And the record holds the event, not the raw request: before and after values for small objects, a reference to a write-once copy, with its hash, for large ones, never a credential or a full account number.

No change without its record

Capture cannot depend on a developer remembering to log after the update. The payroll service writes the audit event in the same database transaction as the change: an outbox row committed with the bank-account update, so both commit or neither does, and a relay ships the rows on (the pattern has its own article, Transactional Outbox). A denied attempt changes nothing, so the service commits the denial's outbox row in a transaction of its own before it answers. The relay can deliver a row twice; skipping a repeat by its event id is covered in Idempotency & exactly-once illusions.

The tempting alternative is writing to the audit store synchronously inside the request. That couples the action to the audit store's availability: when the store is down, you choose between refusing every payroll change and making changes nobody recorded. Recording the event atomically with the action, then shipping it asynchronously, removes that choice. What the outbox cannot see is a change made around the service: an engineer's hand-run UPDATE writes no audit row, so direct access to production data needs its own trail, such as the database's audit logging or a break-glass procedure that opens a ticket first and records every command the session runs.

Captured with the change, chained, anchored elsewhere

The bank-account update and its outbox row commit together. The chain writer appends each event to the audit store, signs the chain head with a key held in another account, archives sealed segments to write-once storage and feeds the search index the query service searches before reading the records themselves from the audit store.

Hashes that notice an edit

The chain writer's role on the audit store can append and nothing else. That stops a bug, not the people who grant permissions: append-only is a setting, and whoever holds the store's admin credentials can lift it, edit record 81,204, and put it back. So the store lives in a separate account, run by people who are not the ones being audited, and the account that stores and signs is not the one that runs the application: a single compromised credential cannot both act and erase the evidence of acting. That narrows who could tamper; it does not prove nobody did.

Tamper evidence means a change to the history leaves proof, even when made with the store's credentials. The chain writer gives every record a sequence number and stores with it a hash of the record's contents together with the previous record's hash. Record 81,204's hash covers 81,203's, which covers 81,202's, back to the first. Change one byte of an old record and its hash no longer matches, and neither does any link after it.

A chain alone has a hole: someone who can rewrite the store can recompute every hash from the edit onward, and the result checks out. So every hour the chain writer takes the newest hash, the chain head, signs it with a key in a separate account's key management service (keys and their custody: Secrets management & encryption), and copies it out of the store's reach: to write-once storage in another account, and to each customer's compliance contact. A rewrite now has to match a value someone else already holds. A database with a built-in chain still needs its digest kept where its own administrators cannot reach it.

The proof has edges. A chain that verifies against its anchors shows that no record up to the last anchor was altered, removed or reordered after that anchor was taken; until its first anchor, a record rests on permissions alone. It does not show a record was true when written: a wrong actor, recorded, is kept faithfully. It does not show every action was captured, because an action that produced no record leaves no gap; that is the capture's job. And it prevents nothing: it makes tampering detectable, by whoever checks. The payroll database stays the truth about Marta's account and the audit log is testimony about it, unlike Event Sourcing, where the event log is the state.

Seven years, and a request to be forgotten

Payroll records are commonly kept for years under tax and employment law, which is no job for the hot database's storage tier. The chain writer seals the chain into daily segments and copies each to object storage under an object lock (write-once) retention policy; in its strictest mode the storage refuses to delete or overwrite a segment before its retention date, even for the account's administrators (more on that tier in Object storage & blobs). A columnar search index in front answers a query over any range without restoring an archive.

Two years later Marta leaves and asks for her data to be erased. The records never named her: they say employee:8812, and the mapping to her name lives in the payroll database, outside the chain, so erasing it breaks no hash. Records that can still be tied to her remain personal data under GDPR, so keeping them needs a legal basis, such as a legal obligation or legitimate interest, and a retention period, both written down, and her request is weighed against them rather than granted or refused by reflex. Which fields count as personal is a question for Data governance: catalogs, lineage & classification, and every other copy of her data is handled in Deleting a person's data: erasure across every copy, which the GDPR Erasure Pipeline Challenge puts into practice.

The auditor's question, answered from the log

The answer to the compliance team comes from the trail, not from anyone's memory:

  1. The query service searches the index for every record whose target is employee 8812 between the 20th and the 28th, and finds the two denied attempts and Dmitri's change.
  2. It reads those records, and the ones around them, from the audit store itself, because the index is a copy and proves nothing.
  3. A verifier recomputes every hash from the last signed chain head before the 20th to the first one after the 28th, finds the sequence numbers unbroken, and checks that every record from the 20th to the 28th targeting employee 8812 is among those the index returned.
  4. The verifier checks both heads' signatures with the public half of the signing key and compares the heads with the copies the customer already holds.

Every hash matched and the heads agreed. The compliance team gets its answer with those heads attached, so its own auditor can repeat the check: Dmitri, acting as Marta under ticket SUP-7730, changed the account at 16:52:07 UTC, an hour after two denied attempts from the employee app, and no record in that stretch was edited or removed after the first signed head that covers it, taken within the hour. To design a trail that can give that answer, try the Tamper-Evident Audit Log Challenge.

Practice this