Learn

Architecture & pattern guides

Read the idea, build it on a Grid, prove it in a challenge. Short, practical guides — illustrated with RodGrid's own nodes — on system design, design patterns, flow modeling, and how to get the most out of the app.

All articles

Secrets management & encryption

Where keys and credentials live, how data is encrypted at rest and in transit without every service holding the master key, and how rotation stops being an event.

Yes, everything is encrypted

An online tax-preparation service keeps its clients' scanned payslips and ID documents in an object store and their records in PostgreSQL. A partner bank's security questionnaire asks whether customer data is encrypted, and the team writes yes: the site uses HTTPS, and the cloud provider encrypts the disks. Both are true, and the answer still hides two separate problems and a third question nobody asked.

Encryption in transit protects bytes on the wire: TLS between the browser and your edge, and (increasingly non-negotiable) between your own services. Encryption at rest protects bytes on disk: the database volume, the object store, the backup someone loses. Both matter, and doing one does not do the other. A TLS-only system leaks through a stolen backup; a disk-encrypted system leaks through a plaintext internal hop. Treating either layer as enough is the mistake.

The third question is the one that decides the design: who holds the key. AES itself is rarely your weakness. Where the key sits, which service can ask for it, how it is rotated and what happens when it leaks: that is the design. (End-to-end encryption answers it with the users' devices, as End-to-end encryption: keys on devices, a server that can't read explains; here the answer is your own services, because they must read the data.) The provider's disk encryption protects the storage media, a drive pulled from a rack or a discarded volume, not what the running database serves: the database process decrypts transparently and hands plaintext to any query that reaches it.

Encrypting the sensitive fields, or each client's rows, with keys of their own moves the question to who can use those keys. When the app does the encrypting and the reader cannot reach the keys, that protects against reads that bypass the app: a DBA, a raw dump, another service reading the table, a bad SELECT * that comes back as ciphertext. Against a compromised app it only limits the blast radius, and only when key policies restrict which keys that app can unwrap, per tenant, rate-limited and audited.

A key for every document

If every server that processes a document encrypts it locally with one shared master key, that key has to sit on all of them, and a leak from any one opens every document. Envelope encryption avoids that: generate a random data key per object (or per tenant, or per record), encrypt the payload with it locally (with an authenticated cipher such as AES-GCM, through a library that manages the nonce), then ask a KMS (key management service) to encrypt the data key with a master key that never leaves the KMS. Store the wrapped data key next to the ciphertext; to read, the KMS unwraps it and you decrypt locally.

The master key is used only for tiny operations, each one a logged KMS call, so with a distinct identity per service the log shows which service unwrapped data keys, and when; it does not show the reads a service then makes with a data key it holds in memory. Routine rotation of the master key means at most re-wrapping small keys (many KMSs keep old key versions for unwrapping), not re-encrypting terabytes; a leaked data key is another matter, and the data it protects must be re-encrypted under a new one. All of that holds only while few callers may unwrap: if every service can call Decrypt with the one master key, the KMS records every unwrap but limits none, so scope key policies per service and per purpose, and alert on unusual callers.

Envelope encryption: data keys locally, master key only inside the KMS

The blob store never sees a plaintext key; the KMS never sees the data.

The API key that was deleted two years ago

A search of the repository turns up the API key for the government's e-filing gateway, in a .env file committed two years ago and removed a week later. Removing it changed the current files, not the history, so every clone and fork made since holds it, and the key itself was never changed. Database passwords, third-party API keys, signing keys for sessions and webhooks: none of these belong in a committed file, .env included, in a config repo however private, or in a CI variable that any pipeline step can read and print. Nor in the build: a secret baked into an image or passed as an ordinary build argument can end up in a layer or the image's history, which are cached, pushed and pulled by every runner. Deliver secrets at runtime, or through the build tool's own secret mechanism when the build itself needs one. (For session-signing keys, Sessions vs tokens: JWT, refresh & revocation explains what rotating one does to tokens already issued.)

A secrets manager (Vault, cloud-native equivalents) stores them encrypted, hands them out over an authenticated call and logs who asked. Services get identity, not passwords: a workload identity (an IAM role, a mesh certificate, a Kubernetes service account) proves who the caller is, and the manager returns the secret, ideally a short-lived, dynamically issued credential (a 24-hour database user, an hour-long cloud token) that expires on its own. For people rather than workloads, see Authentication: passwords, MFA & identity providers.

Secrets the manager cannot issue dynamically, such as the e-filing key, still need scheduled rotation. Rotation then stops being a quarterly ceremony: the manager rotates, services re-fetch on a timer that runs ahead of expiry, and again when authentication fails, and a leaked dynamic credential stops working when its TTL runs out (check whether expiry also ends sessions already open with it); revoke it explicitly when you need it dead sooner. A credential whose change needs a config edit plus a rollout gets rotated when someone finds the time; fetched at start-up and refreshed on rotation, it can change daily.

TipGrep the repo, history included, for anything a leak would hand an attacker; if it finds something useful, that secret is in the wrong place, and it must be rotated, not just deleted. Same test for an image: scan the contents of every layer, not just the final filesystem or the build commands. No layer should hold key material.

Where TLS stops

Decide where TLS terminates and mark it on the diagram: at the load balancer, certificates live in one place and traffic can be inspected (it is where a WAF reads requests, in Edge protection: WAF, DDoS & defence in depth), but everything behind it is plaintext unless you re-encrypt to the origin. A service mesh with mutual TLS (both sides present certificates) adds in-transit encryption plus workload identity between services, and is a common answer once you have more than a handful of them. Certificates expire and their private keys are secrets, and public TLS lifetimes are shrinking toward weeks (200 days since March 2026, 47 by 2029), so manual renewal stops being an option: automate issuance and renewal (ACME, the mesh's CA) or the outage arrives on the day a certificate quietly lapses.

Every key on one diagram

The tax service's diagram is most of an answer to Secrets & Key Management. Draw it there, marking where every signing key and credential lives and where TLS terminates. No master key should appear on the diagram outside the KMS. Each credential and signing key should reach only the service that uses it, from the secrets manager, the KMS or the certificate manager, and never be written into code, an image or a log. Application data is encrypted with data keys the KMS wraps (envelope encryption), stored next to the ciphertext they protect. In Payment Processing System a leaked key costs real money, so the short-lived-credential path earns its keep.

Practice this