Would the controllers install it?
The question came up in a security review at a company that makes irrigation controllers, 60,000 of them on farms, updated over a cellular link: if someone replaced the firmware file in our download bucket, would the controllers install it? They would. Each controller downloaded the new image from the CDN over HTTPS and flashed whatever arrived. HTTPS made sure the controller reached the right server and that the bytes were not changed on the way; how the file got onto the server was outside its view.
Follow the file backwards and the bucket is only the last link. The image was built on a CI runner, from a commit in the firmware repository, linking an open-source TLS library and a few dozen other packages the team never wrote. Together they form a software supply chain: the sources, dependencies, build machines, registries and deploy steps that turn code into the thing running in production. An attacker needs only one weak link, and the device at the end cannot tell which one it was.
Sign the bytes, check them on the device
The first fix moves trust off the bucket. The company creates a signing key pair in a KMS, a key management service whose private keys are used through its API and are not exported. When a build is released, the release step computes the image's SHA-256 digest and asks the KMS to sign it. The public key goes into each controller's bootloader, in storage a firmware update cannot overwrite. Before flashing, the controller hashes the downloaded image, checks the signature against that public key, and refuses the image if either does not match. A replaced file in the bucket is now a download that fails verification. The check has to be on the digest of the exact bytes being flashed, not on a name like fw:4.2 that someone could point elsewhere between check and download; Content-addressed storage & deduplication explains why a digest names exactly one set of bytes.
A signature carries no date of its own, though: last year's image with a known flaw is still validly signed. So the signed manifest pairs a version with the image's digest, and the controller refuses anything below a minimum it stores and never lowers. That minimum rises only when an old build is known to be dangerous, so rolling back to the previous good build still works.
A valid signature on the wrong code
Signing answers one question: did someone who could use this key sign these bytes? Where the bytes came from is another one. If an attacker gets into the build runner and adds code between checkout and compile, the release step signs the result and every controller accepts it. That is roughly what happened to SolarWinds in 2020: code inserted during the build of its Orion software shipped in updates signed with the company's own certificate.
So the build itself has to become something you can check: a fresh runner per build, from a reviewed commit, with network access only to the company's own package mirror. How the pipeline builds once and promotes that one digest is in CI/CD pipelines: integrate, build once, promote; what matters here is the record that travels with the digest.
Next to the image, the build platform produces provenance: a signed statement that this digest was built from this repository at this commit, by this builder, with these parameters. The signature on provenance comes from the build platform, not from the build's own steps, and when the platform keeps that signing key out of the build's reach, a script running inside the build cannot write a convincing record of a different origin. Provenance records where a build came from and how it was declared to run; it does not prove that everything that ran inside it was benign, so code slipped in during the build can sit beside an accurate record. A fresh, isolated runner for each build means an intrusion on one runner does not carry into the next build through that runner; caches, mirrors and credentials shared between builds need the same care. The reviewed commit and the pinned dependencies below are the controls on what the build runs. A compromised build platform can sign any record it likes. Before it asks the KMS for a signature, the release step verifies the provenance's signature, checks that it names this image's digest, and signs only images whose provenance names the expected repository, branch and builder.
The fleet key never reaches the build runner. Images are signed before they are downloadable, and each controller verifies the signature before flashing.
SLSA (Supply-chain Levels for Software Artifacts) grades this in levels, from provenance that merely exists up to a hardened build platform whose builds cannot influence one another and whose provenance signing key the build steps cannot reach. Its provenance is written in the in-toto attestation format, a common framework for signed statements about an artifact.
The code nobody on the team wrote
Most of the image is dependencies, and they arrive by name. A popular package can ship a malicious release after its maintainer's account is taken over. And in dependency confusion, shown by Alex Birsan in 2021 against dozens of companies, someone publishes a public package under the name of a company's internal one with a higher version number, and a resolver that searches both registries can pick the public one.
Pinning stops a new release from slipping in unnoticed. A lockfile records the exact version of every direct and transitive dependency and, where the ecosystem supports it, a hash of each, so a rebuild fetches the same bytes or fails; for a C library vendored into the firmware, the equivalent is a pinned commit and a checksum. Internal names resolve only from the internal registry, which closes the confusion. A new version from outside waits in Quarantine until it has been checked. Running a package's install script isolated, with no credentials and no network it does not need, is covered in Sandboxing untrusted code: isolation, limits & egress.
Which builds carry the library?
Months later a vulnerability is published in the TLS library, affecting versions 3.0 to 3.0.7. The first question is where it runs. A software bill of materials, or SBOM, answers that: a machine-readable list of the components and versions inside an artifact, generated at build time and stored, often attested, with the image's digest. It lists only what its generator found: a C library copied into the source tree can be missing unless the build declares it. SPDX and CycloneDX are the two common formats. A query over the stored SBOMs gives the builds that contain any version from 3.0 to 3.0.7, the fleet inventory gives the controllers running those builds, and the fix reaches them through the usual staged rollout, which Progressive delivery: blue-green, canary & rollback describes.
Scanning works from the same inventory, and its answer moves: a scanner matches components against databases of known vulnerabilities, so a clean result means no match against what was known that day. A build that scanned clean when it shipped can match an entry published a month later, so stored SBOMs are rescanned as the databases update, not only when something is rebuilt. A scanner also cannot find what nobody has reported: code an attacker added on purpose has no database entry until someone spots it. A malicious dependency that was pinned, built cleanly and signed passes every check described here; the decision that counted was admitting it.
Guarding the key
The key is now the most valuable thing the company owns: whoever can use it can sign anything, and 60,000 controllers will flash it. Keep the private key in the KMS or an HSM, grant the sign operation to the release step alone, never to the build runner, and log every use. Plan rotation before you need it: a controller trusts only the public keys it already holds, so the bootloader carries a second public key whose private half stays offline. If the first key is lost or stolen, the offline key signs a numbered key update that replaces it; the bootloader accepts key updates only under that second key and never one older than the last it applied, so the old key cannot be put back. Secrets management & encryption covers storing and rotating keys. Container images can avoid the long-lived key: Sigstore's cosign signs with a short-lived certificate tied to the identity in the CI workflow's OpenID Connect token (OAuth 2.0 & OpenID Connect flows) and records the signature in a public transparency log, and an admission check in the cluster refuses an image whose digest has no valid signature from the expected workflow identity and OIDC issuer. Without a check like that, or the controller's, a signature is a file nobody reads.
The firmware Challenge
The OTA Firmware Updates Challenge asks you to "Sign every image and have the device verify the signature before flashing". Put the KMS on the Grid with an edge to firmware storage, so images are signed before they are ever downloadable, and note on the device what it checks before writing to its inactive partition: the signature over the digest of the image it downloaded. For the container side, the Container & Package Registry Challenge asks you to scan every pushed artifact and surface the result before deploy, and the Code Deployment Platform Challenge adds an audit trail of who deployed what and when.