Supply chain: cosign signing, host verification, and pinning

A bootc distro is a pipeline that turns a Git push into a root filesystem on someone's laptop. Every hop in that pipeline — GitHub Actions runners, third-party actions, the base image, the registry, the pull on the client — is an injection point. This chapter covers how Margine signs what it publishes, how a host verifies what it pulls, and how the CI itself is hardened against tampered dependencies.

The split of responsibilities:

8.1 The cosign keypair

Margine uses key-based cosign signing: an ECDSA P-256 keypair generated once with cosign generate-key-pair. Private material never enters Git:

# margine-image/.gitignore
secrets/MOK.key
secrets/cosign.key
secrets/*.pem

The public half is committed (secrets/cosign.pub) and is the only thing a consumer needs:

# margine-image/secrets/cosign.pub
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEqOUib+6SVxWdP5wKCEBkJZEZTmza
rwTaC+nUx1VQmoRmEl9ZwNH4fL46VHhTHfpQukTinXKSkaDWafXupCRygw==
-----END PUBLIC KEY-----

The private key lives in two places: a GitHub Actions repository secret (COSIGN_PRIVATE_KEY) and an offline local backup. Same dual-custody model as the MOK key. The 2026-06-05 stack audit explicitly weighed this against keyless OIDC and chose to stay key-based for now (see §8.7 for the trade-off table).

8.2 Sign by digest, in a separate CI job

build.yml is split into build_push → sign → notify. The header comment in the workflow states the rationale:

# margine-image/.github/workflows/build.yml (header comment)
# build_push does the heavy work (buildah + rechunk + skopeo push,
# ~25 min). sign is a separate cheap job (~1 min) that signs the
# pushed manifest *by digest* instead of by tag — cosign warns
# against by-tag signing as it's racy.
#
# On a failed sign step, `gh run rerun --failed <run-id>` re-runs
# only the sign job (~1 min) instead of redoing the whole build.

Two design decisions here, both worth copying:

1. Capture the digest at push time. skopeo copy --digestfile records the manifest digest of exactly what was uploaded; the job exports it as an output for the sign job:

# margine-image/.github/workflows/build.yml — "Push rechunked image to GHCR"
for tag in ${{ steps.metadata.outputs.tags }}; do
  sudo skopeo copy --retry-times 3 \
    --dest-creds="${{ github.actor }}:${{ secrets.GITHUB_TOKEN }}" \
    --digestfile=/tmp/digest.txt \
    "${{ steps.rechunk.outputs.ref }}" \
    "docker://${IMG_FULL}:${tag}"
  if [[ -z "$DIGEST" ]]; then
    DIGEST="$(cat /tmp/digest.txt)"
  fi
done
...
echo "image_ref=${IMG_FULL}@${DIGEST}" >> "$GITHUB_OUTPUT"

All tags point at the same manifest, so one digest covers them all. Signing image@sha256:... instead of image:tag eliminates the TOCTOU window where someone retags between push and sign.

2. Sign in a minimal job with only the digest as input:

# margine-image/.github/workflows/build.yml — sign job
- name: Install Cosign
  uses: sigstore/cosign-installer@7e8b541eb2e61bf99390e1afd4be13a184e9ebc5 # v3.10.1

- name: Sign image by digest
  env:
    COSIGN_PRIVATE_KEY: ${{ secrets.COSIGN_PRIVATE_KEY }}
    COSIGN_PASSWORD: ""
  run: |
    set -euo pipefail
    IMAGE_REF="${{ needs.build_push.outputs.image_ref }}"
    ...
    cosign sign -y --key env://COSIGN_PRIVATE_KEY "${IMAGE_REF}"

env://COSIGN_PRIVATE_KEY keeps the key out of the filesystem and out of argv (visible in /proc). COSIGN_PASSWORD: "" declares the key is unencrypted — acceptable because the only at-rest copy is inside GitHub's secret store; encrypting it would just move the secret to a second variable. cosign sign pushes the signature as an OCI artifact to the same repository (the sha256-<digest>.sig tag), so no separate signature distribution channel is needed.

Signatures survive promotion because promotion preserves digests

Margine's :candidate → :stable promotion (after the QEMU smoke boot, chapter 7) is a skopeo copy --preserve-digests of the exact digest the gate booted — the manifest digest does not change, so the signature made against :candidate's digest is automatically valid for :stable:

# margine-image/.github/workflows/smoke-boot.yml — promote step
# ${PINNED} = ${BASE}@sha256:... resolved once (the digest just booted)
for promo_tag in stable "stable.${DATE_TAG}" "${DATE_TAG}"; do
  sudo skopeo copy --retry-times 3 --preserve-digests \
    "docker://${PINNED}" \
    "docker://${REGISTRY_IMAGE}:${promo_tag}"
done

Sign-by-digest plus copy-by-digest means the chapter-7 promotion gate adds zero signing work: the bytes that were smoke-booted are the bytes that were signed are the bytes users pull. (Promoting ${PINNED} rather than re-resolving :candidate also closed a void-gate bug — see chapter 9 §9.5.)

8.3 SBOM as a signed OCI referrer

The image's package inventory ships as an SPDX SBOM attached to the manifest (OCI 1.1 referrer) and signed with the same key:

# margine-image/.github/workflows/build.yml — "Attach + cosign-sign SBOM"
ATTACH_JSON="$(oras attach \
  --artifact-type application/vnd.spdx+json \
  ...
  --format json \
  "${IMAGE_REF}" \
  sbom.spdx.json:application/spdx+json)"
ATTACH_DIGEST="$(jq -r '.reference | split("@")[1] // .digest // empty' <<<"$ATTACH_JSON")"
...
SBOM_REF="${IMG_BASE}@${ATTACH_DIGEST}"
cosign sign -y --key env://COSIGN_PRIVATE_KEY "${SBOM_REF}"

Consumers do oras discover → oras pull → cosign verify against the same cosign.pub. The SBOM is generated in build_push (not in sign) for a reason that cost six PRs to learn:

Lesson: syft OOM on large rechunked images Symptom: the sign job was killed by a runner shutdown signal ~11-14 min into syft, across PRs #49, #52 (timeout bump), #53 (free 30 GB disk), #58 (--scope squashed), #60 (syntax fix) — see margine-image/docs/sbom-revisit-plan.md for the full table. Root cause: syft on a registry image reference always pulls every layer; --scope squashed changes the SBOM representation, not the input. A 14 GB rechunked image's expanded in-memory layer tree exceeds the 16 GB RAM of a stock ubuntu-24.04 runner. Freeing disk didn't help because the bottleneck was RAM. Fix: generate the SBOM inside build_push before rechunk, from a flat filesystem export instead of the layer model — peak RAM ~1 GB:

# margine-image/.github/workflows/build.yml — SBOM step
sudo podman container create --replace --name sbom-export \
  --entrypoint /bin/true \
  "localhost/${{ env.IMAGE_NAME }}:${{ steps.metadata.outputs.version }}"
sudo podman export sbom-export | sudo tar -C "$ROOTFS" -xf -
sudo "$(which syft)" --source-name "..." "$ROOTFS" -o spdx-json=sbom.spdx.json

The SBOM file is handed to the sign job as a 1-day workflow artifact. Pre-rechunk is fine: rechunk repacks layer boundaries, it does not change the package set.

Lesson: oras attach digest extraction Symptom: build #27065187939 (2026-06-06) failed with Signing SBOM: ghcr.io/.../margine@<no value>. Root cause: oras 2.x --format go-template='{{.Digest}}' resolves to <no value> — the JSON key is lowercase digest and the documented template path doesn't match. Fix (PR #73): use --format json and parse with jq -r '.reference | split("@")[1] // .digest // empty', then hard-fail if empty (see snippet above).

8.4 Host-side verification: policy.json + registries.d

Signing is worthless unless the client checks. Container verification on a Fedora/bootc host is configured by two files, both consulted by everything that pulls through containers/image (podman, skopeo, bootc, rpm-ostree's container backend):

/etc/containers/policy.json — maps registry scopes to requirements. To require Margine's cosign signature:

{
  "transports": {
    "docker": {
      "ghcr.io/daniel-g-carrasco/margine": [
        {
          "type": "sigstoreSigned",
          "keyPath": "/etc/pki/containers/margine.pub",
          "signedIdentity": { "type": "matchRepository" }
        }
      ]
    }
  }
}

sigstoreSigned means "a cosign-style signature verifiable with this key must exist for the pulled digest". matchRepository accepts any tag in the repo (necessary, because the signature is made against :candidate's digest but pulled via :stable).

/etc/containers/registries.d/margine.yaml — tells the stack where signatures live. Cosign stores them as OCI artifacts in the same repository, which must be opted into:

docker:
  ghcr.io/daniel-g-carrasco/margine:
    use-sigstore-attachments: true

Without this, verification looks for a lookaside (web-server) sigstore and fails with "no signature found" even though the signature exists on GHCR.

Because Margine derives from Bluefin DX, the base image already ships a populated /etc/containers/policy.json and /etc/pki/containers/ for the ghcr.io/ublue-os scope; the Margine-specific scope and key are the distro's job to add at image build time, so every installed host verifies its own updates. The 2026-06-05 audit flags this as the load-bearing check (§6.5):

Verify /etc/containers/policy.json allows your registry path with cosign verification, not just insecureAcceptAnything. This is what makes bootc switch --enforce-container-sigpolicy ghcr.io/daniel-g-carrasco/margine:stable actually verify, not just succeed. — margine-fedora-atomic/docs/audits/2026-06-05-margine-stack-audit.md

The same audit section cites the cautionary upstream incident: ublue-os/bluefin#4197 (2026-02-12), where bluefin-dx:stable shipped without /etc/pki/containers/ublue-os.pub, breaking bootc upgrade for every downstream consumer enforcing signature policy. Policy enforcement cuts both ways — if the key file is missing from the image, verified updates brick themselves. Margine's end-to-end check of this path on a booted install is tracked as deferred in the audit status delta (2026-06-05-margine-stack-audit-status-delta.md: "Verify /etc/pki/containers/<key>.pub + policy.json (§6.5) … ⏸ Deferred — needs a running install"), and margine-fedora-atomic/docs/roadmap.md keeps the honest TODO:

- ⏳ Move the `:stable` redirect to a *signed cosign verification* on
  the user side (today `bootc` trusts the registry; we could
  configure rpm-ostree's `verify-by-key` to enforce cosign at the
  client). Defense in depth.

The ostree-image-signed: transport

The documented rebase command selects the policy-enforcing transport explicitly:

# margine-image/README.md
rpm-ostree rebase ostree-image-signed:docker://ghcr.io/daniel-g-carrasco/margine:stable

rpm-ostree's container transports encode the trust decision in the ref itself:

Putting ostree-image-signed: in the user-facing docs means the deployment origin file records the signed transport, and every subsequent rpm-ostree upgrade/bootc upgrade on that origin re-verifies. bootc switch --enforce-container-sigpolicy is the bootc-native equivalent. Per the SBOM revisit plan: "Consumer verification flow (bootc switch --enforce-container-sigpolicy) works on cosign-by-digest alone" — the SBOM is hygiene, the image signature is the actual trust gate.

8.5 SHA-pinning actions and base images

Every third-party action in Margine's workflows is pinned to a full 40-character commit SHA, with the human-readable version kept as a comment so Renovate can bump both in lockstep (Margine retired dependabot.yml for a renovate.json5):

# margine-image/.github/workflows/build.yml
- name: Checkout
  # SHA-pinned for supply-chain safety. Comment is the human-readable
  # version the bump bot (Renovate) uses to bump both fields in lockstep.
  # See the tj-actions/changed-files incident (2025-03) for why @vN alone
  # is unsafe.
  uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3

A @v6 tag is a mutable pointer in someone else's repo; the tj-actions/changed-files compromise (March 2025) retroactively poisoned the floating tags of an action used by ~23k repos, exfiltrating CI secrets. A SHA cannot be moved. The same pattern covers docker/metadata-action, anchore/sbom-action, actions/upload-artifact/download-artifact, oras-project/setup-oras, ublue-os/remove-unwanted-software, osbuild/bootc-image-builder-action, and daniel-g-carrasco/titanoboa (a personal fork carrying the margine patch set — see the ISO chapter — SHA-pinned via the TITANOBOA_REF env in build-disk.yml) in the disk/ISO workflows.

Lesson: a floating action tag is also a floating tool version (CVE-2026-39395) Symptom: audit §6.2 flagged sigstore/cosign-installer@v3 (floating) in both build workflows as CRITICAL. Root cause: @v3 doesn't pin which cosign binary gets installed. CVE-2026-39395 / GHSA-w6c6-c85g-mmv6 (April 2026): cosign verify-blob-attestation returns false positives on malformed payloads; patched in cosign v3.0.6. A floating installer tag gives no guarantee of >= v3.0.6. Fix (margine-image #41):

uses: sigstore/cosign-installer@7e8b541eb2e61bf99390e1afd4be13a184e9ebc5 # v3.10.1

v3.10.1 of the installer pulls cosign v3.0.6. Pinning the action SHA pins the toolchain version transitively.

One deliberate exception, worth stating because pinning is a policy, not a reflex (a former second one — hhd-dev/rechunk, once tag-pinned at @v1.2.4 — is now closed: it is SHA-pinned hhd-dev/rechunk@5fbe1d3a639615d2548d83bc888360de6267b1a2 # v1.2.4 like every other action):

• FROM ghcr.io/ublue-os/bluefin-dx:stable in the Containerfile floats on purpose. Margine wants upstream drift: the weekly cron (schedule: '0 4 * * 0' in build.yml) rebuilds against whatever Bluefin DX currently is, and the QEMU smoke gate (chapter 7) catches breakage before :stable moves. Digest-pinning the base would trade silent drift for a Renovate-style bump treadmill; the gate makes the float survivable. If you have no boot gate, pin the FROM digest.

A third, subtler pinning move: the spec repo (margine-fedora-atomic) is fetched at build time, and to keep that fetch reproducible build.yml resolves the ref to a commit SHA at build start, passes it into the build as --build-arg MARGINE_REF=<sha>, and also stamps it as an OCI label:

# margine-image/.github/workflows/build.yml — specref step + build-arg + label
SHA="$(gh api repos/${{ github.repository_owner }}/margine-fedora-atomic/commits/${REF} --jq .sha)"
...
buildah build --build-arg "MARGINE_REF=${{ steps.specref.outputs.sha }}" ...
place.the-empty.margine.spec-ref=${{ steps.specref.outputs.sha }}

The Containerfile declares ARG MARGINE_REF=main, which the fetch scripts consume — so the image pulls scripts/branding/declarations from the exact pinned SHA, not from a moving main that could change between the SHA resolution and the fetch. (This closed an earlier TOCTOU caveat: the ref was recorded as a SHA in the label but the fetch still followed main.) Any published image can be traced back to — and is byte-reproducible from — the exact spec-repo commit that produced it.

Linting the pipeline itself, and automating the bumps

Pinning is only half a policy; the other half is keeping the pins fresh and the glue scripts honest. Two pieces close that loop:

• lint.yml in each repo runs actionlint (workflow schema + shellcheck over every run: block), a shebang-aware shellcheck pass (tracked *.sh plus the extensionless system_files payloads discovered by their #! line — the GUI probe and the seed scripts would otherwise be invisible to shellcheck), and ruff over the Python build helpers. This is why the inline heredocs got extracted into real files: a script shellcheck can't see is a script nobody is checking.
• Renovate replaced Dependabot: dependabot.yml was retired for a renovate.json5 that bumps the SHA pins (and their version comments) in lockstep, including the # Renovate disabled carve-out for the personal Titanoboa fork that must not be auto-bumped.

8.6 Secrets handling in GHA

Margine's CI holds three secrets: MOK_KEY, MOK_CERT (kernel signing) and COSIGN_PRIVATE_KEY. (The mokutil enrollment passphrase is not a secret — it's a hardcoded constant MOK_PASSWORD="margine-os" in custom-kernel/install.sh, public by design; §4.6.) Handling rules visible in the workflow:

# margine-image/.github/workflows/build.yml
- name: Stage MOK secrets for BuildKit
  env:
    MOK_KEY: ${{ secrets.MOK_KEY }}
    MOK_CERT: ${{ secrets.MOK_CERT }}
  run: |
    mkdir -p /tmp/margine-secrets
    chmod 700 /tmp/margine-secrets
    printf '%s' "$MOK_KEY"      > /tmp/margine-secrets/MOK.key
    ...
    chmod 600 /tmp/margine-secrets/*
...
- name: Wipe staged secrets
  if: always()
  run: rm -rf /tmp/margine-secrets

Secrets are passed via env: (never string-interpolated into run: script bodies, which would land them in the rendered script), staged with restrictive modes, and wiped in an if: always() step so a failed build doesn't leave key material on a runner that might persist for later steps. Inside the build they enter only as BuildKit secret mounts, which exist for the duration of one RUN and never become a layer:

# margine-image/Containerfile
RUN --mount=type=bind,from=ctx,source=/,target=/ctx \
    ...
    --mount=type=secret,id=mok-key,target=/tmp/certs/MOK.key \
    --mount=type=secret,id=mok-cert,target=/tmp/certs/MOK.pem \
    /ctx/custom-kernel/install.sh

A COPY MOK.key + rm would leave the key recoverable in the layer history; a secret mount cannot.

Token scoping follows least privilege per job: GITHUB_TOKEN permissions are declared explicitly (contents: read, packages: write, id-token: write) instead of inheriting the repo default, and the cosign job authenticates with an explicit cosign login ghcr.io because cosign sign reads ~/.docker/config.json, which a fresh job hasn't populated. The notify job receives only job results, never secrets beyond the ntfy URL — and degrades to a no-op if that secret is absent.

8.7 Alternatives & other distros

Signing schemes:

• Key-based cosign (Margine, Bazzite, most ublue community images, the ublue image-template historically): one keypair, cosign.pub committed in-repo and baked into /etc/pki/containers/. Pros/cons per Margine's audit: "Works in air-gapped CI; signature verifiable without sigstore trust root" vs "Key rotation is a maintenance task; private key in repo secrets."
• Keyless sigstore — Fulcio + Rekor (Universal Blue's direction for first-party images): id-token: write → cosign sign $IMAGE with no --key; a short-lived cert from Fulcio binds the signature to the GHA workflow's OIDC identity, logged in Rekor. No key to leak or rotate, provenance is the workflow identity itself; but verification needs the sigstore trust root and an identity-matching policy (--certificate-identity-regexp), and policy.json support uses fulcio/rekorPublicKey stanzas — more moving parts on every client. Margine's audit verdict: migration is "a future improvement, not a fix."
• GPG-signed ostree commits (stock Fedora Silverblue/Kinoite ostree remotes): the classic pre-OCI model — the compose server signs the ostree commit; clients verify via gpg-verify=true + keyring in the remote config (ostree-remote-image transport bridges this to containers). Solid, but ties you to ostree remotes rather than plain registries, and signs commits, not OCI manifests — useless for podman pull consumers.
• Notation / Notary v2 (CNCF, Azure ecosystem): signs OCI manifests with X.509 chains. Fine for cluster admission controllers; effectively unsupported in containers-policy.json, so wrong tool for a bootc host.
• Sealed bootable images (systemd-boot + UKI + composefs fs-verity): moves integrity from pull time to every boot — Margine tracks this as ADR 0007 (margine-fedora-atomic/docs/adr/0007-sealed-bootable-images-tracker.md, status Watching). Complementary, not alternative: cosign authenticates the download, fs-verity would authenticate the running tree.

Other distros' supply chains, for calibration:

• Bluefin / Aurora / Bazzite (Universal Blue): same shape as Margine (which copied it): cosign sign in GHA, key in /etc/pki/containers/ublue-os.pub, policy.json scoped to ghcr.io/ublue-os — and the #4197 incident shows the failure mode when the key file goes missing from the image.
• Fedora Silverblue (registry path): Fedora's official bootc images are signed with Fedora's infrastructure (sigstore keys shipped in fedora-repos); the legacy ostree remote path uses Fedora's GPG key.
• openSUSE MicroOS / Aeon: no OCI signing — trust is RPM GPG signatures + signed repo metadata, applied through transactional-update snapshots. Verification granularity is per-package, not per-image.
• Vanilla OS (ABRoot v2): OCI-image-based A/B transactions; trust rests primarily on registry TLS + their build pipeline, no end-user signature policy comparable to containers-policy enforcement.
• NixOS: no image to sign — closures are verified via Ed25519 signatures on binary-cache narinfo (cache.nixos.org-1:... trusted-public-keys), and full source reproducibility is the fallback. Strongest story on paper, completely different mechanism.
• ChimeraOS: frzr deploys squashfs images from GitHub releases; integrity is HTTPS + release checksums, no client-side signature policy.

CI pinning alternatives: Renovate/Dependabot with pinDigests (automates the SHA+comment dance Margine does manually), Chainguard's frizbee/StepSecurity to mass-pin existing workflows, or GitHub's allowed-actions policy as an org-level backstop. For the base image, digest-pinned FROM + automated bump PRs (common in Renovate-managed ublue forks) trades Margine's "float + boot gate" for explicit review of every upstream change.

8.8 What this buys, and what it doesn't

End state: a Margine host that pulled via ostree-image-signed: with the margine key in /etc/pki/containers/ will refuse an update whose manifest wasn't signed by the Margine key — a compromised GHCR token alone can push a tag but cannot mint a valid signature. What it does not cover: a compromised GHA runner during the build (it holds the cosign key via secrets), a malicious upstream bluefin-dx:stable (floated by design, gated only behaviorally by the smoke boot), and the deferred §6.5 end-to-end verification on a booted install. Supply-chain work is a ratchet; the audit documents each remaining click.