Secure Boot for a custom kernel: shim → MOK

Chapter 3 swapped Bluefin's stock kernel for CachyOS. That swap breaks exactly one link in the UEFI trust chain: the kernel image is no longer signed by Fedora. This chapter covers how Margine repairs that link with a Machine Owner Key (MOK) — signing at image build, certificate shipping, and the first-boot enrollment UX — and what the alternatives would have cost.

4.1 The trust chain and where a custom kernel breaks it

On a stock Fedora Atomic / Universal Blue system with Secure Boot enabled:

UEFI firmware db (Microsoft 3rd-party UEFI CA)
  └─ verifies → shim-x64.efi          (Fedora's shim, Microsoft-signed)
       └─ verifies → grubx64.efi      (Fedora-signed; shim embeds the Fedora CA)
            └─ verifies → vmlinuz     (Fedora-signed, via the shim_lock protocol:
                                       checked against firmware db + shim's MokList)
                 └─ verifies → *.ko   (kernel module signature enforcement against
                                       the kernel's builtin keys + the MOK keyring)

Everything above vmlinuz is inherited unchanged from the base image — Margine never touches shim or GRUB, so the Microsoft-signed entry point keeps working on every consumer machine without firmware changes. A COPR kernel (kernel-cachyos) carries no Fedora signature, so with Secure Boot on, GRUB refuses to load it ("bad shim signature") and every out-of-tree .ko is rejected by the module loader.

shim's escape hatch is MokList: a list of extra certificates stored in EFI variables, managed by the user through mokutil (stages a request from the running OS) and MokManager (the blue pre-boot UI that confirms it with physical presence). Anything signed by a MokList cert is as trusted as anything Fedora-signed. So the design is:

1. Generate one Margine keypair (RSA-2048).
2. At image build: sign vmlinuz with sbsign and every module with the kernel's sign-file.
3. Ship the public cert in the image at /usr/share/cert/MOK.der.
4. Get the cert into MokList on first boot with the least possible user pain.

Note the project did not start here. ADR 0003 (2026-05-22) explicitly deferred custom keys — "Fedora signed shim → Fedora signed GRUB → Fedora signed kernel […] Do not use Limine, sbctl, custom MOK keys" — until the stock chain plus LUKS2/TPM2 was proven in the lab (/home/daniel/dev/margine-fedora-atomic/docs/adr/0003-fedora-native-boot-security.md). The MOK path arrived only with ADR 0006's CachyOS decision. Prove the boring baseline first.

4.2 Key material: what is secret and what is not

The build refuses to proceed with mismatched material — a wrong-cert build would produce an image whose kernel can never be trusted, discovered only at a user's boot screen:

# margine-image/build_files/custom-kernel/install.sh:36-44
openssl pkey -in "$SIGNING_KEY"  -noout >/dev/null \
  || { err "MOK.key is not a valid private key"; exit 1; }
openssl x509 -in "$SIGNING_CERT" -noout >/dev/null \
  || { err "MOK.pem is not a valid X509 cert"; exit 1; }
_tmp1=$(mktemp); _tmp2=$(mktemp)
openssl pkey -in "$SIGNING_KEY"  -pubout        >"$_tmp1"
openssl x509 -in "$SIGNING_CERT" -pubkey -noout >"$_tmp2"
cmp -s "$_tmp1" "$_tmp2" \
  || { rm -f "$_tmp1" "$_tmp2"; err "MOK.key and MOK.pem don't match"; exit 1; }

Extracts the public key from both halves and byte-compares them. Fail-fast at minute 1 of a 28-minute build instead of fail-silent at the user's firmware.

Getting secrets into the build without leaking them

CI materializes the GitHub secrets as files, then hands them to buildah as BuildKit secrets:

# margine-image/.github/workflows/build.yml:125-136
- name: Stage MOK secrets for BuildKit
  env:
    MOK_KEY: ${{ secrets.MOK_KEY }}
    MOK_CERT: ${{ secrets.MOK_CERT }}
  run: |
    mkdir -p /tmp/margine-secrets
    chmod 700 /tmp/margine-secrets
    printf '%s' "$MOK_KEY"      > /tmp/margine-secrets/MOK.key
    printf '%s' "$MOK_CERT"     > /tmp/margine-secrets/MOK.pem

# margine-image/Containerfile:39-46
RUN --mount=type=bind,from=ctx,source=/,target=/ctx \
    --mount=type=cache,dst=/var/cache \
    --mount=type=cache,dst=/var/log \
    --mount=type=tmpfs,dst=/tmp \
    --mount=type=secret,id=mok-key,target=/tmp/certs/MOK.key \
    --mount=type=secret,id=mok-cert,target=/tmp/certs/MOK.pem \
    /ctx/custom-kernel/install.sh

type=secret mounts exist only during this RUN and never become an image layer; /tmp is a tmpfs mount on top of that. The private key cannot end up in the published OCI image even by accident — COPY secrets/ into a layer is the classic way projects leak signing keys.

4.3 Signing at image build

vmlinuz with sbsign

# margine-image/build_files/custom-kernel/install.sh:98-108
sign_kernel() {
  _vmlinuz="/usr/lib/modules/${KERNEL_VERSION}/vmlinuz"
  [[ -f "$_vmlinuz" ]] || { err "vmlinuz not found at $_vmlinuz"; return 1; }
  _tmp=$(mktemp)
  sbsign --key "$SIGNING_KEY" --cert "$SIGNING_CERT" --output "$_tmp" "$_vmlinuz"
  sbverify --cert "$SIGNING_CERT" "$_tmp" \
    || { rm -f "$_tmp"; err "sbverify failed on signed kernel"; return 1; }
  cp "$_tmp" "$_vmlinuz"
  chmod 0644 "$_vmlinuz"
  rm -f "$_tmp"
}

sbsign (from sbsigntools, installed transiently and removed at the end of the layer) embeds an Authenticode signature in the PE binary. sbverify re-checks before the original is overwritten, so a half-written signature can't ship. The path is the ostree-canonical /usr/lib/modules/<KVER>/vmlinuz — sign in place, before initramfs regeneration.

Every module with sign-file

The kernel verifies modules itself (CONFIG_MODULE_SIG), with a detached-appended signature format that sbsign does not produce — the kernel tree's own scripts/sign-file does. Fedora kernels ship modules compressed, and signatures must go on the uncompressed ELF:

# margine-image/build_files/custom-kernel/install.sh:110-129 (trimmed: .gz arm omitted)
sign_kernel_modules() {
  _module_root="/usr/lib/modules/${KERNEL_VERSION}"
  _sign_file="${_module_root}/build/scripts/sign-file"
  [[ -x "$_sign_file" ]] || { err "sign-file missing: $_sign_file"; return 1; }
  find "$_module_root" -type f \( \
      -name "*.ko" -o -name "*.ko.xz" -o -name "*.ko.zst" -o -name "*.ko.gz" \
    \) | while IFS= read -r _mod; do
    case "$_mod" in
      *.ko)
        "$_sign_file" sha256 "$SIGNING_KEY" "$SIGNING_CERT" "$_mod" ;;
      *.ko.xz)
        _raw="${_mod%.xz}"
        xz -d -q "$_mod"
        "$_sign_file" sha256 "$SIGNING_KEY" "$SIGNING_CERT" "$_raw"
        xz -z -q "$_raw" ;;
      *.ko.zst)
        _raw="${_mod%.zst}"
        zstd -d -q --rm "$_mod"
        "$_sign_file" sha256 "$SIGNING_KEY" "$SIGNING_CERT" "$_raw"
        zstd -q "$_raw" ;;

Decompress → sign → recompress, per compression format. sign-file comes from kernel-cachyos-devel-matched (also transient). One pass covers thousands of modules.

Ordering matters. sign_kernel / sign_kernel_modules run near the end of install.sh (lines 384-388), after the v4l2loopback akmod build has dropped its kmod- RPM into /usr/lib/modules/${KERNEL_VERSION}. Any module-producing step added after the signing pass ships an unsigned .ko that the kernel will reject under lockdown (§4.7) — a class of bug invisible in CI (the QEMU smoke gate boots without Secure Boot) and visible only on enrolled hardware. Keep signing last among module producers.

4.4 Shipping the cert + the first-boot fallback service

The same build layer converts the cert to DER (the format mokutil wants), drops it at a fixed in-image path, and writes the fallback enrollment unit:

# margine-image/build_files/custom-kernel/install.sh:139-162 (trimmed)
create_mok_enroll_unit() {
  _mok_cert="/usr/share/cert/MOK.der"
  _unit_file="/usr/lib/systemd/system/mok-enroll.service"
  mkdir -p "$(dirname "$_mok_cert")"
  openssl x509 -in "$SIGNING_CERT" -outform DER -out "$_mok_cert"
  ...
  cat > "$_unit_file" <<EOF
[Unit]
Description=Enroll Margine MOK on first boot
ConditionPathExists=${_mok_cert}
ConditionPathExists=!/var/.mok-enrolled

[Service]
Type=oneshot
ExecStart=/bin/sh -c '(echo "${MOK_PASSWORD}"; echo "${MOK_PASSWORD}") | mokutil --import "${_mok_cert}"'
ExecStartPost=/usr/bin/touch /var/.mok-enrolled
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF
  systemctl -f enable mok-enroll.service
}

Mechanics worth knowing:

• mokutil --import does not modify MokList directly. It writes a pending request (cert + password hash) into the MokNew EFI variable. On the next boot, shim sees the request and chains into MokManager — the blue/grey pre-boot screen — where a human selects Enroll MOK → Continue → Yes, types the passphrase, and reboots. Only then does the cert enter MokList. No amount of root access enrolls a key without that console step.
• The two echos feed mokutil's password + confirmation prompts non-interactively.
• The marker file lives in /var, which on ostree systems is machine-local state shared across deployments: the unit runs once per machine, not once per image update. ConditionPathExists=! makes the unit a no-op forever after, while leaving a trivially scriptable reset (§4.8).
• The unit is enabled at build time (systemctl -f enable works in a container build — it just creates the multi-user.target.wants/ symlink in /usr), so every fresh deployment has it armed with zero installer cooperation.

The user-visible rebase flow is therefore: rebase → reboot (service stages the request) → reboot again → MokManager → type margine-os → done. Two reboots; the kernel chain is verified on every boot thereafter.

4.5 The ISO path: stage the request before the first installed boot

The fallback unit has a structural flaw on fresh installs: it runs inside the OS whose kernel is not yet trusted. Margine's installer ISO closes the loop from Anaconda instead.

Lesson — ISO MOK enrollment timing (PR #88, 2026-06-08) Symptom: fresh ISO installs got no MOK Manager screen on the first reboot; the installed system had to boot once (possible only because the lab machine had Secure Boot relaxed) just so mok-enroll.service could stage the request, then reboot again. On a strict Secure Boot machine the first installed boot would simply fail — the enrollment service can never run on a kernel that can't boot. Chicken-and-egg. Root cause: enrollment was staged only from inside the installed OS; nothing ran in the installer environment, which boots a Fedora-signed Anaconda kernel and is already fully trusted under Secure Boot. Fix: an Anaconda %post --nochroot (running in the trusted installer environment, with the EFI variables of the target machine) submits the import request before the first installed boot — mirroring Bluefin/Bazzite's ISO flow. shim opens MokManager on the very first post-install reboot, before the Margine kernel is ever loaded.

# margine-image/live-env/src/anaconda/post-scripts/secureboot-enroll-key.ks:15-64 (trimmed)
%post --nochroot --log=/mnt/sysimage/var/log/anaconda-post-mok-enroll.log
if [[ ! -d /sys/firmware/efi ]]; then
  log "EFI mode not detected — skipping MOK import"; exit 0
fi
MOK_CERT=""
for candidate in \
  /mnt/sysimage/usr/share/cert/MOK.der \
  /mnt/sysimage/ostree/deploy/default/deploy/*.0/usr/share/cert/MOK.der
do
  [[ -f "$candidate" ]] && { MOK_CERT="$candidate"; break; }
done
...
if mokutil --test-key "$MOK_CERT" >/dev/null 2>&1; then
  log "Margine MOK is already enrolled — nothing to import"; exit 0
fi
mokutil --timeout -1 || log "WARN: failed to set MokTimeout; continuing"
if printf '%s\n%s\n' 'margine-os' 'margine-os' | mokutil --import "$MOK_CERT"; then
  log "MOK import request submitted — shim should launch MokManager on the next boot"
fi
%end

Details that earn their bytes:

• The cert is read from the target deployment (both the /mnt/sysimage flat view and the raw ostree deploy path are probed), so the request always matches the exact image being installed — no second copy of the cert to drift.
• mokutil --test-key makes reinstalls idempotent: already-enrolled machines get no prompt.
• mokutil --timeout -1 disables MokManager's 10-second auto-continue, so an unattended first reboot parks on the prompt instead of silently skipping enrollment.
• It deliberately does not create /var/.mok-enrolled. If the user mashes Enter past MokManager, the in-OS mok-enroll.service re-stages the request on the next successful boot. Belt and suspenders, each path covering the other's miss.
• Every exit path is a soft exit 0 — a BIOS-mode install or a missing mokutil degrades to the service fallback instead of failing the whole install.

This fragment is the one the Titanoboa live ISO ships (ADR 0008 ported it verbatim from the retired iso-gnome.toml), and it carries one wrinkle the BIB ISO didn't have: the live environment itself runs Margine's CachyOS kernel, which is untrusted before enrollment. The documented flow there is disable Secure Boot → boot live ISO → install (request staged) → re-enable Secure Boot → MokManager → enroll. That is the cost of shipping one kernel in both the live and installed environments instead of keeping a Fedora-signed live kernel.

4.6 Why the passphrase is public by design

margine-os is printed in the README, the docs site, and this handbook. That is correct, not sloppy, because the password is not a secret-keeping mechanism:

• The real gate is physical presence. MokManager runs pre-OS, on the console, before any network or remote-access stack exists. The password's only job is to bind the console confirmation to the request staged earlier from the OS — proving the person at the keyboard is acting on that request, not rubber-stamping noise.
• An attacker with root could stage their own mokutil --import with their own password anyway; knowing Margine's adds nothing. An attacker without root can't stage a request at all. The password protects against exactly one thing — a user confirming a request they did not initiate — and a documented distro-wide value preserves that property: users are told "if the screen asks for a passphrase and margine-os works, this is the Margine request."
• Precedent: Universal Blue ships the same pattern with their public ublue-os key passphrase for Bazzite/Bluefin akmod signing. Margine copied it deliberately.

Lesson — passphrase rotation (2026-06-06) Symptom: test installs stalled at MokManager — the original passphrase was a 24-character random base64 string, and MokManager is a pre-boot UI with no clipboard, no second screen docs, and a US-layout keymap. Root cause: treating a public-by-design value as if it were a secret; entropy bought nothing and cost typability. Fix: rotate MOK_PASSWORD to the short human-typable margine-os (same pattern as ublue-os) and print it in the install docs. Recorded in margine-fedora-atomic/docs/07-secure-boot-tpm2.md ("rotated 2026-06-06 […] so users can type it at the MOK Manager screen without copy-paste").

Avoid characters that move between keymaps (y/z, symbols) — MokManager will not honor the user's configured layout.

4.7 Kernel lockdown implications

When Secure Boot is enabled, Fedora kernels (CachyOS COPR builds included) activate lockdown=integrity. For a distro builder this changes what users can do post-install:

• Unsigned modules will not load. No runtime DKMS/akmods for users: the private key is in CI, not on their disk. Anything module-shaped must be built and signed at image build time — which is exactly why v4l2loopback is compiled in the kernel layer (before the signing pass) rather than documented as a user rpm-ostree install akmod-v4l2loopback. A user-layered akmod produces a module the kernel rejects with Key was rejected by service.
• Hibernation is blocked (unsigned/unverified resume image), kexec of unsigned kernels is blocked, and raw /dev/mem, MSR writes, and ACPI table overrides are restricted — relevant to undervolting/overclocking tools some performance-distro users expect.
• The flip side: disabling Secure Boot also disables lockdown. "Just turn off SB" (§4.9) is not only a trust-chain downgrade; it silently changes kernel behavior users may depend on.
• Interaction with chapter 5's TPM2 story: the hardware PCR policy is 0+7 — PCR 7 measures Secure Boot state, so toggling SB or enrolling new keys changes PCR 7 and TPM auto-unlock falls back to the LUKS passphrase (which Margine never removes). Enroll the MOK first, then bind TPM2.

4.8 Recovery and verification

If both enrollment paths were missed (or the user hit "Continue boot" at MokManager), the marker-file design makes retry a three-liner:

sudo rm /var/.mok-enrolled
sudo systemctl start mok-enroll.service
sudo systemctl reboot

Verification on an enrolled system:

mokutil --sb-state                          # → SecureBoot enabled
mokutil --list-enrolled | grep -i margine   # cert visible in MokList
mokutil --test-key /usr/share/cert/MOK.der  # → "is already enrolled"

The on-image validator margine-validate-atomic-layout (chapter 9) checks Secure Boot state as part of its layout assertions, and margine-collect-diagnostics captures mokutil output for bug reports.

4.9 Alternatives & other distros

• Keep the stock Fedora-signed kernel — Bluefin, Aurora, Silverblue/Kinoite stock, Fedora CoreOS. Zero enrollment UX, zero key custody; you give up the custom kernel entirely. This was Margine's own phase-1 position (ADR 0003) until ADR 0006 accepted the MOK cost.
• MOK for akmods only, stock kernel underneath — Universal Blue's ublue-os/akmods (Bazzite/Bluefin NVIDIA + extra kmods, public ublue-os passphrase). Smallest possible MOK surface: only out-of-tree modules need the key, the kernel link stays Fedora-signed. The precedent Margine extended to a whole kernel.
• Document "disable Secure Boot" — Bazzite's docs fallback for stubborn firmware, and effectively mandatory on ChimeraOS and most Arch-derived gaming distros. Zero friction, works everywhere; loses boot-chain verification, disables lockdown, and changes PCR 7 (breaks TPM-bound LUKS policies that include it).
• Enroll your own PK/KEK/db (full owner keys) — sbctl on Arch/CachyOS classic; NixOS via lanzaboote (signs generations with owner keys since upstream NixOS has no shim). No shim, no MokManager, cryptographic ownership of the whole chain; but firmware-fiddly, per-machine (a distro can't pre-enroll for you), and dropping the Microsoft CA can brick GPU option ROMs unless the MS certs are re-added to db.
• Distro-own CA inside shim — openSUSE MicroOS/Aeon/Tumbleweed: Microsoft signs their shim, shim embeds openSUSE's CA, openSUSE signs kernels and kmod packages; MOK is used automatically for things like the NVIDIA driver. Same architecture as Fedora — viable only if you are big enough to get a shim review (shim-review is a months-long process; out of reach for a one-person distro, which is exactly why Margine rides Fedora's shim).
• UKI + sealed images — systemd-boot + Unified Kernel Images + composefs/fs-verity (Fedora's tracked future, ADR 0007 "Watching"; openSUSE Aeon is moving this way with FDE). Measures and signs the whole kernel+initramfs+cmdline as one PE; strictly stronger than signing vmlinuz alone (Margine's initramfs is unsigned today), but the bootc/ostree tooling isn't there yet.
• Vanilla OS (ABRoot) — sticks to the distro-signed kernel within its A/B image scheme; custom-kernel users are on their own, same trade as stock-kernel atomic distros.

The decision table reduces to: who signs the kernel, and who has to click through firmware to trust it? MOK is the only option where a third-party builder signs once and every user trusts it with a single physical-presence confirmation — no shim review, no firmware key surgery, no Secure Boot off.

Recap: sign everything at build (sbsign for vmlinuz, sign-file for every .ko, keys as BuildKit secrets); ship MOK.der in /usr; stage enrollment from the installer before the first boot of the untrusted kernel, with a marker-gated oneshot service as the rebase/missed- prompt fallback; make the passphrase short, public, and documented. Chapter 5 builds on the enrolled state: LUKS2, TPM2 PCR policy, and why PCR 7 only makes sense after this chapter's work is done.