Application payload: Flatpaks and the offline-docs module

The OCI image owns /usr. Apps live in /var/lib/flatpak — and ostree/bootc reset /var per deployment: anything you put there in the Containerfile is silently absent on the installed system. So "shipping apps" in a bootc distro is really a question of when and through which channel /var/lib/flatpak gets populated. Margine uses three tiers plus one supporting subsystem (system Flatpak overrides), and the same machinery powers the offline documentation mirror.

6.1 Three delivery tiers

The split criterion is stated in the build script itself:

#   BAKE (kickstart %post --nochroot at install time, ~22 apps):
#     Browser, mail, password, office, image+pdf+video viewer,
#     GNOME productivity suite. Apps the user expects to find ALREADY
#     INSTALLED on the desktop the first time they log in.
#
#   DEFER (.preinstall files + flatpak-preinstall.service at first
#   boot, ~12 apps):
#     Heavy creative apps (GIMP, Inkscape, darktable, OBS, Reaper,
#     ...) the user doesn't need in the first 10 min after first
#     login. flatpak-preinstall.service downloads them in background.

margine-image/build_files/20-flatpaks/install.sh (lines 14–28). The boundary is UX, not size alone: "first 10 minutes after login" defines BAKE.

6.2 One list, three consumers

The BAKE list is a single flat file, installer/flatpaks-base, read by (a) the OS-image build (copied to /usr/share/margine/installer-flatpaks-base for the kickstart), (b) the BIB installer-image build (installer/build.sh), and (c) the Titanoboa live-env build (live-env/src/flatpaks). One edit propagates everywhere.

app.zen_browser.zen
org.mozilla.thunderbird_esr
com.bitwarden.desktop
org.libreoffice.LibreOffice
...
# fm.reaper.Reaper — INTENTIONALLY EXCLUDED from BAKE 2026-06-05:
# Reaper's apply_extra script downloads the proprietary binary at
# install time, which fails inside the podman build container with
# "apply_extra script failed, exit status 256"
...
com.github.tchx84.Flatseal         # Flatpak permissions GUI
io.github.flattool.Warehouse       # Flatpak management GUI

margine-image/installer/flatpaks-base (trimmed). Note the two embedded decisions: apps whose apply_extra hook downloads proprietary blobs may not survive a container build (Reaper stays DEFER-only), and inline # comments are allowed — which caused a real failure, below.

Lesson — inline comments become Flatpak IDs. Symptom: flatpak install fails with Invalid id #: Name can't start with # (build #27075455521). Root cause: grep -v '^#' only strips whole-line comments; Flatseal # Flatpak permissions GUI passes #, Flatpak, permissions, GUI as literal app IDs. Fix:

APPS=$(grep -v '^[[:space:]]*#\|^[[:space:]]*$' "$LIST_PATH" \
       | sed -E 's/[[:space:]]*#.*$//; s/^[[:space:]]+//; s/[[:space:]]+$//' \
       | grep -v '^$')

margine-image/installer/build.sh (lines 52–54). Same sed replicated in live-env/src/build.sh.

6.3 BAKE: build-time install, install-time rsync

Why not flatpak install inside the kickstart?

Margine's first approach ran flatpak install --system directly in Anaconda %post. It failed silently on a fresh install.

Lesson — install-time downloads of a 5 GB set are fragile. Symptom: 2026-06-04 fresh install completed "successfully" but the apps were missing. Root cause: probably /tmp tmpfs OOM in the installer environment — the BAKE set is ~5 GB and the installer's /var is RAM-backed. Network blips produce the same silent partial result (--noninteractive returns 0 on partial failure). Fix: move the download to OCI build time (the CI runner has real disk). A dedicated installer image is built with the Flatpaks already in its /var/lib/flatpak, and the kickstart degrades to a pure local rsync. Documented in the kickstart itself:

# Previously this %post also did `flatpak install --system` at
# install time. That failed silently on the 2026-06-04 fresh install
# (probably /tmp tmpfs OOM in installer env, since the BAKE set is
# ~5 GB). Switching to installer-image moves that download to build
# time (CI runner has plenty of disk) so install-time stays fast +
# reliable.

Originally documented inline in the BIB kickstart (disk_config/iso-gnome.toml, now deleted); the same install-time-vs-build-time rationale carries over to live-env/src/anaconda/post-scripts/install-flatpaks.ks and ADR-0008 §4.

Installing Flatpaks inside a container build

flatpak install inside podman build needs two environment shims, because bwrap (used by apply_extra) expects a real /root and a writable /proc/sys:

# Copied straight from Bazzite's installer/build.sh — without these the
# apply_extra step (used by Reaper, Steam, openh264 for binary blobs)
# fails with:
#   F: Unable to provide a temporary home directory in the sandbox:
#      Unable to open path "/var/roothome": No such file or directory
#   bwrap: cannot open /proc/sys/user/max_user_namespaces:
#      Read-only file system
mkdir -p "$(realpath /root)"
mount -o remount,rw /proc/sys

flatpak remote-add --if-not-exists --system flathub \
  https://dl.flathub.org/repo/flathub.flatpakrepo
...
flatpak install --system --noninteractive --or-update flathub $APPS

margine-image/installer/build.sh (lines 21–31, 44–45, 66). The build also requires podman build --cap-add sys_admin --security-opt label=disable (set in build-disk.yml) — bwrap needs user namespaces.

The rsync into the target

ostree deployments mount their own /var; the kickstart locates the freshly written deployment and copies the populated tree in:

DEPLOY_DIR=$(ls -d /mnt/sysimage/ostree/deploy/default/deploy/*.0 2>/dev/null | head -1)
...
mkdir -p "$DEPLOY_DIR/var/lib"
rsync -aAXUHKP --filter='-x security.selinux' /var/lib/flatpak "$DEPLOY_DIR/var/lib/"
sync

margine-image/live-env/src/anaconda/post-scripts/install-flatpaks.ks (lines 26–40). This %post is deliberately not --erroronfail: the bake is quality-of-life, and every BAKE app is also in the DEFER list (next section), so a failed rsync degrades to a first-boot download — never a bricked install. bootc switch keeps --erroronfail because that is install-critical.

Lesson — copy POSIX xattrs, strip SELinux labels. Symptom: baked Flatpaks can fail to launch with AVC denials on the installed system. Root cause: rsync -X copies all xattrs, including security.selinux labels minted in the installer environment — wrong contexts for the target filesystem. Fix: Bluefin's verified-in-production incantation — keep -AX (ACLs + xattrs, needed by Flatpak's deploy metadata) but exclude the SELinux namespace; ostree's finalize relabels the target correctly:

rsync -aAXUHKP --filter='-x security.selinux' /var/lib/flatpak "$DEPLOY_DIR/var/lib/"

install-flatpaks.ks line 39; rationale in margine-fedora-atomic/docs/adr/0008-titanoboa-migration-plan.md §4. (The earlier BIB kickstart used plain -aAXUHK --open-noatime; the Titanoboa path adopted the filter as the invariant, and that BIB kickstart — iso-gnome.toml — has since been deleted.)

One more guard in the Titanoboa live environment: the live session and the installer share /var/lib/flatpak, so the baked set is bind-mounted read-only to keep the live user from tainting it before the rsync (var-lib-flatpak.mount, Options=bind,ro, live-env/src/build.sh lines 173–188 — Bazzite pattern).

6.4 DEFER: declarative first-boot via preinstall.d

Flatpak 1.16 introduced an upstream declarative preinstall API: .preinstall keyfiles under /usr/share/flatpak/preinstall.d/ are consumed by flatpak-preinstall.service at boot. This lives in /usr — image-owned, survives every update, no kickstart involved. Margine generates one file at image build:

mkdir -p /usr/share/flatpak/preinstall.d
{
  ...
  for app in \
      org.gimp.GIMP \
      org.inkscape.Inkscape \
      org.darktable.Darktable \
      com.obsproject.Studio \
      app.zen_browser.zen \
      ...
      it.mijorus.smile ; do
    echo "[Flatpak Preinstall $app]"
    echo "Branch=stable"
    echo "IsRuntime=false"
    echo
  done
} > /usr/share/flatpak/preinstall.d/margine-defaults.preinstall

margine-image/build_files/20-flatpaks/install.sh (lines 72–154, trimmed). Two design points:

1. Belt and suspenders: every BAKE app is also listed here. If the install-time rsync silently fails, flatpak-preinstall.service catches the gap at first boot (5–15 min wait instead of instant, but never "apps missing"). On a successful BAKE the entries are no-ops — flatpak skips already-installed refs.
2. The legacy uBlue mechanism is dead: /etc/ublue-os/system-flatpaks.list is silently ignored on current Bluefin DX. The build deletes it (rm -f, line 162) to prevent confusion. If you derive from a Universal Blue image, target preinstall.d, not the old list.

6.5 Notify-and-install-later: first-boot UX for DEFER

A background download with no feedback reads as "broken install". Margine ships an XDG autostart notifier that watches flatpak-preinstall.service and posts GNOME notifications at start and completion:

svc_state() {
  local s
  s=$(systemctl is-active flatpak-preinstall.service 2>/dev/null) || true
  [[ -z "$s" ]] && s=unknown
  printf '%s' "$s"
}
...
case "$initial" in
  active|activating|reloading)
    notify-send --app-name="Margine" --icon="org.gnome.Software" --urgency=low \
      --hint=string:desktop-entry:io.github.kolunmi.Bazaar \
      "Margine sta installando alcune app aggiuntive" "..."
    # Poll for completion. Cap at 60 min so we don't sit here forever
    deadline=$(( $(date +%s) + 3600 ))

margine-image/build_files/system_files/usr/libexec/margine-first-boot-status (lines 63–92, trimmed). Idempotent via a ~/.cache/margine/first-boot-notified marker; if the service already finished before first login, it stays silent. A failed final state posts a recovery hint (systemctl restart flatpak-preinstall.service).

Lesson — systemctl is-active on an activating unit prints and fails. Symptom: no notification on the 2026-06-06 fresh install; log shows unexpected initial state: activating / unknown. Root cause: is-active exits 3 for activating while still printing activating. The naive systemctl is-active ... || echo unknown therefore yields TWO lines (activating\nunknown), which matches no case arm. Fix: capture stdout, ignore the exit code, fall back to unknown only when stdout is empty — the svc_state helper above.

Lesson — GNOME 50+ skips autostart entries with X-GNOME-Autostart-Phase. Symptom: notifier never ran at login ("non ho visto nessun messaggio", 2026-06-04). Root cause: gnome-session no longer manages session phases; entries carrying the key are warned about and skipped entirely. Fix: delete the key from the .desktop file — see the warning comment in system_files/etc/xdg/autostart/margine-first-boot-status.desktop (lines 12–19).

6.6 On-demand: ujust margine-gaming

The heaviest payload (Steam, Lutris, Heroic, Bottles, RetroArch + host gamescope/vkBasalt) is not preinstalled at all. A dedicated image variant existed and was retired 2026-06-06; the supported path is two interactive recipes, each with a symmetric -remove. The default, ujust margine-gaming, installs the gaming launchers as Flatpaks system-wide and layers only the two gaming-only RPMs (gamescope + vkBasalt) via rpm-ostree. For maximum Proton/Wine compatibility (anti-cheat, VR, NVIDIA-proprietary + Mesa side-by-side), ujust margine-gaming-native instead layers Steam + Lutris + RetroArch as native RPMs — the full 32-bit dependency closure is baked into the base image so the layering resolves offline. The Flatpak recipe below is the default path:

flatpak install --system -y --or-update flathub \
    com.valvesoftware.Steam \
    net.lutris.Lutris \
    com.heroicgameslauncher.hgl \
    ...
# gamescope + vkBasalt are the only RPMs strictly gaming-only.

margine-image/build_files/60-custom.just (lines 150–162, trimmed). The recipe prints the trade-off before asking confirmation: layered RPMs branch the deployment from the base OCI image, add ~30–60 s per bootc upgrade, and can conflict when upstream relocates a file. Keeping this opt-in keeps the default image unbranched.

6.7 System Flatpak overrides

Sandboxed apps cannot see host paths the image wants them to read. Margine writes system-level overrides (/var/lib/flatpak/overrides/) from root services, and documents per-user overrides in recipes:

• Global, written by docs-refresh (next section): flatpak override --system --filesystem=/var/lib/margine/offline-docs:ro — grants every Flatpak read access to the docs mirror, so it keeps working if the user swaps Zen for another Flatpak browser.
• Per-user, suggested by ujust margine-gaming: flatpak override --user --filesystem=xdg-config/MangoHud:ro com.valvesoftware.Steam — lets Flatpak Steam read the host MangoHud config.

Rule of thumb: image-level guarantees → --system override written by a unit; user preference → --user override in a recipe.

6.8 The offline-docs module, end-to-end

A self-contained case study tying the above together: ship the project documentation offline, keep it fresh, and make it readable from a sandboxed browser.

6.8.1 Build: fetch + rewrite for file://

build-offline-docs.py crawls a fixed route list from the live docs site and rewrites each page for offline use: strip <script>, inline stylesheets, drop preload/prefetch/preconnect hints, and convert links — same-host /docs/* links become relative paths into the mirror, other root-relative URLs become absolute back to the live site:

ROUTES = [
    "/docs",
    "/docs/what-is-margine",
    ...
    "/docs/faq",
]

def inline_or_remove_link(match, base_url):
    tag = match.group(0)
    rel = (attr_value(tag, "rel") or "").lower()
    href = attr_value(tag, "href")
    if "stylesheet" in rel and href:
        css_url = urljoin(base_url, href)
        css = rewrite_css_urls(fetch_text(css_url), base_url)
        return f'<style data-margine-offline="stylesheet">\n{css}\n</style>'
    if "modulepreload" in rel or "preload" in rel or "prefetch" in rel or "preconnect" in rel:
        return ""

margine-image/build_files/50-branding/build-offline-docs.py (lines 17–34, 95–106, trimmed). It also writes a manifest.txt and a stamp file (epoch seconds) — the stamp is how runtime decides whether the image seed is newer than the runtime mirror.

6.8.2 Seed in /usr at image build

The branding stage installs the builder itself into the image and runs it, so build and runtime use the exact same fetch/rewrite logic:

install -Dm0755 /ctx/50-branding/build-offline-docs.py /usr/libexec/margine/build-offline-docs
python3 /usr/libexec/margine/build-offline-docs \
  --base-url "$MARGINE_DOCS_BASE_URL" \
  --output-dir "$OFFLINE_DOCS_DIR"   # /usr/share/margine/offline-docs
...
ln -sf ../margine-docs-refresh.timer \
   /usr/lib/systemd/system/timers.target.wants/margine-docs-refresh.timer
ln -sf ../margine-docs-refresh.service \
   /usr/lib/systemd/system/multi-user.target.wants/margine-docs-refresh.service

margine-image/build_files/50-branding/install.sh (lines 332–346, trimmed). Units are enabled with build-time wants-symlinks (no systemctl enable at runtime needed). CI validates the seed before push: 14+ index.html files, no live JS/CSS, no root-relative links (build.yml validator §A.4.bis).

6.8.3 Runtime: seed, grant, refresh

docs-refresh (run at boot by the service, periodically by the timer) does three ordered jobs — the first two work offline:

1. SEED: if /var/lib/margine/offline-docs is missing or its stamp is older than the /usr seed (e.g. right after a bootc upgrade shipped fresher docs), copy the seed into /var. Pure local cp — the mirror exists seconds after first boot, network or not.
2. FLATPAK ACCESS: flatpak override --system --filesystem="${DOCS_DIR}:ro". Without it, a Flatpak browser opening a file:// URI gets only that single file via the portal — the page renders, every relative link is dead.
3. REFRESH: gate on a 10 s curl ${BASE_URL}/healthz, then rebuild into ${DOCS_DIR}.new with the shipped builder and sync in. Offline → keep current copy, exit 0 (not a failure).

The service is a tightly sandboxed oneshot: ProtectSystem=strict with writes confined to StateDirectory=margine plus ReadWritePaths=/var/lib/flatpak/overrides (for step 2), empty CapabilityBoundingSet, ProtectHome, NoNewPrivileges (system_files/usr/lib/systemd/system/margine-docs-refresh.service). Deliberately no DynamicUser — it would relocate state to /var/lib/private (0700), unreadable by users' browsers. The timer is monotonic (OnBootSec=10min, OnUnitActiveSec=24h, RandomizedDelaySec=1h) because monotonic timers re-arm at every boot — a laptop that is never up 24 h still refreshes ~10 min after each boot; Persistent= would be a no-op since it only applies to OnCalendar=.

Lesson — never swap a directory a Flatpak sandbox has bind-mounted. Symptom: after a background refresh, already-running Flatpak browsers show "File not found" on every docs click until the app restarts. Root cause: Flatpak bind-mounts the override path at app start. The original refresh swapped the directory (mv away + rm -rf + rename new into place) — the sandbox keeps the bind mount on the now-emptied old inode. Fix: sync into the existing directory; rsync replaces each file atomically (tmpfile+rename) and --checksum leaves unchanged files untouched, so live readers never see a partial mirror:

sync_in() {
  chmod -R a+rX "$1"
  mkdir -p "$DOCS_DIR"
  rsync -a --delete --checksum "$1"/ "$DOCS_DIR"/
  rm -rf "$1"
}

system_files/usr/libexec/margine/docs-refresh (lines 38–51).

6.8.4 Open: offline-first launcher

docs-open (behind margine-documentation.desktop) opens the /var mirror instantly with no blocking network probe; fallbacks are live site (3 s healthz) then the /usr seed:

if [[ -f "${VAR_DIR}/docs/index.html" ]]; then
  exec xdg-open "file://${VAR_DIR}/docs/index.html"
fi
if curl -fsS --max-time 3 "https://margine.the-empty.place/healthz" >/dev/null 2>&1; then
  exec xdg-open "$ONLINE_URL"
fi
if [[ -f "${SEED_DIR}/docs/index.html" ]]; then
  exec xdg-open "file://${SEED_DIR}/docs/index.html"
fi

system_files/usr/libexec/margine/docs-open (lines 25–38). Why never the /usr seed first: Flatpak reserves /usr — no override can ever expose it to a sandbox, so the seed only works for non-Flatpak browsers. The whole reason the /var mirror exists is to give Flatpak browsers a path they are allowed to read.

Alternatives & other distros

• Bluefin / Aurora (Universal Blue): upstream Flatpak preinstall.d (Flatpak 1.16) for system apps, plus a Homebrew system-flatpaks.Brewfile first-boot path for some apps. Trade-off: zero installer complexity, but everything downloads at first boot — empty desktop for the first minutes. (Margine BAKEs DistroShelf directly instead of going through brew — see the comment in installer/flatpaks-base.)
• Bazzite: the installer-image bake Margine copied (install-flatpaks.ks rsync) plus ujust install-* recipes for optional apps. Trade-off: best first-login UX; costs a dedicated installer image build and 5–10 min of Anaconda %post.
• Fedora Silverblue/Workstation stock: no preinstall; GNOME Software shows Flathub (filtered) and Fedora Flatpaks. Trade-off: zero image complexity, user assembles everything.
• GNOME Software deploy lists / org.gnome.software.first-run + distro EULA-style curated lists: declare apps via GSettings/flatpak-repo files and let Software prompt. Trade-off: discoverable and consentful, but not unattended.
• Endless OS: Flatpaks fully baked into the disk image itself (their /var is part of the image build via eos-image-builder). Trade-off: enormous images; perfect offline story — their target market.
• openSUSE Aeon (MicroOS Desktop): tik installer + first-boot flatpak install of a curated set. Trade-off: simple, network-dependent first boot.
• Vanilla OS (ABRoot): apps via the apx/vso layer and Flatpak by default; system images stay app-free. Trade-off: clean separation, no offline-first option.
• NixOS: services.flatpak.enable plus the community nix-flatpak module for declarative app lists; or skip Flatpak and declare everything as Nix packages. Trade-off: fully declarative and reproducible; outside the Flathub runtime-dedup model.
• ChimeraOS / SteamOS: the primary app (Steam) is baked into the read-only image; Flatpak relegated to extras on the user partition. Trade-off: appliance-grade for one app, generic apps second-class.
• For offline docs specifically: most distros ship none (online wikis), Debian-likes ship -doc packages into /usr/share/doc (works, but unreadable from sandboxed browsers — exactly the problem Margine's /var mirror + system override solves), and GNOME ships Yelp with local Mallard help (no sandbox issue, but a separate authoring toolchain).

Takeaways

1. /var is per-deployment: app payload is a delivery pipeline (build-time bake → install-time rsync → first-boot preinstall → on-demand recipe), not a Containerfile line.
2. Make every tier a fallback for the previous one — BAKE apps duplicated in preinstall.d turn silent failures into a 15-minute delay instead of missing apps.
3. Downloads belong at build time (CI disk, retries, logs), not install time (tmpfs, silent partial failure).
4. Strip security.selinux when rsyncing /var/lib/flatpak across environments; let ostree finalize relabel.
5. Anything a Flatpak must read lives outside /usr, gets a --system override, and must be refreshed in place — sandboxes hold bind mounts on the directory inode they started with.