Cache the expensive half of the build so iteration is cheap

The dnf transaction is essentially the whole cost of a build — emulated rpm
scriptlets for 502 packages on minimal, 1929 on workstation. Everything after
it is minutes. Getting this laptop to boot will take several attempts at the
kernel command line and the dracut driver list, and paying for a reinstall each
time is not tenable.

Stage the post-dnf tree under <work>/base, keyed on a hash of the package
lists, release and variant, and copy it per build with --reflink=auto (a CoW
clone on btrfs). Config and overlay edits now reuse it; package list edits
invalidate it on their own, so --fresh is only needed to force the issue.

Keep downloaded rpms in a cachedir outside the install root, so even --fresh
re-runs the scriptlets without re-downloading. keepcache=0 was exactly the
wrong setting for a build meant to be run repeatedly.

Add --work so CI can put both outside the job workspace, which is wiped between
runs, and default the build container to the gongfoo aarch64 build base so the
assembly tooling is not installed under emulation every time. That image is a
speedup, not a dependency: fall back to stock Fedora when it is unreachable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XWRjNJMistCy6ngXH5aJLS

.gitea/workflows/build-image.yaml

@@ -72,13 +72,34 @@ jobs:
           fi
           cat "$handler"
 
+      # Both of these live outside the job workspace, which is wiped between
+      # runs. On metal runners a host path persists for free and avoids
+      # shuttling multi-gigabyte caches through Gitea's cache store — the
+      # tradeoff being that they are per-runner, so a job landing on a runner
+      # that has not built before starts cold.
+      - name: Prepare persistent build state
+        run: |
+          echo "CACHE_DIR=/var/tmp/c630-build/dnf"   >> "$GITHUB_ENV"
+          echo "WORK_DIR=/var/tmp/c630-build/work"   >> "$GITHUB_ENV"
+          mkdir -p /var/tmp/c630-build/{dnf,work}
+          # Keep it bounded: drop cached rpms nothing has touched in a month.
+          find /var/tmp/c630-build/dnf -type f -atime +30 -delete 2>/dev/null || true
+          du -sh /var/tmp/c630-build/* 2>/dev/null || true
+
       - name: Build
         run: |
           case "${{ matrix.variant }}" in
             workstation) size=16384 ;;
             *)           size=8192  ;;
           esac
-          ./build/build-image.sh --variant "${{ matrix.variant }}" --size "$size"
+          # No --fresh: the stamp in stage2.sh hashes the package lists, so a
+          # change there invalidates the staged base on its own. Checkout is
+          # shallow here anyway, so diffing against HEAD~1 would not be reliable.
+          ./build/build-image.sh \
+            --variant "${{ matrix.variant }}" \
+            --size "$size" \
+            --cache "$CACHE_DIR" \
+            --work "$WORK_DIR"
 
       - name: Checksums
         run: cat output/*.sha256

.gitignore

@@ -1,5 +1,6 @@
-# Build output
+# Build output and caches
 /output/
+/.cache/
 *.img
 *.img.zst
 *.raw

README.md

@@ -65,6 +65,39 @@ filesystems from directory trees with `mke2fs -d` and `mcopy` rather than
 mounting loop devices, so it does not need `/dev/loop-control` — which CI
 runners generally will not hand out.
 
+### Iterating
+
+Every aarch64 binary runs under emulation, and the `dnf` transaction is
+essentially all of the cost — 500-odd packages' worth of rpm scriptlets for
+`minimal`, four times that for `workstation`. Everything after it takes
+minutes. Since getting this machine to boot will take a few attempts, the build
+is arranged so you only pay that once:
+
+- The post-`dnf` root filesystem is staged under `<work>/base`, keyed on a hash
+  of the package lists, release and variant. Editing `config/device.env`,
+  `overlay/`, or the bootloader config reuses it. Editing `config/packages/`
+  invalidates it automatically.
+- The working copy is made with `cp --reflink=auto`, so on btrfs or xfs it is
+  a copy-on-write clone rather than a real copy.
+- Downloaded rpms live in `.cache/dnf`, outside the staged tree, so even
+  `--fresh` re-runs the scriptlets without re-downloading.
+
+In practice a kernel-command-line change rebuilds in a few minutes.
+
+```sh
+./build/build-image.sh --variant minimal            # reuses the staged base
+./build/build-image.sh --variant minimal --fresh    # forces a reinstall
+./build/build-image.sh --variant minimal --keep-rootfs   # keep the tree to poke at
+```
+
+CI points `--work` and `--cache` at `/var/tmp/c630-build` so both survive
+between jobs. That is per-runner, so the first build on a given runner is cold.
+
+The build container defaults to `git.lair.cafe/gongfoo/build-fedora-44-aarch64`,
+which ships the assembly tooling so it does not have to be installed under
+emulation on every run. If that image is not reachable the build falls back to
+stock Fedora and installs the tooling itself — slower, but it works.
+
 ## Repository layout
 
 ```

build/build-image.sh

@@ -16,8 +16,11 @@ cd "$REPO_DIR"
 
 VARIANT=minimal
 OUTPUT_DIR="$REPO_DIR/output"
+CACHE_DIR="$REPO_DIR/.cache/dnf"
+WORK_DIR=""
 CONTAINER_IMAGE=""
 KEEP_ROOTFS=0
+FRESH=0
 
 usage() {
     cat <<EOF
@@ -26,9 +29,20 @@ Usage: $0 [options]
   --variant NAME     Package variant from config/packages/ (default: minimal)
   --output DIR       Where to write the image (default: ./output)
   --size MIB         Image size in MiB (default: from config/device.env)
-  --image REF        Build container image (default: registry.fedoraproject.org/fedora:\$FEDORA_RELEASE)
-  --keep-rootfs      Leave the staged rootfs behind for inspection
+  --image REF        Build container image (default: the gongfoo aarch64 build
+                     base, falling back to registry.fedoraproject.org/fedora)
+  --cache DIR        Persistent dnf package cache (default: ./.cache/dnf)
+  --work DIR         Where the staged base lives (default: <output>/.work).
+                     Point this at a path outside the checkout on CI so the
+                     staged base survives between jobs.
+  --fresh            Re-run the dnf transaction instead of reusing the staged
+                     base. Needed after changing config/packages/.
+  --keep-rootfs      Leave the working rootfs behind for inspection
   -h, --help         This message
+
+The staged root filesystem is cached under <work>/base and reused when the
+package set is unchanged, so edits to config/device.env or overlay/ rebuild in
+minutes rather than hours.
 EOF
 }
 
@@ -38,6 +52,9 @@ while [ $# -gt 0 ]; do
         --output)      OUTPUT_DIR="$2"; shift 2 ;;
         --size)        export IMAGE_SIZE_MIB="$2"; shift 2 ;;
         --image)       CONTAINER_IMAGE="$2"; shift 2 ;;
+        --cache)       CACHE_DIR="$2"; shift 2 ;;
+        --work)        WORK_DIR="$2"; shift 2 ;;
+        --fresh)       FRESH=1; shift ;;
         --keep-rootfs) KEEP_ROOTFS=1; shift ;;
         -h|--help)     usage; exit 0 ;;
         *) echo "unknown option: $1" >&2; usage >&2; exit 2 ;;
@@ -47,7 +64,20 @@ done
 # shellcheck source=../config/device.env
 source "$REPO_DIR/config/device.env"
 
-: "${CONTAINER_IMAGE:=registry.fedoraproject.org/fedora:${FEDORA_RELEASE}}"
+# The gongfoo build base carries the image-assembly tooling already, which
+# saves an emulated dnf transaction on every build. It is only a speedup, so
+# fall back to stock Fedora rather than failing when it is not reachable.
+BASE_IMAGE="git.lair.cafe/gongfoo/build-fedora-${FEDORA_RELEASE}-aarch64:latest"
+STOCK_IMAGE="registry.fedoraproject.org/fedora:${FEDORA_RELEASE}"
+if [ -z "$CONTAINER_IMAGE" ]; then
+    if podman image exists "$BASE_IMAGE" || podman pull -q "$BASE_IMAGE" >/dev/null 2>&1; then
+        CONTAINER_IMAGE="$BASE_IMAGE"
+    else
+        echo "note: ${BASE_IMAGE} unavailable, falling back to ${STOCK_IMAGE}"
+        echo "      (the build works either way, it just installs its tooling first)"
+        CONTAINER_IMAGE="$STOCK_IMAGE"
+    fi
+fi
 
 if [ ! -f "$REPO_DIR/config/packages/${VARIANT}.pkgs" ]; then
     echo "no such variant: ${VARIANT}" >&2
@@ -88,13 +118,19 @@ fi
 
 command -v podman >/dev/null || { echo "error: podman not found" >&2; exit 1; }
 
-mkdir -p "$OUTPUT_DIR"
+# The staged base is per-variant — two variants sharing one directory would
+# thrash the stamp and reinstall on every alternating build.
+: "${WORK_DIR:=${OUTPUT_DIR}/.work}"
+WORK_DIR="${WORK_DIR}/${VARIANT}"
+
+mkdir -p "$OUTPUT_DIR" "$CACHE_DIR" "$WORK_DIR"
 
 BUILD_REF="$(git -C "$REPO_DIR" rev-parse --short HEAD 2>/dev/null || echo unknown)"
 BUILD_DATE="$(date -u +%Y-%m-%d)"
 
 echo "==> variant=${VARIANT} release=${FEDORA_RELEASE} arch=${TARGET_ARCH} ref=${BUILD_REF}"
 echo "==> build container: ${CONTAINER_IMAGE}"
+echo "==> work=${WORK_DIR} cache=${CACHE_DIR}"
 
 # --privileged is what lets stage2 bind-mount /proc and /sys into the staged
 # rootfs so dracut can run in a chroot. Rootless podman grants only the caps
@@ -106,11 +142,14 @@ exec podman run --rm \
     --security-opt label=disable \
     -v "$REPO_DIR:/src:ro" \
     -v "$OUTPUT_DIR:/out" \
+    -v "$CACHE_DIR:/var/cache/c630-dnf" \
+    -v "$WORK_DIR:/work" \
     -e VARIANT="$VARIANT" \
     -e FEDORA_RELEASE="$FEDORA_RELEASE" \
     -e IMAGE_SIZE_MIB="${IMAGE_SIZE_MIB}" \
     -e BUILD_REF="$BUILD_REF" \
     -e BUILD_DATE="$BUILD_DATE" \
     -e KEEP_ROOTFS="$KEEP_ROOTFS" \
+    -e FRESH="$FRESH" \
     "$CONTAINER_IMAGE" \
     /bin/bash /src/build/stage2.sh

build/stage2.sh

@@ -12,8 +12,11 @@ set -euo pipefail
 
 SRC=/src
 OUT=/out
-WORK="$OUT/.work"
-ROOTFS="$WORK/rootfs"
+WORK=/work               # persists between builds; see --work in build-image.sh
+BASE="$WORK/base"        # pristine post-dnf tree, reused between builds
+ROOTFS="$WORK/rootfs"    # disposable working copy of the above
+STAMP="$WORK/base.stamp"
+DNF_CACHE=/var/cache/c630-dnf   # bind-mounted from the host, survives the run
 
 # shellcheck source=../config/device.env
 source "$SRC/config/device.env"
@@ -22,6 +25,7 @@ source "$SRC/config/device.env"
 : "${BUILD_REF:=unknown}"
 : "${BUILD_DATE:=unknown}"
 : "${KEEP_ROOTFS:=0}"
+: "${FRESH:=0}"
 
 IMAGE_NAME="fedora-${FEDORA_RELEASE}-${VARIANT}-${DEVICE_NAME}-${BUILD_DATE}-${BUILD_REF}"
 IMAGE_PATH="$OUT/${IMAGE_NAME}.img"
@@ -40,15 +44,23 @@ trap unbind_all EXIT
 
 bind() { mount --bind "$1" "$2" && MOUNTED+=("$2"); }
 
+mkdir -p "$WORK" "$DNF_CACHE"
+
 # ---------------------------------------------------------------------------
-log "Installing build tooling into the container"
+# Build tooling. Every dnf transaction in here runs emulated, so the prebuilt
+# base image (gongfoo's build-fedora-44-aarch64) carries these already and this
+# becomes a no-op. Only a stock Fedora image pays for it.
 # ---------------------------------------------------------------------------
-dnf -y install --setopt=install_weak_deps=False \
+if command -v mke2fs >/dev/null && command -v mcopy >/dev/null \
+   && command -v sgdisk >/dev/null && command -v zstd >/dev/null; then
+    echo "build tooling already present in the container image"
+else
+    log "Installing build tooling into the container"
+    dnf -y install --setopt=install_weak_deps=False \
+        --setopt=cachedir="$DNF_CACHE" --setopt=keepcache=1 \
         e2fsprogs dosfstools mtools gdisk util-linux rsync zstd findutils \
         >/dev/null
-
-rm -rf "$WORK"
-mkdir -p "$ROOTFS" "$WORK/esp"
+fi
 
 # ---------------------------------------------------------------------------
 log "Resolving package list (base + ${VARIANT})"
@@ -67,27 +79,60 @@ mapfile -t PACKAGES < <(
 echo "${#PACKAGES[@]} package specs"
 
 # ---------------------------------------------------------------------------
-log "Seeding repository configuration into the install root"
+# The dnf transaction is the only genuinely expensive step — an hour or more of
+# emulated rpm scriptlets. Everything after it is minutes. So stage it once into
+# a pristine tree keyed on the inputs that would change it, and copy that tree
+# per build. Iterating on the kernel command line or the overlay then costs a
+# copy instead of a reinstall.
 # ---------------------------------------------------------------------------
-# dnf reads its repo definitions from inside --installroot. Seed them from the
-# container (same release, same arch) so the first transaction has somewhere to
-# fetch from and something to verify signatures against. The fedora-repos
-# package overwrites these with its own copies during the transaction.
-mkdir -p "$ROOTFS/etc/yum.repos.d" "$ROOTFS/etc/pki/rpm-gpg" "$ROOTFS/etc/dnf"
-cp -a /etc/yum.repos.d/. "$ROOTFS/etc/yum.repos.d/"
-cp -a /etc/pki/rpm-gpg/. "$ROOTFS/etc/pki/rpm-gpg/"
-if [ -d /etc/dnf/vars ]; then cp -a /etc/dnf/vars "$ROOTFS/etc/dnf/"; fi
+WANT_STAMP="$(printf '%s\n' "$FEDORA_RELEASE" "$TARGET_ARCH" "$VARIANT" \
+                "${PACKAGES[@]}" | sha256sum | cut -d' ' -f1)"
 
-# ---------------------------------------------------------------------------
-log "Installing Fedora ${FEDORA_RELEASE} (${TARGET_ARCH}) — this is the slow part"
-# ---------------------------------------------------------------------------
-dnf -y \
-    --installroot="$ROOTFS" \
+if [ "$FRESH" = 1 ]; then
+    log "Discarding the staged base (--fresh)"
+    rm -rf "$BASE" "$STAMP"
+fi
+
+if [ -d "$BASE" ] && [ "$(cat "$STAMP" 2>/dev/null || true)" = "$WANT_STAMP" ]; then
+    log "Reusing the staged base — package set is unchanged"
+    echo "pass --fresh to force a reinstall"
+else
+    rm -rf "$BASE" "$STAMP"
+    mkdir -p "$BASE/etc/yum.repos.d" "$BASE/etc/pki/rpm-gpg" "$BASE/etc/dnf"
+
+    # dnf reads its repo definitions from inside --installroot. Seed them from
+    # the container (same release, same arch) so the first transaction has
+    # somewhere to fetch from and something to verify signatures against. The
+    # fedora-repos package overwrites these with its own during the transaction.
+    cp -a /etc/yum.repos.d/. "$BASE/etc/yum.repos.d/"
+    cp -a /etc/pki/rpm-gpg/. "$BASE/etc/pki/rpm-gpg/"
+    if [ -d /etc/dnf/vars ]; then cp -a /etc/dnf/vars "$BASE/etc/dnf/"; fi
+
+    log "Installing Fedora ${FEDORA_RELEASE} (${TARGET_ARCH}) — this is the slow part"
+    # keepcache=1 with a cachedir outside the install root: the downloaded rpms
+    # outlive both the transaction and the staged tree, so a --fresh rebuild
+    # re-runs the scriptlets but does not re-download 500-odd packages.
+    dnf -y \
+        --installroot="$BASE" \
         --releasever="$FEDORA_RELEASE" \
-    --setopt=keepcache=0 \
+        --setopt=cachedir="$DNF_CACHE" \
+        --setopt=keepcache=1 \
         --setopt=install_weak_deps=True \
         install "${PACKAGES[@]}"
 
+    printf '%s\n' "$WANT_STAMP" > "$STAMP"
+fi
+
+# ---------------------------------------------------------------------------
+log "Copying the staged base into a working tree"
+# ---------------------------------------------------------------------------
+# --reflink=auto is near-instant on btrfs (Fedora's default) and degrades to a
+# real copy elsewhere. The working tree gets mutated heavily below — accounts,
+# initramfs, bootloader — so the base has to stay untouched.
+rm -rf "$ROOTFS"
+cp -a --reflink=auto "$BASE" "$ROOTFS"
+mkdir -p "$WORK/esp"
+
 KVER="$(rpm --root "$ROOTFS" -q kernel-core --qf '%{VERSION}-%{RELEASE}.%{ARCH}\n' \
         | sort -V | tail -1)"
 [ -n "$KVER" ] || { echo "could not determine installed kernel version" >&2; exit 1; }
@@ -329,9 +374,16 @@ log "Compressing"
 zstd -12 -T0 --rm -f -o "${IMAGE_PATH}.zst" "$IMAGE_PATH"
 ( cd "$OUT" && sha256sum "${IMAGE_NAME}.img.zst" > "${IMAGE_NAME}.img.zst.sha256" )
 
+# The intermediate filesystem images are multi-gigabyte and worthless once
+# they are inside the disk image. The staged base is the opposite: expensive to
+# produce and the whole point of the cache, so it always stays.
+rm -f "$WORK/esp.img" "$WORK/boot.img" "$WORK/root.img"
+rm -rf "$WORK/esp"
 if [ "$KEEP_ROOTFS" != "1" ]; then
-    rm -rf "$WORK"
+    rm -rf "$ROOTFS" "$WORK/boot"
 fi
 
 log "Done"
 ls -lh "$OUT"
+printf '\nstaged base kept (%s) — the next build reuses it unless the package set changes\n' \
+    "$(du -sh "$BASE" 2>/dev/null | cut -f1)"

docs/runner-setup.md

@@ -36,17 +36,32 @@ before debugging a build failure.
 
 ## Disk space
 
-The build stages a full root filesystem and three filesystem images alongside
-the final disk image, under the job workspace. Budget roughly:
+The build keeps persistent state in `/var/tmp/c630-build` on each runner:
 
-- `minimal` — about 20 GiB
-- `workstation` — about 45 GiB
+| Path | Contents | Rough size |
+|---|---|---|
+| `/var/tmp/c630-build/work/<variant>/base` | Staged post-dnf root filesystem | 2 GiB minimal, 8 GiB workstation |
+| `/var/tmp/c630-build/dnf` | Downloaded rpms and repo metadata | 2 GiB minimal, 6 GiB workstation |
+
+Plus, transiently in the job workspace, the working rootfs copy, three
+filesystem images and the final disk image. Budget roughly 20 GiB for
+`minimal` and 45 GiB for `workstation` per runner.
+
+The workflow prunes cached rpms untouched for 30 days. The staged base is not
+pruned — it is invalidated by content hash, not age, so a stale one is
+harmless. Delete `/var/tmp/c630-build` to reclaim the space.
 
 ## Runtime
 
 Every aarch64 binary runs under qemu-user emulation, and rpm scriptlets are the
-worst case. Expect roughly 45–90 minutes for `minimal` and several hours for
-`workstation`. The workflow's `timeout-minutes` is set to 600 accordingly.
+worst case. A cold build is roughly 45–90 minutes for `minimal` and several
+hours for `workstation`; `timeout-minutes` is set to 600 accordingly.
+
+A warm build — one where the package set has not changed since that runner last
+built — skips the dnf transaction entirely and finishes in minutes. Because the
+state is per-runner and jobs are scheduled across all nine, expect the first
+build on each runner to be cold. Pinning this workflow to a single runner with
+a dedicated label would make warm builds the norm at the cost of parallelism.
 
 If this becomes tiresome, the fix is a native aarch64 runner. Register one with
 an `aarch64` label and change `runs-on: metal` to `runs-on: aarch64` in