diff --git a/.gitea/workflows/build-image.yaml b/.gitea/workflows/build-image.yaml index 283ad2e..f718323 100644 --- a/.gitea/workflows/build-image.yaml +++ b/.gitea/workflows/build-image.yaml @@ -72,13 +72,34 @@ jobs: fi cat "$handler" + # Both of these live outside the job workspace, which is wiped between + # runs. On metal runners a host path persists for free and avoids + # shuttling multi-gigabyte caches through Gitea's cache store — the + # tradeoff being that they are per-runner, so a job landing on a runner + # that has not built before starts cold. + - name: Prepare persistent build state + run: | + echo "CACHE_DIR=/var/tmp/c630-build/dnf" >> "$GITHUB_ENV" + echo "WORK_DIR=/var/tmp/c630-build/work" >> "$GITHUB_ENV" + mkdir -p /var/tmp/c630-build/{dnf,work} + # Keep it bounded: drop cached rpms nothing has touched in a month. + find /var/tmp/c630-build/dnf -type f -atime +30 -delete 2>/dev/null || true + du -sh /var/tmp/c630-build/* 2>/dev/null || true + - name: Build run: | case "${{ matrix.variant }}" in workstation) size=16384 ;; *) size=8192 ;; esac - ./build/build-image.sh --variant "${{ matrix.variant }}" --size "$size" + # No --fresh: the stamp in stage2.sh hashes the package lists, so a + # change there invalidates the staged base on its own. Checkout is + # shallow here anyway, so diffing against HEAD~1 would not be reliable. + ./build/build-image.sh \ + --variant "${{ matrix.variant }}" \ + --size "$size" \ + --cache "$CACHE_DIR" \ + --work "$WORK_DIR" - name: Checksums run: cat output/*.sha256 diff --git a/.gitignore b/.gitignore index bf5ad4c..d11c5f1 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ -# Build output +# Build output and caches /output/ +/.cache/ *.img *.img.zst *.raw diff --git a/README.md b/README.md index 352311d..1805498 100644 --- a/README.md +++ b/README.md @@ -65,6 +65,39 @@ filesystems from directory trees with `mke2fs -d` and `mcopy` rather than mounting loop devices, so it does not need `/dev/loop-control` — which CI runners generally will not hand out. +### Iterating + +Every aarch64 binary runs under emulation, and the `dnf` transaction is +essentially all of the cost — 500-odd packages' worth of rpm scriptlets for +`minimal`, four times that for `workstation`. Everything after it takes +minutes. Since getting this machine to boot will take a few attempts, the build +is arranged so you only pay that once: + +- The post-`dnf` root filesystem is staged under `/base`, keyed on a hash + of the package lists, release and variant. Editing `config/device.env`, + `overlay/`, or the bootloader config reuses it. Editing `config/packages/` + invalidates it automatically. +- The working copy is made with `cp --reflink=auto`, so on btrfs or xfs it is + a copy-on-write clone rather than a real copy. +- Downloaded rpms live in `.cache/dnf`, outside the staged tree, so even + `--fresh` re-runs the scriptlets without re-downloading. + +In practice a kernel-command-line change rebuilds in a few minutes. + +```sh +./build/build-image.sh --variant minimal # reuses the staged base +./build/build-image.sh --variant minimal --fresh # forces a reinstall +./build/build-image.sh --variant minimal --keep-rootfs # keep the tree to poke at +``` + +CI points `--work` and `--cache` at `/var/tmp/c630-build` so both survive +between jobs. That is per-runner, so the first build on a given runner is cold. + +The build container defaults to `git.lair.cafe/gongfoo/build-fedora-44-aarch64`, +which ships the assembly tooling so it does not have to be installed under +emulation on every run. If that image is not reachable the build falls back to +stock Fedora and installs the tooling itself — slower, but it works. + ## Repository layout ``` diff --git a/build/build-image.sh b/build/build-image.sh index edc9942..5bac94c 100755 --- a/build/build-image.sh +++ b/build/build-image.sh @@ -16,8 +16,11 @@ cd "$REPO_DIR" VARIANT=minimal OUTPUT_DIR="$REPO_DIR/output" +CACHE_DIR="$REPO_DIR/.cache/dnf" +WORK_DIR="" CONTAINER_IMAGE="" KEEP_ROOTFS=0 +FRESH=0 usage() { cat </.work). + Point this at a path outside the checkout on CI so the + staged base survives between jobs. + --fresh Re-run the dnf transaction instead of reusing the staged + base. Needed after changing config/packages/. + --keep-rootfs Leave the working rootfs behind for inspection -h, --help This message + +The staged root filesystem is cached under /base and reused when the +package set is unchanged, so edits to config/device.env or overlay/ rebuild in +minutes rather than hours. EOF } @@ -38,6 +52,9 @@ while [ $# -gt 0 ]; do --output) OUTPUT_DIR="$2"; shift 2 ;; --size) export IMAGE_SIZE_MIB="$2"; shift 2 ;; --image) CONTAINER_IMAGE="$2"; shift 2 ;; + --cache) CACHE_DIR="$2"; shift 2 ;; + --work) WORK_DIR="$2"; shift 2 ;; + --fresh) FRESH=1; shift ;; --keep-rootfs) KEEP_ROOTFS=1; shift ;; -h|--help) usage; exit 0 ;; *) echo "unknown option: $1" >&2; usage >&2; exit 2 ;; @@ -47,7 +64,20 @@ done # shellcheck source=../config/device.env source "$REPO_DIR/config/device.env" -: "${CONTAINER_IMAGE:=registry.fedoraproject.org/fedora:${FEDORA_RELEASE}}" +# The gongfoo build base carries the image-assembly tooling already, which +# saves an emulated dnf transaction on every build. It is only a speedup, so +# fall back to stock Fedora rather than failing when it is not reachable. +BASE_IMAGE="git.lair.cafe/gongfoo/build-fedora-${FEDORA_RELEASE}-aarch64:latest" +STOCK_IMAGE="registry.fedoraproject.org/fedora:${FEDORA_RELEASE}" +if [ -z "$CONTAINER_IMAGE" ]; then + if podman image exists "$BASE_IMAGE" || podman pull -q "$BASE_IMAGE" >/dev/null 2>&1; then + CONTAINER_IMAGE="$BASE_IMAGE" + else + echo "note: ${BASE_IMAGE} unavailable, falling back to ${STOCK_IMAGE}" + echo " (the build works either way, it just installs its tooling first)" + CONTAINER_IMAGE="$STOCK_IMAGE" + fi +fi if [ ! -f "$REPO_DIR/config/packages/${VARIANT}.pkgs" ]; then echo "no such variant: ${VARIANT}" >&2 @@ -88,13 +118,19 @@ fi command -v podman >/dev/null || { echo "error: podman not found" >&2; exit 1; } -mkdir -p "$OUTPUT_DIR" +# The staged base is per-variant — two variants sharing one directory would +# thrash the stamp and reinstall on every alternating build. +: "${WORK_DIR:=${OUTPUT_DIR}/.work}" +WORK_DIR="${WORK_DIR}/${VARIANT}" + +mkdir -p "$OUTPUT_DIR" "$CACHE_DIR" "$WORK_DIR" BUILD_REF="$(git -C "$REPO_DIR" rev-parse --short HEAD 2>/dev/null || echo unknown)" BUILD_DATE="$(date -u +%Y-%m-%d)" echo "==> variant=${VARIANT} release=${FEDORA_RELEASE} arch=${TARGET_ARCH} ref=${BUILD_REF}" echo "==> build container: ${CONTAINER_IMAGE}" +echo "==> work=${WORK_DIR} cache=${CACHE_DIR}" # --privileged is what lets stage2 bind-mount /proc and /sys into the staged # rootfs so dracut can run in a chroot. Rootless podman grants only the caps @@ -106,11 +142,14 @@ exec podman run --rm \ --security-opt label=disable \ -v "$REPO_DIR:/src:ro" \ -v "$OUTPUT_DIR:/out" \ + -v "$CACHE_DIR:/var/cache/c630-dnf" \ + -v "$WORK_DIR:/work" \ -e VARIANT="$VARIANT" \ -e FEDORA_RELEASE="$FEDORA_RELEASE" \ -e IMAGE_SIZE_MIB="${IMAGE_SIZE_MIB}" \ -e BUILD_REF="$BUILD_REF" \ -e BUILD_DATE="$BUILD_DATE" \ -e KEEP_ROOTFS="$KEEP_ROOTFS" \ + -e FRESH="$FRESH" \ "$CONTAINER_IMAGE" \ /bin/bash /src/build/stage2.sh diff --git a/build/stage2.sh b/build/stage2.sh index 1d8ed59..38b9a10 100755 --- a/build/stage2.sh +++ b/build/stage2.sh @@ -12,8 +12,11 @@ set -euo pipefail SRC=/src OUT=/out -WORK="$OUT/.work" -ROOTFS="$WORK/rootfs" +WORK=/work # persists between builds; see --work in build-image.sh +BASE="$WORK/base" # pristine post-dnf tree, reused between builds +ROOTFS="$WORK/rootfs" # disposable working copy of the above +STAMP="$WORK/base.stamp" +DNF_CACHE=/var/cache/c630-dnf # bind-mounted from the host, survives the run # shellcheck source=../config/device.env source "$SRC/config/device.env" @@ -22,6 +25,7 @@ source "$SRC/config/device.env" : "${BUILD_REF:=unknown}" : "${BUILD_DATE:=unknown}" : "${KEEP_ROOTFS:=0}" +: "${FRESH:=0}" IMAGE_NAME="fedora-${FEDORA_RELEASE}-${VARIANT}-${DEVICE_NAME}-${BUILD_DATE}-${BUILD_REF}" IMAGE_PATH="$OUT/${IMAGE_NAME}.img" @@ -40,15 +44,23 @@ trap unbind_all EXIT bind() { mount --bind "$1" "$2" && MOUNTED+=("$2"); } -# --------------------------------------------------------------------------- -log "Installing build tooling into the container" -# --------------------------------------------------------------------------- -dnf -y install --setopt=install_weak_deps=False \ - e2fsprogs dosfstools mtools gdisk util-linux rsync zstd findutils \ - >/dev/null +mkdir -p "$WORK" "$DNF_CACHE" -rm -rf "$WORK" -mkdir -p "$ROOTFS" "$WORK/esp" +# --------------------------------------------------------------------------- +# Build tooling. Every dnf transaction in here runs emulated, so the prebuilt +# base image (gongfoo's build-fedora-44-aarch64) carries these already and this +# becomes a no-op. Only a stock Fedora image pays for it. +# --------------------------------------------------------------------------- +if command -v mke2fs >/dev/null && command -v mcopy >/dev/null \ + && command -v sgdisk >/dev/null && command -v zstd >/dev/null; then + echo "build tooling already present in the container image" +else + log "Installing build tooling into the container" + dnf -y install --setopt=install_weak_deps=False \ + --setopt=cachedir="$DNF_CACHE" --setopt=keepcache=1 \ + e2fsprogs dosfstools mtools gdisk util-linux rsync zstd findutils \ + >/dev/null +fi # --------------------------------------------------------------------------- log "Resolving package list (base + ${VARIANT})" @@ -67,26 +79,59 @@ mapfile -t PACKAGES < <( echo "${#PACKAGES[@]} package specs" # --------------------------------------------------------------------------- -log "Seeding repository configuration into the install root" +# The dnf transaction is the only genuinely expensive step — an hour or more of +# emulated rpm scriptlets. Everything after it is minutes. So stage it once into +# a pristine tree keyed on the inputs that would change it, and copy that tree +# per build. Iterating on the kernel command line or the overlay then costs a +# copy instead of a reinstall. # --------------------------------------------------------------------------- -# dnf reads its repo definitions from inside --installroot. Seed them from the -# container (same release, same arch) so the first transaction has somewhere to -# fetch from and something to verify signatures against. The fedora-repos -# package overwrites these with its own copies during the transaction. -mkdir -p "$ROOTFS/etc/yum.repos.d" "$ROOTFS/etc/pki/rpm-gpg" "$ROOTFS/etc/dnf" -cp -a /etc/yum.repos.d/. "$ROOTFS/etc/yum.repos.d/" -cp -a /etc/pki/rpm-gpg/. "$ROOTFS/etc/pki/rpm-gpg/" -if [ -d /etc/dnf/vars ]; then cp -a /etc/dnf/vars "$ROOTFS/etc/dnf/"; fi +WANT_STAMP="$(printf '%s\n' "$FEDORA_RELEASE" "$TARGET_ARCH" "$VARIANT" \ + "${PACKAGES[@]}" | sha256sum | cut -d' ' -f1)" + +if [ "$FRESH" = 1 ]; then + log "Discarding the staged base (--fresh)" + rm -rf "$BASE" "$STAMP" +fi + +if [ -d "$BASE" ] && [ "$(cat "$STAMP" 2>/dev/null || true)" = "$WANT_STAMP" ]; then + log "Reusing the staged base — package set is unchanged" + echo "pass --fresh to force a reinstall" +else + rm -rf "$BASE" "$STAMP" + mkdir -p "$BASE/etc/yum.repos.d" "$BASE/etc/pki/rpm-gpg" "$BASE/etc/dnf" + + # dnf reads its repo definitions from inside --installroot. Seed them from + # the container (same release, same arch) so the first transaction has + # somewhere to fetch from and something to verify signatures against. The + # fedora-repos package overwrites these with its own during the transaction. + cp -a /etc/yum.repos.d/. "$BASE/etc/yum.repos.d/" + cp -a /etc/pki/rpm-gpg/. "$BASE/etc/pki/rpm-gpg/" + if [ -d /etc/dnf/vars ]; then cp -a /etc/dnf/vars "$BASE/etc/dnf/"; fi + + log "Installing Fedora ${FEDORA_RELEASE} (${TARGET_ARCH}) — this is the slow part" + # keepcache=1 with a cachedir outside the install root: the downloaded rpms + # outlive both the transaction and the staged tree, so a --fresh rebuild + # re-runs the scriptlets but does not re-download 500-odd packages. + dnf -y \ + --installroot="$BASE" \ + --releasever="$FEDORA_RELEASE" \ + --setopt=cachedir="$DNF_CACHE" \ + --setopt=keepcache=1 \ + --setopt=install_weak_deps=True \ + install "${PACKAGES[@]}" + + printf '%s\n' "$WANT_STAMP" > "$STAMP" +fi # --------------------------------------------------------------------------- -log "Installing Fedora ${FEDORA_RELEASE} (${TARGET_ARCH}) — this is the slow part" +log "Copying the staged base into a working tree" # --------------------------------------------------------------------------- -dnf -y \ - --installroot="$ROOTFS" \ - --releasever="$FEDORA_RELEASE" \ - --setopt=keepcache=0 \ - --setopt=install_weak_deps=True \ - install "${PACKAGES[@]}" +# --reflink=auto is near-instant on btrfs (Fedora's default) and degrades to a +# real copy elsewhere. The working tree gets mutated heavily below — accounts, +# initramfs, bootloader — so the base has to stay untouched. +rm -rf "$ROOTFS" +cp -a --reflink=auto "$BASE" "$ROOTFS" +mkdir -p "$WORK/esp" KVER="$(rpm --root "$ROOTFS" -q kernel-core --qf '%{VERSION}-%{RELEASE}.%{ARCH}\n' \ | sort -V | tail -1)" @@ -329,9 +374,16 @@ log "Compressing" zstd -12 -T0 --rm -f -o "${IMAGE_PATH}.zst" "$IMAGE_PATH" ( cd "$OUT" && sha256sum "${IMAGE_NAME}.img.zst" > "${IMAGE_NAME}.img.zst.sha256" ) +# The intermediate filesystem images are multi-gigabyte and worthless once +# they are inside the disk image. The staged base is the opposite: expensive to +# produce and the whole point of the cache, so it always stays. +rm -f "$WORK/esp.img" "$WORK/boot.img" "$WORK/root.img" +rm -rf "$WORK/esp" if [ "$KEEP_ROOTFS" != "1" ]; then - rm -rf "$WORK" + rm -rf "$ROOTFS" "$WORK/boot" fi log "Done" ls -lh "$OUT" +printf '\nstaged base kept (%s) — the next build reuses it unless the package set changes\n' \ + "$(du -sh "$BASE" 2>/dev/null | cut -f1)" diff --git a/docs/runner-setup.md b/docs/runner-setup.md index 5105add..78926fb 100644 --- a/docs/runner-setup.md +++ b/docs/runner-setup.md @@ -36,17 +36,32 @@ before debugging a build failure. ## Disk space -The build stages a full root filesystem and three filesystem images alongside -the final disk image, under the job workspace. Budget roughly: +The build keeps persistent state in `/var/tmp/c630-build` on each runner: -- `minimal` — about 20 GiB -- `workstation` — about 45 GiB +| Path | Contents | Rough size | +|---|---|---| +| `/var/tmp/c630-build/work//base` | Staged post-dnf root filesystem | 2 GiB minimal, 8 GiB workstation | +| `/var/tmp/c630-build/dnf` | Downloaded rpms and repo metadata | 2 GiB minimal, 6 GiB workstation | + +Plus, transiently in the job workspace, the working rootfs copy, three +filesystem images and the final disk image. Budget roughly 20 GiB for +`minimal` and 45 GiB for `workstation` per runner. + +The workflow prunes cached rpms untouched for 30 days. The staged base is not +pruned — it is invalidated by content hash, not age, so a stale one is +harmless. Delete `/var/tmp/c630-build` to reclaim the space. ## Runtime Every aarch64 binary runs under qemu-user emulation, and rpm scriptlets are the -worst case. Expect roughly 45–90 minutes for `minimal` and several hours for -`workstation`. The workflow's `timeout-minutes` is set to 600 accordingly. +worst case. A cold build is roughly 45–90 minutes for `minimal` and several +hours for `workstation`; `timeout-minutes` is set to 600 accordingly. + +A warm build — one where the package set has not changed since that runner last +built — skips the dnf transaction entirely and finishes in minutes. Because the +state is per-runner and jobs are scheduled across all nine, expect the first +build on each runner to be cold. Pinning this workflow to a single runner with +a dedicated label would make warm builds the norm at the cost of parallelism. If this becomes tiresome, the fix is a native aarch64 runner. Register one with an `aarch64` label and change `runs-on: metal` to `runs-on: aarch64` in