architecture/gitea-runners.md

Gitea Actions runners

The catalogue of CI runner images available to workflows on the self-hosted Gitea (git.lair.cafe / git.internal), what each provides, and — importantly — how to add or extend one when your workflow needs a toolchain no existing runner ships. Read this alongside deployment-gitea-actions.md §5 ("Runner images"), which states the governing rule this doc makes concrete:

Bake build dependencies into the runner image, not into the workflow. A workflow should never dnf install / apt-get install at run time.

If you find yourself reaching for a package-install step in a job, stop — the answer is to extend an image here, not to mutate the runner at run time.

---

1. Where runners come from: gongfoo

Runners are ephemeral, single-use Podman containers managed by gongfoo (git.lair.cafe/gongfoo/gongfoo). A controller watches the Gitea job queue and, per (image, label-set), spawns a fresh container that registers with Gitea, claims exactly one job, runs it, and exits. There is no shared state between jobs and no long-lived runner to mutate — which is why toolchains must live in the image.

The image definitions (Containerfiles + build.sh) live in gongfoo/images/. Each registered runner is declared in gongfoo/asset/manifest.yml under runner_images, which maps an image to its labels (what runs-on: matches), CPU/memory reservation, and any reaper overrides.

---

2. The runner catalogue

runs-on: in a workflow selects a runner by label. A job's runs-on label set must be a subset of a runner image's labels; when several images qualify, the one with the fewest extra labels wins (so runs-on: rust lands on runner-rust, not runner-rust-gtk3 which also carries rust). See the gongfoo readme ("Queue handling → Image selection") for the full placement semantics.

runs-on:	Image	Base	Key toolchain beyond the base	CPU / MEM
fedora-43	runner-fedora-43	Fedora 43	git, curl, jq, nodejs + npm + pnpm, podman-remote, rsync, python3-pip + yq	2 / 2 GiB
fedora-44	runner-fedora-44	Fedora 44	same package set as fedora-43; this image's Containerfile is the GITEA_RUNNER_VERSION source of truth	2 / 2 GiB
rust	runner-rust	runner-fedora-44	rustup stable (gnu + x86_64-unknown-linux-musl targets, clippy, rustfmt), sccache, gcc, musl-gcc, cmake, pkg-config, openssl-devel, perl-core	4 / 4 GiB
rust-gtk3	runner-rust-gtk3	runner-rust	GTK3 stack: gtk3-devel, glib2/gdk-pixbuf2/pango-devel, webkit2gtk4.1-devel, libsoup3-devel, libappindicator-gtk3-devel (Tauri/GTK desktop builds)	4 / 4 GiB
rpm	runner-rpm	runner-rust	rpm-build, rpmdevtools, rpmlint, createrepo_c, copr-cli, rpm-sign, sequoia-sq, gnupg2, autotools, openssh-clients	2 / 2 GiB
ffmpeg	runner-ffmpeg	runner-rust	rpmfusion-free, ffmpeg-devel, clang (for ffmpeg-sys/ac-ffmpeg-style builds)	4 / 4 GiB
infra	runner-infra	runner-fedora-44	stalwart-cli (+ xz for its tarballs). The home for infra service CLIs / IaC tooling	2 / 2 GiB
cuda-13.0	runner-cuda-13.0	runner-fedora-43	rustup stable (as runner-rust, inlined) + CUDA 13.0 (nvcc, cudart/nvrtc/cublas/cusparse/curand/cusolver-devel, cudnn9, nccl, cmake, ninja, gcc-c++)	8 / 32 GiB
ubuntu-24.04	runner-ubuntu-24.04	Ubuntu 24.04	git, curl, jq, nodejs + npm + pnpm, rsync, python3-pip + yq	2 / 2 GiB
deb	runner-deb	runner-rust-ubuntu	Debian packaging: build-essential, debhelper, devscripts, dh-make, dpkg-dev, fakeroot, lintian, reprepro (+ apt rustc/cargo from the base)	2 / 2 GiB

Notes:

• runner-rust-ubuntu (Ubuntu 24.04 + apt rustc/cargo/clippy/rustfmt + sccache) exists only as the base for runner-deb; it is not registered as a standalone runner, so there is no rust-ubuntu label. Default to rust (Fedora, rustup) for Rust work; the Ubuntu Rust base is for producing .debs.
• Every Fedora/Ubuntu base ships pnpm installed globally via npm install -g pnpm, not corepack — Fedora's/Ubuntu's nodejs rpm/deb does not bundle corepack. Call pnpm directly; do not corepack enable (see §4 gotchas).
• ssh + rsync are on every runner. rsync is in each base's install list, and the git every image installs (for actions/checkout) hard-requires git-core, which in turn hard-requires openssh-clients — so the ssh client is present even though the bare fedora/ubuntu container base doesn't ship it and weak deps are disabled. A CI-driven deploy (ssh + rsync to targets) therefore runs fine on fedora-43/fedora-44; it does not need infra. (runner-infra's own Containerfile re-installs openssh-clients belt-and-suspenders and its comment claiming "the ssh client is not in the base" is misleading — it's pulled by git-core regardless.)
• The rust toolchain is rustup stable, not Fedora's distro rustc, specifically so the x86_64-unknown-linux-musl std matches the compiler and static musl builds work.
• sccache is installed on the Rust images but backing it with the shared S3 cache is opt-in per workflow (set RUSTC_WRAPPER=sccache + SCCACHE_*/AWS_* — see helexa's ci.yml), not baked into the image.

Lineage

Fedora 43 ── runner-fedora-43 ─┬─ runner-cuda-13.0        (CUDA needs gcc 15 → F43, not F44)
                               (rust toolchain inlined)

Fedora 44 ── runner-fedora-44 ─┬─ runner-infra
                               └─ runner-rust ─┬─ runner-rust-gtk3
                                               ├─ runner-rpm
                                               └─ runner-ffmpeg

Ubuntu 24.04 ─ runner-ubuntu-24.04 ─ runner-rust-ubuntu ─ runner-deb

runner-cuda-13.0 deliberately branches from Fedora 43, not the runner-rust (Fedora 44) chain: CUDA 13.0's nvcc rejects host compilers newer than gcc 15, and Fedora 44 ships gcc 16. It re-inlines the rustup recipe rather than inheriting it.

---

3. Choosing the right runner

Your job does…	Use runs-on:	Why
cargo build/test/clippy/fmt	rust	the only images with a Rust toolchain (rust, rpm, ffmpeg, rust-gtk3, cuda-13.0); plain fedora-* have no cargo
Build a Vite/React/npm/pnpm frontend	fedora-43 / fedora-44	node + npm + pnpm preinstalled
Deploy over ssh + rsync (CI-driven deploy)	fedora-43 / fedora-44	ssh (via git-core's openssh-clients dep) + rsync + curl are present on every runner
Infra service admin (Stalwart mail, IaC CLIs)	infra	ships stalwart-cli and is the home for infra service tooling
Build/sign RPMs, publish to COPR	rpm	rpmbuild, createrepo_c, copr-cli, signing
Build .debs	deb	debhelper/devscripts/reprepro
CUDA / GPU-accelerated Rust (candle, flash-attn)	cuda-13.0	nvcc + CUDA dev libs; pinned to GPU hosts
ffmpeg-linked builds	ffmpeg	ffmpeg-devel + clang
GTK3 / Tauri desktop	rust-gtk3	GTK/webkit dev libs on top of Rust
Plain shell / jq / yq / git plumbing	fedora-43 / fedora-44 / ubuntu-24.04	smallest images

Multi-step deploys commonly split across two runners — a rust job to build + upload the artifact, and a fedora-43 job (needs: the build) to ssh/rsync it to the targets — mostly so the deploy job isn't billed the heavier rust image (4 CPU / 4 GiB) just to run ssh. This is a convenience, not a necessity: the rust runner also has git-core's ssh and rsync, so a single job could build and deploy if you prefer.

---

4. Gotchas that cost a run

These are the traps that make a job fail seconds in with a "command not found":

• runs-on: fedora-43 for a Rust build → cargo: command not found. The fedora-* images are node/shell runners with no Rust. Use rust.
• Not a gotcha: ssh on fedora-*. It's tempting to think the fedora runners lack an ssh client (the bare container base doesn't ship one, and runner-infra re-installs openssh-clients with a comment saying so). But every runner installs git for checkout, and git-core hard-requires openssh-clients — so ssh is present on all of them. Deploy on fedora-43; you do not need infra for ssh/rsync.
• corepack enable anywhere → fails / no-op. No image bundles corepack; pnpm is already on PATH (installed via npm i -g pnpm). Just run pnpm install / pnpm build. A packageManager field in package.json is honoured for information but standalone pnpm won't auto-switch versions, so pin your lockfile and use the image's pnpm.
• Assuming mikefarah/yq syntax. yq here is the PyPI kislyuk/yq (jq-syntax wrapper), not the Go mikefarah/yq. Pipe jq expressions accordingly.
• runs-on: rust for cargo build --target x86_64-unknown-linux-musl. Supported — the musl std + musl-gcc are present. But if the whole target fleet is the same distro/version as the runner, a native build is simpler and avoids the musl surface entirely (deployment-gitea-actions.md §6: build on a runner no newer than the oldest target).

---

5. Adding or extending a runner

When no runner ships a dependency your workflow needs, add it to an image — never dnf/apt install at run time (slow, flaky, and runners may be unprivileged). Prefer, in order:

1. Extend an existing base. If an infra/service CLI is missing, add it to runner-infra. If a system dev lib is missing for a Rust crate's -sys build, add it to runner-rust (or a narrower child). Editing a shared base benefits every downstream image. This is usually a one-line addition to a dnf install list.

2. Add a new image only when the toolchain is large, conflicting, or host-pinned (GPUs, a specific compiler version). Create gongfoo/images/runner-<name>/ with a Containerfile that FROMs the closest existing image, and a build.sh.

Then register it in gongfoo/asset/manifest.yml under runner_images:

- name: runner-<name>
  image_ref: git.lair.cafe/gongfoo/runner-<name>:latest
  labels: [<label>]            # what `runs-on:` matches; keep the set minimal
  cpu_request: 2               # placement reservation + basis for the cgroup cap
  mem_request_mb: 2048
  # max_runtime_secs / log_stall_secs: reaper overrides for unusually long builds

Label discipline (matters for placement):

• Keep the label set minimal and specific. A child image inherits its purpose from its own label, not its base's — e.g. runner-cuda-13.0 is labelled [cuda-13.0] alone (not [fedora-43, cuda-13.0]) so the subset-match doesn't route plain fedora-43 jobs onto scarce GPU hosts.
• Hosts can restrict which labels they accept (allowed_labels in the manifest) to protect scarce hardware; a new specialised label must land on a host that allows it.

How images get built

The images workflow in gongfoo (.gitea/workflows/images.yml) builds and pushes on push to main touching images/**, and on a daily cron that also refreshes the act_runner binary and distro packages (so images don't drift). Base images build on bare-metal runs-on: [metal, podman] runners and podman push to git.lair.cafe. Tool versions are pinned by ARG in the Containerfile (e.g. GITEA_RUNNER_VERSION, SCCACHE_VERSION, STALWART_CLI_VERSION) and the workflow overrides pins to the latest upstream release at build time; the ARG default is the offline fallback.

After a new/updated image is pushed and the manifest change is deployed (gongfoo picks up runner_images from manifest.yml), the new label is immediately available to runs-on:.

---

6. Checklist for a workflow author

1. Match each job to a runner by what it runs, not by distro habit (§3). Rust → rust; frontend → fedora-*; ssh/rsync deploy → fedora-43 (ssh + rsync are on every runner).
2. Never install packages at run time. Missing tool → extend an image (§5).
3. Don't corepack enable; call pnpm directly.
4. Reserve infra for its actual payload (stalwart-cli / infra CLIs), not for ssh.
5. If you add an image, register it in gongfoo/asset/manifest.yml with a minimal label set and sensible CPU/MEM.