architecture/gitea-runners.md

# Gitea Actions runners

The catalogue of CI runner images available to workflows on the self-hosted Gitea
(`git.lair.cafe` / `git.internal`), what each provides, and — importantly — how to add or
extend one when your workflow needs a toolchain no existing runner ships. Read this
alongside `deployment-gitea-actions.md` §5 ("Runner images"), which states the governing
rule this doc makes concrete:

> **Bake build dependencies into the runner image, not into the workflow.** A workflow
> should never `dnf install` / `apt-get install` at run time.

If you find yourself reaching for a package-install step in a job, stop — the answer is to
extend an image here, not to mutate the runner at run time.

---

## 1. Where runners come from: gongfoo

Runners are **ephemeral, single-use Podman containers** managed by
[`gongfoo`](https://git.lair.cafe/gongfoo/gongfoo) (`git.lair.cafe/gongfoo/gongfoo`). A
controller watches the Gitea job queue and, per `(image, label-set)`, spawns a fresh
container that registers with Gitea, claims exactly one job, runs it, and exits. There is
no shared state between jobs and no long-lived runner to mutate — which is *why* toolchains
must live in the image.

The image definitions (Containerfiles + `build.sh`) live in **`gongfoo/images/`**. Each
registered runner is declared in **`gongfoo/asset/manifest.yml`** under `runner_images`,
which maps an image to its **labels** (what `runs-on:` matches), CPU/memory reservation,
and any reaper overrides.

---

## 2. The runner catalogue

`runs-on:` in a workflow selects a runner by **label**. A job's `runs-on` label set must be
a **subset** of a runner image's labels; when several images qualify, the one with the
**fewest extra labels wins** (so `runs-on: rust` lands on `runner-rust`, not
`runner-rust-gtk3` which also carries `rust`). See the gongfoo readme ("Queue handling →
Image selection") for the full placement semantics.

| `runs-on:` | Image | Base | Key toolchain beyond the base | CPU / MEM |
| --- | --- | --- | --- | --- |
| `fedora-43` | `runner-fedora-43` | Fedora 43 | git, curl, jq, **nodejs + npm + pnpm**, podman-remote, rsync, python3-pip + `yq` | 2 / 2 GiB |
| `fedora-44` | `runner-fedora-44` | Fedora 44 | same package set as fedora-43; **this image's Containerfile is the `GITEA_RUNNER_VERSION` source of truth** | 2 / 2 GiB |
| `rust` | `runner-rust` | `runner-fedora-44` | rustup **stable** (gnu + `x86_64-unknown-linux-musl` targets, clippy, rustfmt), `sccache`, gcc, musl-gcc, cmake, pkg-config, openssl-devel, perl-core | 4 / 4 GiB |
| `rust-gtk3` | `runner-rust-gtk3` | `runner-rust` | GTK3 stack: gtk3-devel, glib2/gdk-pixbuf2/pango-devel, webkit2gtk4.1-devel, libsoup3-devel, libappindicator-gtk3-devel (Tauri/GTK desktop builds) | 4 / 4 GiB |
| `rpm` | `runner-rpm` | `runner-rust` | rpm-build, rpmdevtools, rpmlint, createrepo_c, copr-cli, rpm-sign, sequoia-sq, gnupg2, autotools, openssh-clients | 2 / 2 GiB |
| `ffmpeg` | `runner-ffmpeg` | `runner-rust` | rpmfusion-free, ffmpeg-devel, clang (for `ffmpeg-sys`/`ac-ffmpeg`-style builds) | 4 / 4 GiB |
| `infra` | `runner-infra` | `runner-fedora-44` | `stalwart-cli` (+ xz for its tarballs). The home for infra service CLIs / IaC tooling | 2 / 2 GiB |
| `cuda-13.0` | `runner-cuda-13.0` | `runner-fedora-43` | rustup stable (as `runner-rust`, inlined) **+ CUDA 13.0** (nvcc, cudart/nvrtc/cublas/cusparse/curand/cusolver-devel, cudnn9, nccl, cmake, ninja, gcc-c++) | 8 / 32 GiB |
| `ubuntu-24.04` | `runner-ubuntu-24.04` | Ubuntu 24.04 | git, curl, jq, nodejs + npm + pnpm, rsync, python3-pip + `yq` | 2 / 2 GiB |
| `deb` | `runner-deb` | `runner-rust-ubuntu` | Debian packaging: build-essential, debhelper, devscripts, dh-make, dpkg-dev, fakeroot, lintian, reprepro (+ apt `rustc`/`cargo` from the base) | 2 / 2 GiB |

Notes:

- **`runner-rust-ubuntu`** (Ubuntu 24.04 + apt `rustc`/`cargo`/clippy/rustfmt + sccache)
  exists only as the base for `runner-deb`; it is **not** registered as a standalone
  runner, so there is no `rust-ubuntu` label. Default to `rust` (Fedora, rustup) for Rust
  work; the Ubuntu Rust base is for producing `.deb`s.
- Every Fedora/Ubuntu base ships **`pnpm` installed globally via `npm install -g pnpm`**,
  **not** corepack — Fedora's/Ubuntu's `nodejs` rpm/deb does not bundle corepack. Call
  `pnpm` directly; do **not** `corepack enable` (see §4 gotchas).
- **`ssh` + `rsync` are on every runner.** `rsync` is in each base's install list, and the
  `git` every image installs (for `actions/checkout`) hard-requires `git-core`, which in
  turn hard-requires `openssh-clients` — so the ssh client is present even though the bare
  `fedora`/`ubuntu` container base doesn't ship it and weak deps are disabled. A CI-driven
  deploy (ssh + rsync to targets) therefore runs fine on `fedora-43`/`fedora-44`; it does
  **not** need `infra`. (`runner-infra`'s own Containerfile re-installs `openssh-clients`
  belt-and-suspenders and its comment claiming "the ssh client is not in the base" is
  misleading — it's pulled by git-core regardless.)
- The **`rust`** toolchain is `rustup` stable, not Fedora's distro rustc, specifically so
  the `x86_64-unknown-linux-musl` std matches the compiler and static musl builds work.
- `sccache` is installed on the Rust images but backing it with the shared S3 cache is
  opt-in **per workflow** (set `RUSTC_WRAPPER=sccache` + `SCCACHE_*`/`AWS_*` — see helexa's
  `ci.yml`), not baked into the image.

### Lineage

```
Fedora 43 ── runner-fedora-43 ─┬─ runner-cuda-13.0        (CUDA needs gcc 15 → F43, not F44)
                               (rust toolchain inlined)

Fedora 44 ── runner-fedora-44 ─┬─ runner-infra
                               └─ runner-rust ─┬─ runner-rust-gtk3
                                               ├─ runner-rpm
                                               └─ runner-ffmpeg

Ubuntu 24.04 ─ runner-ubuntu-24.04 ─ runner-rust-ubuntu ─ runner-deb
```

`runner-cuda-13.0` deliberately branches from Fedora **43**, not the `runner-rust` (Fedora
44) chain: CUDA 13.0's `nvcc` rejects host compilers newer than gcc 15, and Fedora 44 ships
gcc 16. It re-inlines the rustup recipe rather than inheriting it.

---

## 3. Choosing the right runner

| Your job does… | Use `runs-on:` | Why |
| --- | --- | --- |
| `cargo build/test/clippy/fmt` | `rust` | the only images with a Rust toolchain (`rust`, `rpm`, `ffmpeg`, `rust-gtk3`, `cuda-13.0`); plain `fedora-*` have **no cargo** |
| Build a Vite/React/npm/pnpm frontend | `fedora-43` / `fedora-44` | node + npm + **pnpm** preinstalled |
| **Deploy over ssh + rsync** (CI-driven deploy) | `fedora-43` / `fedora-44` | ssh (via git-core's `openssh-clients` dep) + rsync + curl are present on every runner |
| Infra service admin (Stalwart mail, IaC CLIs) | `infra` | ships `stalwart-cli` and is the home for infra service tooling |
| Build/sign RPMs, publish to COPR | `rpm` | rpmbuild, createrepo_c, copr-cli, signing |
| Build `.deb`s | `deb` | debhelper/devscripts/reprepro |
| CUDA / GPU-accelerated Rust (candle, flash-attn) | `cuda-13.0` | nvcc + CUDA dev libs; pinned to GPU hosts |
| ffmpeg-linked builds | `ffmpeg` | ffmpeg-devel + clang |
| GTK3 / Tauri desktop | `rust-gtk3` | GTK/webkit dev libs on top of Rust |
| Plain shell / `jq` / `yq` / git plumbing | `fedora-43` / `fedora-44` / `ubuntu-24.04` | smallest images |

**Multi-step deploys commonly split across two runners** — a `rust` job to build + upload
the artifact, and a `fedora-43` job (`needs:` the build) to ssh/rsync it to the targets —
mostly so the deploy job isn't billed the heavier `rust` image (4 CPU / 4 GiB) just to run
ssh. This is a convenience, not a necessity: the `rust` runner also has git-core's ssh and
rsync, so a single job could build and deploy if you prefer.

---

## 4. Gotchas that cost a run

These are the traps that make a job fail seconds in with a "command not found":

- **`runs-on: fedora-43` for a Rust build → `cargo: command not found`.** The `fedora-*`
  images are node/shell runners with no Rust. Use `rust`.
- **Not a gotcha: ssh on `fedora-*`.** It's tempting to think the fedora runners lack an
  ssh client (the bare container base doesn't ship one, and `runner-infra` re-installs
  `openssh-clients` with a comment saying so). But every runner installs `git` for
  checkout, and `git-core` hard-requires `openssh-clients` — so `ssh` is present on all of
  them. Deploy on `fedora-43`; you do not need `infra` for ssh/rsync.
- **`corepack enable` anywhere → fails / no-op.** No image bundles corepack; `pnpm` is
  already on `PATH` (installed via `npm i -g pnpm`). Just run `pnpm install` / `pnpm build`.
  A `packageManager` field in `package.json` is honoured for information but standalone
  pnpm won't auto-switch versions, so pin your lockfile and use the image's pnpm.
- **Assuming `mikefarah/yq` syntax.** `yq` here is the PyPI `kislyuk/yq` (jq-syntax
  wrapper), not the Go `mikefarah/yq`. Pipe jq expressions accordingly.
- **`runs-on: rust` for `cargo build --target x86_64-unknown-linux-musl`.** Supported —
  the musl std + musl-gcc are present. But if the whole target fleet is the same
  distro/version as the runner, a **native** build is simpler and avoids the musl surface
  entirely (`deployment-gitea-actions.md` §6: build on a runner no newer than the oldest
  target).

---

## 5. Adding or extending a runner

When no runner ships a dependency your workflow needs, **add it to an image** — never
`dnf`/`apt` install at run time (slow, flaky, and runners may be unprivileged). Prefer, in
order:

1. **Extend an existing base.** If an infra/service CLI is missing, add it to
   `runner-infra`. If a system dev lib is missing for a Rust crate's `-sys` build, add it
   to `runner-rust` (or a narrower child). Editing a shared base benefits every downstream
   image. This is usually a one-line addition to a `dnf install` list.

2. **Add a new image** only when the toolchain is large, conflicting, or host-pinned
   (GPUs, a specific compiler version). Create `gongfoo/images/runner-<name>/` with a
   `Containerfile` that `FROM`s the closest existing image, and a `build.sh`.

Then **register it** in `gongfoo/asset/manifest.yml` under `runner_images`:

```yaml
- name: runner-<name>
  image_ref: git.lair.cafe/gongfoo/runner-<name>:latest
  labels: [<label>]            # what `runs-on:` matches; keep the set minimal
  cpu_request: 2               # placement reservation + basis for the cgroup cap
  mem_request_mb: 2048
  # max_runtime_secs / log_stall_secs: reaper overrides for unusually long builds
```

Label discipline (matters for placement):

- Keep the label set **minimal and specific**. A child image inherits its purpose from its
  own label, not its base's — e.g. `runner-cuda-13.0` is labelled `[cuda-13.0]` alone (not
  `[fedora-43, cuda-13.0]`) so the subset-match doesn't route plain `fedora-43` jobs onto
  scarce GPU hosts.
- Hosts can restrict which labels they accept (`allowed_labels` in the manifest) to protect
  scarce hardware; a new specialised label must land on a host that allows it.

### How images get built

The **`images`** workflow in `gongfoo` (`.gitea/workflows/images.yml`) builds and pushes on
push to `main` touching `images/**`, and on a **daily cron** that also refreshes the
`act_runner` binary and distro packages (so images don't drift). Base images build on
bare-metal `runs-on: [metal, podman]` runners and `podman push` to `git.lair.cafe`. Tool
versions are pinned by `ARG` in the Containerfile (e.g. `GITEA_RUNNER_VERSION`,
`SCCACHE_VERSION`, `STALWART_CLI_VERSION`) and the workflow overrides pins to the latest
upstream release at build time; the `ARG` default is the offline fallback.

After a new/updated image is pushed and the manifest change is deployed (gongfoo picks up
`runner_images` from `manifest.yml`), the new label is immediately available to `runs-on:`.

---

## 6. Checklist for a workflow author

1. Match each job to a runner by what it *runs*, not by distro habit (§3). Rust → `rust`;
   frontend → `fedora-*`; ssh/rsync deploy → `fedora-43` (ssh + rsync are on every runner).
2. Never install packages at run time. Missing tool → extend an image (§5).
3. Don't `corepack enable`; call `pnpm` directly.
4. Reserve `infra` for its actual payload (`stalwart-cli` / infra CLIs), not for ssh.
5. If you add an image, register it in `gongfoo/asset/manifest.yml` with a minimal label
   set and sensible CPU/MEM.