docs: add gitea-runners.md — CI runner catalogue and selection guide

Document the gongfoo-managed runner images (labels, lineage, toolchains, CPU/MEM), how `runs-on:` label matching selects one, which runner to pick for each job type, and how to add/extend an image instead of installing packages at workflow run time. Captures the traps that cost a run: `fedora-*` has no cargo (use `rust`); the base fedora images ship rsync but no ssh client (deploy jobs use `infra`); `corepack` isn't bundled — pnpm is preinstalled via npm, call it directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
2026-07-08 12:06:51 +03:00
parent 135601d8aa
commit 6043b8e6c3
2 changed files with 191 additions and 0 deletions
--- a/gitea-runners.md
+++ b/gitea-runners.md
@@ -0,0 +1,190 @@
+# Gitea Actions runners
+
+The catalogue of CI runner images available to workflows on the self-hosted Gitea
+(`git.lair.cafe` / `git.internal`), what each provides, and — importantly — how to add or
+extend one when your workflow needs a toolchain no existing runner ships. Read this
+alongside `deployment-gitea-actions.md` §5 ("Runner images"), which states the governing
+rule this doc makes concrete:
+
+> **Bake build dependencies into the runner image, not into the workflow.** A workflow
+> should never `dnf install` / `apt-get install` at run time.
+
+If you find yourself reaching for a package-install step in a job, stop — the answer is to
+extend an image here, not to mutate the runner at run time.
+
+---
+
+## 1. Where runners come from: gongfoo
+
+Runners are **ephemeral, single-use Podman containers** managed by
+[`gongfoo`](https://git.lair.cafe/gongfoo/gongfoo) (`git.lair.cafe/gongfoo/gongfoo`). A
+controller watches the Gitea job queue and, per `(image, label-set)`, spawns a fresh
+container that registers with Gitea, claims exactly one job, runs it, and exits. There is
+no shared state between jobs and no long-lived runner to mutate — which is *why* toolchains
+must live in the image.
+
+The image definitions (Containerfiles + `build.sh`) live in **`gongfoo/images/`**. Each
+registered runner is declared in **`gongfoo/asset/manifest.yml`** under `runner_images`,
+which maps an image to its **labels** (what `runs-on:` matches), CPU/memory reservation,
+and any reaper overrides.
+
+---
+
+## 2. The runner catalogue
+
+`runs-on:` in a workflow selects a runner by **label**. A job's `runs-on` label set must be
+a **subset** of a runner image's labels; when several images qualify, the one with the
+**fewest extra labels wins** (so `runs-on: rust` lands on `runner-rust`, not
+`runner-rust-gtk3` which also carries `rust`). See the gongfoo readme ("Queue handling →
+Image selection") for the full placement semantics.
+
+| `runs-on:` | Image | Base | Key toolchain beyond the base | CPU / MEM |
+| --- | --- | --- | --- | --- |
+| `fedora-43` | `runner-fedora-43` | Fedora 43 | git, curl, jq, **nodejs + npm + pnpm**, podman-remote, rsync, python3-pip + `yq` | 2 / 2 GiB |
+| `fedora-44` | `runner-fedora-44` | Fedora 44 | same package set as fedora-43; **this image's Containerfile is the `GITEA_RUNNER_VERSION` source of truth** | 2 / 2 GiB |
+| `rust` | `runner-rust` | `runner-fedora-44` | rustup **stable** (gnu + `x86_64-unknown-linux-musl` targets, clippy, rustfmt), `sccache`, gcc, musl-gcc, cmake, pkg-config, openssl-devel, perl-core | 4 / 4 GiB |
+| `rust-gtk3` | `runner-rust-gtk3` | `runner-rust` | GTK3 stack: gtk3-devel, glib2/gdk-pixbuf2/pango-devel, webkit2gtk4.1-devel, libsoup3-devel, libappindicator-gtk3-devel (Tauri/GTK desktop builds) | 4 / 4 GiB |
+| `rpm` | `runner-rpm` | `runner-rust` | rpm-build, rpmdevtools, rpmlint, createrepo_c, copr-cli, rpm-sign, sequoia-sq, gnupg2, autotools, openssh-clients | 2 / 2 GiB |
+| `ffmpeg` | `runner-ffmpeg` | `runner-rust` | rpmfusion-free, ffmpeg-devel, clang (for `ffmpeg-sys`/`ac-ffmpeg`-style builds) | 4 / 4 GiB |
+| `infra` | `runner-infra` | `runner-fedora-44` | **openssh-clients + rsync + xz**, `stalwart-cli`. The home for infra deploy/reconcile tooling | 2 / 2 GiB |
+| `cuda-13.0` | `runner-cuda-13.0` | `runner-fedora-43` | rustup stable (as `runner-rust`, inlined) **+ CUDA 13.0** (nvcc, cudart/nvrtc/cublas/cusparse/curand/cusolver-devel, cudnn9, nccl, cmake, ninja, gcc-c++) | 8 / 32 GiB |
+| `ubuntu-24.04` | `runner-ubuntu-24.04` | Ubuntu 24.04 | git, curl, jq, nodejs + npm + pnpm, rsync, python3-pip + `yq` | 2 / 2 GiB |
+| `deb` | `runner-deb` | `runner-rust-ubuntu` | Debian packaging: build-essential, debhelper, devscripts, dh-make, dpkg-dev, fakeroot, lintian, reprepro (+ apt `rustc`/`cargo` from the base) | 2 / 2 GiB |
+
+Notes:
+
+- **`runner-rust-ubuntu`** (Ubuntu 24.04 + apt `rustc`/`cargo`/clippy/rustfmt + sccache)
+  exists only as the base for `runner-deb`; it is **not** registered as a standalone
+  runner, so there is no `rust-ubuntu` label. Default to `rust` (Fedora, rustup) for Rust
+  work; the Ubuntu Rust base is for producing `.deb`s.
+- Every Fedora/Ubuntu base ships **`pnpm` installed globally via `npm install -g pnpm`**,
+  **not** corepack — Fedora's/Ubuntu's `nodejs` rpm/deb does not bundle corepack. Call
+  `pnpm` directly; do **not** `corepack enable` (see §4 gotchas).
+- The **`rust`** toolchain is `rustup` stable, not Fedora's distro rustc, specifically so
+  the `x86_64-unknown-linux-musl` std matches the compiler and static musl builds work.
+- `sccache` is installed on the Rust images but backing it with the shared S3 cache is
+  opt-in **per workflow** (set `RUSTC_WRAPPER=sccache` + `SCCACHE_*`/`AWS_*` — see helexa's
+  `ci.yml`), not baked into the image.
+
+### Lineage
+
+```
+Fedora 43 ── runner-fedora-43 ─┬─ runner-cuda-13.0        (CUDA needs gcc 15 → F43, not F44)
+                               (rust toolchain inlined)
+
+Fedora 44 ── runner-fedora-44 ─┬─ runner-infra
+                               └─ runner-rust ─┬─ runner-rust-gtk3
+                                               ├─ runner-rpm
+                                               └─ runner-ffmpeg
+
+Ubuntu 24.04 ─ runner-ubuntu-24.04 ─ runner-rust-ubuntu ─ runner-deb
+```
+
+`runner-cuda-13.0` deliberately branches from Fedora **43**, not the `runner-rust` (Fedora
+44) chain: CUDA 13.0's `nvcc` rejects host compilers newer than gcc 15, and Fedora 44 ships
+gcc 16. It re-inlines the rustup recipe rather than inheriting it.
+
+---
+
+## 3. Choosing the right runner
+
+| Your job does… | Use `runs-on:` | Why |
+| --- | --- | --- |
+| `cargo build/test/clippy/fmt` | `rust` | the only images with a Rust toolchain (`rust`, `rpm`, `ffmpeg`, `rust-gtk3`, `cuda-13.0`); plain `fedora-*` have **no cargo** |
+| Build a Vite/React/npm/pnpm frontend | `fedora-43` / `fedora-44` | node + npm + **pnpm** preinstalled |
+| **Deploy over ssh + rsync** (CI-driven deploy) | `infra` | the **only** general image with an **ssh client**; the base `fedora-*` images have `rsync` but **not** `openssh-clients` |
+| Build/sign RPMs, publish to COPR | `rpm` | rpmbuild, createrepo_c, copr-cli, signing |
+| Build `.deb`s | `deb` | debhelper/devscripts/reprepro |
+| CUDA / GPU-accelerated Rust (candle, flash-attn) | `cuda-13.0` | nvcc + CUDA dev libs; pinned to GPU hosts |
+| ffmpeg-linked builds | `ffmpeg` | ffmpeg-devel + clang |
+| GTK3 / Tauri desktop | `rust-gtk3` | GTK/webkit dev libs on top of Rust |
+| Plain shell / `jq` / `yq` / git plumbing | `fedora-43` / `fedora-44` / `ubuntu-24.04` | smallest images |
+
+**Multi-step deploys typically split across two runners:** a `rust` job to build the
+artifact + upload it, and an `infra` job (`needs:` the build) to ssh/rsync it to the
+targets. Don't try to do both on one runner — no single image has both cargo and an ssh
+client.
+
+---
+
+## 4. Gotchas that cost a run
+
+These are the traps that make a job fail seconds in with a "command not found":
+
+- **`runs-on: fedora-43` for a Rust build → `cargo: command not found`.** The `fedora-*`
+  images are node/shell runners with no Rust. Use `rust`.
+- **`ssh`/`rsync-over-ssh` on `fedora-*` → `ssh: command not found`.** The base images ship
+  `rsync` but not the ssh client. Deploy jobs go on `infra`.
+- **`corepack enable` anywhere → fails / no-op.** No image bundles corepack; `pnpm` is
+  already on `PATH` (installed via `npm i -g pnpm`). Just run `pnpm install` / `pnpm build`.
+  A `packageManager` field in `package.json` is honoured for information but standalone
+  pnpm won't auto-switch versions, so pin your lockfile and use the image's pnpm.
+- **Assuming `mikefarah/yq` syntax.** `yq` here is the PyPI `kislyuk/yq` (jq-syntax
+  wrapper), not the Go `mikefarah/yq`. Pipe jq expressions accordingly.
+- **`runs-on: rust` for `cargo build --target x86_64-unknown-linux-musl`.** Supported —
+  the musl std + musl-gcc are present. But if the whole target fleet is the same
+  distro/version as the runner, a **native** build is simpler and avoids the musl surface
+  entirely (`deployment-gitea-actions.md` §6: build on a runner no newer than the oldest
+  target).
+
+---
+
+## 5. Adding or extending a runner
+
+When no runner ships a dependency your workflow needs, **add it to an image** — never
+`dnf`/`apt` install at run time (slow, flaky, and runners may be unprivileged). Prefer, in
+order:
+
+1. **Extend an existing base.** If an infra/service CLI is missing, add it to
+   `runner-infra`. If a system dev lib is missing for a Rust crate's `-sys` build, add it
+   to `runner-rust` (or a narrower child). Editing a shared base benefits every downstream
+   image. This is usually a one-line addition to a `dnf install` list.
+
+2. **Add a new image** only when the toolchain is large, conflicting, or host-pinned
+   (GPUs, a specific compiler version). Create `gongfoo/images/runner-<name>/` with a
+   `Containerfile` that `FROM`s the closest existing image, and a `build.sh`.
+
+Then **register it** in `gongfoo/asset/manifest.yml` under `runner_images`:
+
+```yaml
+- name: runner-<name>
+  image_ref: git.lair.cafe/gongfoo/runner-<name>:latest
+  labels: [<label>]            # what `runs-on:` matches; keep the set minimal
+  cpu_request: 2               # placement reservation + basis for the cgroup cap
+  mem_request_mb: 2048
+  # max_runtime_secs / log_stall_secs: reaper overrides for unusually long builds
+```
+
+Label discipline (matters for placement):
+
+- Keep the label set **minimal and specific**. A child image inherits its purpose from its
+  own label, not its base's — e.g. `runner-cuda-13.0` is labelled `[cuda-13.0]` alone (not
+  `[fedora-43, cuda-13.0]`) so the subset-match doesn't route plain `fedora-43` jobs onto
+  scarce GPU hosts.
+- Hosts can restrict which labels they accept (`allowed_labels` in the manifest) to protect
+  scarce hardware; a new specialised label must land on a host that allows it.
+
+### How images get built
+
+The **`images`** workflow in `gongfoo` (`.gitea/workflows/images.yml`) builds and pushes on
+push to `main` touching `images/**`, and on a **daily cron** that also refreshes the
+`act_runner` binary and distro packages (so images don't drift). Base images build on
+bare-metal `runs-on: [metal, podman]` runners and `podman push` to `git.lair.cafe`. Tool
+versions are pinned by `ARG` in the Containerfile (e.g. `GITEA_RUNNER_VERSION`,
+`SCCACHE_VERSION`, `STALWART_CLI_VERSION`) and the workflow overrides pins to the latest
+upstream release at build time; the `ARG` default is the offline fallback.
+
+After a new/updated image is pushed and the manifest change is deployed (gongfoo picks up
+`runner_images` from `manifest.yml`), the new label is immediately available to `runs-on:`.
+
+---
+
+## 6. Checklist for a workflow author
+
+1. Match each job to a runner by what it *runs*, not by distro habit (§3). Rust → `rust`;
+   frontend → `fedora-*`; ssh/rsync deploy → `infra`.
+2. Never install packages at run time. Missing tool → extend an image (§5).
+3. Don't `corepack enable`; call `pnpm` directly.
+4. Split build (`rust`) from deploy (`infra`) — no image has both cargo and ssh.
+5. If you add an image, register it in `gongfoo/asset/manifest.yml` with a minimal label
+   set and sensible CPU/MEM.
--- a/readme.md
+++ b/readme.md
@@ -12,6 +12,7 @@ The goal is boring consistency: the same crate layout, the same deploy flow, the

 - **`generic.md`** — the baseline. Applies to every project unless that project explicitly overrides a section. Covers workspace layout, separation of concerns, configuration, secrets, deployment, service accounts, firewalld, SELinux, and code quality.
 - **`deployment-gitea-actions.md`** — CI-driven deployment via a Gitea Actions workflow, as an alternative to the `deploy.sh` + `manifest.yml` flow in `generic.md` §7. The workflow is the source of infra truth; the runner deploys as a scoped `gitea_ci` user.
+- **`gitea-runners.md`** — the catalogue of `gongfoo`-managed CI runner images (what `runs-on:` label gives you which toolchain), how to pick the right one, and how to add or extend an image instead of `dnf`-installing at run time. Makes `deployment-gitea-actions.md` §5 concrete.
 - **`internal-tls.md`** — provisioning and renewing per-service internal TLS certs (`<service>.internal`) for mesh-only nginx vhosts, extending the PKI conventions in `generic.md` §11.
 - **`external-tls.md`** — publicly-trusted certs for WAN-facing vhosts via Let's Encrypt (certbot, Cloudflare DNS-01, ECDSA). The external counterpart to `internal-tls.md`.
 - **`reverse-proxies.md`** — the per-site nginx edge proxies (`oolon` for kosherinata, `hanzalova.internal` for the office), what sits behind each, the public-vs-mesh access paths, and the per-vhost cert choice. Names the topology behind `generic.md` §11 Ingress.