gongfoo

Ephemeral, autoscaling Gitea Actions runners on Podman. Written in Rust.

gongfoo watches the Gitea job queue and spawns single-use containerised runners across a fleet of hosts on demand. Each runner picks up exactly one job, executes it, and exits. The controller maintains desired capacity per runner-image / label-set, balanced across hosts via a spread-placement strategy.

The name is a deliberate misspelling of gongfu — the tea ceremony style of many small, precise, repeated brews. That's what the runners are: small, precise, ephemeral.

Why

The previous setup ran multiple long-lived act_runner processes per host via templated systemd units on bare metal. That works but is rigid: capacity is fixed, distro/version variants require static provisioning, and there's shared state between jobs. gongfoo replaces that with on-demand containerised runners that scale with queue depth and isolate every job in a fresh container.

Architecture

┌─────────────┐     ┌──────────────────┐     ┌─────────────────────┐
│  Gitea API  │◄────│   controller     │────►│  agent (per host)   │
│  (queue)    │     │   (reconciler)   │     │                     │
└─────────────┘     └──────────────────┘     └─────────────────────┘
                            │                          │
                            ▼                          ▼
                    ┌──────────────────┐     ┌─────────────────────┐
                    │   PostgreSQL     │     │  Podman + ephemeral │
                    │  (runner state)  │     │  act_runner         │
                    └──────────────────┘     └─────────────────────┘

• controller polls Gitea for queued jobs, reconciles desired vs actual runner count per (image, label-set), and instructs agents to spawn or reports orphans for reaping. Postgres is the source of truth.
• agent runs on each runner host, receives spawn RPCs, drives the local Podman socket, watches container events, and reports lifecycle transitions back to the controller.
• runners are short-lived Podman containers running act_runner in --once / --ephemeral mode. Each container registers with Gitea, takes one job, exits. The container has the host's Podman socket mounted so it can spawn job containers for the actual workflow steps.

Workspace layout

gongfoo/
├── Cargo.toml                    # workspace root
├── crates/
│   ├── gongfoo-proto/            # shared types and RPC definitions
│   ├── gongfoo-controller/       # reconciler + Gitea/Postgres client
│   ├── gongfoo-agent/            # per-host daemon, Podman driver
│   └── gongfoo-migrations/       # sqlx migrations
├── images/
│   ├── runner-fedora-43/         # Fedora 43 base: act_runner + nodejs + CA
│   ├── runner-rust/              # Fedora + rust, cargo, clippy, rustfmt
│   ├── runner-rpm/               # Fedora + rpm-build, createrepo_c
│   ├── runner-ubuntu-24.04/      # Ubuntu 24.04 base: act_runner + nodejs + CA
│   └── runner-deb/               # Ubuntu + debhelper, devscripts, reprepro
├── asset/                        # deployment artifacts
│   ├── manifest.yml              # environments → components → hosts
│   ├── systemd/                  # service units, cert-reload .path units
│   ├── firewalld/                # named service definitions
│   ├── config/                   # config templates with {{SECRET}} placeholders
│   └── sql/                      # bootstrap SQL for Postgres role/db creation
└── script/
    └── deploy.sh                 # deploy postgres, controller, agent components

Crates

Crate	Binary	Purpose
gongfoo-proto	—	Shared request/response types, error enums, state machine
gongfoo-controller	gongfoo-controller	Long-running daemon. Polls Gitea, reconciles, dispatches spawn RPCs
gongfoo-agent	gongfoo-agent	Long-running daemon on runner hosts. Receives RPCs, drives Podman
gongfoo-migrations	—	sqlx migration files and an embed helper for the controller

Vocabulary

The codebase uses tea-ceremony terms where they're genuinely clearer than generic alternatives, and plain English elsewhere. Specifically:

• brew loop — the reconciliation cycle (every 5s by default).

host, image, job, runner are kept as plain terms.

Placement strategy

Spread: pick the host with the fewest active runners that has capacity for the requested image, breaking ties randomly. CPU and memory budgets are checked against the host's reported totals.

Per-host label restrictions

Hosts can opt into a label allow-list via hosts.allowed_labels TEXT[], configured per-host in asset/manifest.yml. Semantics:

• Empty allowed_labels (the default) means no restriction — the host accepts any runner image. Existing hosts behave exactly as before.
• A non-empty list means the host only accepts images whose labels are a subset of that list. Example: beast with allowed_labels: [cuda-13.0] accepts an image labeled [cuda-13.0] but rejects one labeled [fedora-43, cuda-13.0] because fedora-43 isn't in beast's allow-list. A multi-purpose host that wants CUDA and Fedora-rust workloads lists both labels.

The intent is hardware protection: a host with scarce or specialised resources (GPUs, large RAM, non-gongfoo workloads) declares exactly which label-sets are allowed to land there. Everything else is rejected at placement time and queues until a permissive host has capacity.

Corollary worth knowing: an image with labels = '{}' would match every host including restricted ones (empty set is a subset of anything). Every current image has at least one label, so this is theoretical — but if you ever register a label-less image it will run anywhere.

This is intentionally simple. Once cichlid (a separate distributed orchestration project) is ready, placement will be delegated to its rules engine.

Queue handling

The controller polls Gitea every 5s (the brew loop) and reconciles per label-set.

• What counts as queued. Only jobs Gitea marks queued (ready to run) trigger spawns. Jobs in waiting status — gated by needs: dependencies that haven't completed — are skipped. They re-enter the queue once Gitea promotes them to queued.
• FIFO by oldest job. Label sets are processed in the order of their oldest queued job's workflow-run created_at. The oldest waiting work gets the first attempt at any free capacity each tick. Jobs with missing timestamps sort to the front so unknown-age work isn't starved.
• No reservation, throughput over strict age. When the front-of-queue label set can't place (its image's CPU/memory don't fit on any host), the controller skips it and tries the next. A 2-CPU rpm job can land ahead of a queued 4-CPU rust-gtk3 if only 2 CPU is free. Large jobs land as soon as a sufficiently large capacity window opens.
• Image selection prefers the most specific match. When several registered images satisfy a label set, the one carrying the fewest extra labels wins (fedora-43 → runner-fedora-43, not runner-rust-gtk3 which also carries fedora-43). Name is the stable tie-break.
• Hysteresis. Scale-up requires 2 consecutive observations above the current runner count to absorb single-tick queue blips.
• Stuck-registered reap at 5min. A runner that registers with Gitea but doesn't claim a job within 5 minutes is force-reaped, freeing its capacity. Self-healing: if a claim was about to happen, the next tick spawns a fresh runner for the still-queued job.

Runner state machine

pending → spawning → started → registered → running → completed → reaped
                  ↘         ↘              ↘        ↘
                   failed ──────────────────────────→ reaped

• pending: controller decided to spawn, placement chosen
• spawning: spawn RPC sent to agent
• started: container running, Gitea registration not yet confirmed
• registered: confirmed in Gitea's runner list
• running: runner is busy executing a job (detected via Gitea API)
• completed/failed: runner exited
• reaped: cleanup done, row awaiting deletion

Deployment

./script/deploy.sh <environment> [component...] [--dry-run]
./script/deploy.sh prod controller agent
./script/deploy.sh prod postgres

Components: postgres (bootstrap DB, register hosts and images), controller, agent (deployed to all hosts in manifest).

Both controller and agent communicate over mTLS using the host's step-ca-issued certificates.

Status

Working. Dogfooding — gongfoo's own CI runs on gongfoo-managed runners.

Repository

git.lair.cafe/gongfoo/gongfoo

Licence

TBD.