Part of #1. Prerequisite for every other perf issue: none of them merges on an estimate, and every origin merge has to be shown not to regress us.
Problem
Origin's measurement tooling is three things that do not compose: the benchmark CLI subcommand, crates/engine-gpu/examples/hashrate.rs, and the criterion bench. None of them pins the variables that move the number (power limit, batch size, clocks, which kernel was selected), none writes a machine-readable result, and none runs in CI. Origin's own history shows the cost: PR 87 was measured on Apple only and turned out 37% slower on NVIDIA, which PR 92 then had to unpick.
Deliverable
A bench workflow in this repo's .gitea/workflows that runs on a GPU host via the fleet's Gitea runner (see ~/git/architecture/gitea-runners.md; benjy or beast, whichever is not serving inference at the time) and produces one JSON record per run. It runs on every PR here and on every origin-main merge PR.
What it records
Field
Source
commit, kernel id (u64, u64-apple, u32, later cuda)
binary
GPU name, driver, power.limit, clocks.sm at start and end
nvidia-smi --query-gpu
batch size, workers per device, device count
CLI flags under test
MH/s, median of N runs of fixed duration after warm-up
harness
parity: pass/fail, jobs checked
gpu_cpu_parity example
Protocol
Fixed power limit for the run, asserted at start (the fleet's nvidia-power-limit.service sets it; the harness reads and refuses to run if it is not what the matrix says). Power limit is the single biggest confound on these cards.
Warm-up batch, then N timed windows of 30 s, report median and spread. Reject a run whose spread exceeds 2%.
Same batch size across the compared pair unless batch size is the variable under test.
Parity runs on the same build, every time. A faster kernel that is not bit-exact is a consensus failure, not a win.
Where it lives
New crate crates/bench-harness (ours, never touched by origin merges) that drives MinerEngine implementations through the trait, so a new engine crate gets benchmarked by adding one match arm. It reuses origin's gpu_cpu_parity logic by calling it, not copying it, so a change to pow_core at origin flows through.
Also in scope: per-device visibility
The miner exports only aggregate miner_hashes_total. On a two-card host a worker that exits on device-lost halves the hashrate and nothing in the metrics says which card. Add a device label on the hash counter (new metric name, origin's aggregate stays as-is so the fleet dashboard keeps working). Small, but without it the harness cannot tell a two-card result from a one-card result on a two-card host.
Origin coupling
Zero. New crate, new workflow, one additional metric. The only origin file touched is the metrics registration, a few lines behind a // lair: marker.
Definition of done
A PR that changes nothing in the kernel shows the same MH/s within spread on two consecutive runs, and a PR that sets --gpu-batch-size 16000000 shows a different number. Both records are in the closing comment.
Part of #1. Prerequisite for every other `perf` issue: none of them merges on an estimate, and every origin merge has to be shown not to regress us.
## Problem
Origin's measurement tooling is three things that do not compose: the `benchmark` CLI subcommand, `crates/engine-gpu/examples/hashrate.rs`, and the criterion bench. None of them pins the variables that move the number (power limit, batch size, clocks, which kernel was selected), none writes a machine-readable result, and none runs in CI. Origin's own history shows the cost: PR 87 was measured on Apple only and turned out 37% slower on NVIDIA, which PR 92 then had to unpick.
## Deliverable
A `bench` workflow in this repo's `.gitea/workflows` that runs on a GPU host via the fleet's Gitea runner (see `~/git/architecture/gitea-runners.md`; benjy or beast, whichever is not serving inference at the time) and produces one JSON record per run. It runs on every PR here and on every `origin-main` merge PR.
### What it records
| Field | Source |
| --- | --- |
| commit, kernel id (`u64`, `u64-apple`, `u32`, later `cuda`) | binary |
| GPU name, driver, `power.limit`, `clocks.sm` at start and end | `nvidia-smi --query-gpu` |
| batch size, workers per device, device count | CLI flags under test |
| MH/s, median of N runs of fixed duration after warm-up | harness |
| parity: pass/fail, jobs checked | `gpu_cpu_parity` example |
### Protocol
- Fixed power limit for the run, asserted at start (the fleet's `nvidia-power-limit.service` sets it; the harness reads and refuses to run if it is not what the matrix says). Power limit is the single biggest confound on these cards.
- Warm-up batch, then N timed windows of 30 s, report median and spread. Reject a run whose spread exceeds 2%.
- Same batch size across the compared pair unless batch size is the variable under test.
- Parity runs on the same build, every time. A faster kernel that is not bit-exact is a consensus failure, not a win.
### Where it lives
New crate `crates/bench-harness` (ours, never touched by origin merges) that drives `MinerEngine` implementations through the trait, so a new engine crate gets benchmarked by adding one match arm. It reuses origin's `gpu_cpu_parity` logic by calling it, not copying it, so a change to `pow_core` at origin flows through.
## Also in scope: per-device visibility
The miner exports only aggregate `miner_hashes_total`. On a two-card host a worker that exits on device-lost halves the hashrate and nothing in the metrics says which card. Add a `device` label on the hash counter (new metric name, origin's aggregate stays as-is so the fleet dashboard keeps working). Small, but without it the harness cannot tell a two-card result from a one-card result on a two-card host.
## Origin coupling
Zero. New crate, new workflow, one additional metric. The only origin file touched is the metrics registration, a few lines behind a `// lair:` marker.
## Definition of done
A PR that changes nothing in the kernel shows the same MH/s within spread on two consecutive runs, and a PR that sets `--gpu-batch-size 16000000` shows a different number. Both records are in the closing comment.
Constraint found while checking readiness (gongfoo asset/manifest.yml): the only hosts that accept the cuda-13.0 label are beast, benjy and quadbrat, the mining hosts themselves. There is no separate GPU box. So a harness job runs on a card that quantus-miner.service is already driving to full utilisation, and the number is meaningless unless the miner is paused.
Design change for the harness workflow:
The job stops quantus-miner.service on its own host before the run and starts it after, under trap so a failed run cannot leave the host idle. Two sudoers lines (systemctl stop / start quantus-miner.service) for gitea_ci on the GPU hosts; the runner container runs as the gitea agent, not as gitea_ci, so the job does it over loopback ssh exactly like a deploy does.
Record systemctl is-active quantus-miner.service and nvidia_gpu_utilization_ratio at start and refuse to run if anything else is on the card (on beast that means inference; check the helexa model server's slot count is zero or run on a card it is not using).
The reference card for merge decisions is benjy's 4090: single card, no inference, dedicated. Beast's 5090s are the target, so run there too, but treat it as confirmation, not the gate, while it also serves inference.
Cost: every bench run is a few minutes of lost hashing on one host. Acceptable for PRs that touch engine-gpu, engine-cuda or miner-service; skip the harness for PRs that do not (path filter), so a docs change does not idle a card.
Because the harness stops production, it is production-touching in the sense of architecture/generic.md and should not run on every push to main; run it on PRs and on manual dispatch, and rely on #9 for post-merge measurement.
Constraint found while checking readiness (gongfoo `asset/manifest.yml`): the only hosts that accept the `cuda-13.0` label are **beast, benjy and quadbrat**, the mining hosts themselves. There is no separate GPU box. So a harness job runs on a card that `quantus-miner.service` is already driving to full utilisation, and the number is meaningless unless the miner is paused.
Design change for the harness workflow:
- The job stops `quantus-miner.service` on its own host before the run and starts it after, under `trap` so a failed run cannot leave the host idle. Two sudoers lines (`systemctl stop` / `start quantus-miner.service`) for `gitea_ci` on the GPU hosts; the runner container runs as the gitea agent, not as `gitea_ci`, so the job does it over loopback ssh exactly like a deploy does.
- Record `systemctl is-active quantus-miner.service` and `nvidia_gpu_utilization_ratio` at start and refuse to run if anything else is on the card (on beast that means inference; check the helexa model server's slot count is zero or run on a card it is not using).
- The reference card for merge decisions is **benjy's 4090**: single card, no inference, dedicated. Beast's 5090s are the target, so run there too, but treat it as confirmation, not the gate, while it also serves inference.
- Cost: every bench run is a few minutes of lost hashing on one host. Acceptable for PRs that touch `engine-gpu`, `engine-cuda` or `miner-service`; skip the harness for PRs that do not (path filter), so a docs change does not idle a card.
- Because the harness stops production, it is production-touching in the sense of `architecture/generic.md` and should not run on every push to `main`; run it on PRs and on manual dispatch, and rely on #9 for post-merge measurement.
Harness from #11, run twice on benjy by the bench workflow with the miner paused. This is the number every perf workstream is measured against on the reference card.
Both at batch 1M, 5 x 30 s windows after 5 s warm-up, one worker. Run-to-run agreement 0.2%. The live miner on the same host reported miner_hash_rate 144.5 MH/s at the same time, so harness and production agree within noise.
The deploy repo's readme is stale on this card. lair/quantus readme.md has the 4090 at 180.3 MH/s at 250 W and 183.2 at 300 W. On v4.0.2 it is 144 MH/s at 250 W. Either those figures were taken on an earlier kernel (the u64 kernel selection changed across 4.0.0, 4.0.1 and 4.0.2) or at a different limit; either way the harness record is now the authority and the readme table should be regenerated from harness records, not carried forward. Likely the 5090 figures (368 MH/s at 600 W, 281 at 400 W) need re-measuring too before anyone treats them as a target; beast is not mining until the mainnet call, so that is a manual dispatch of the workflow with host=beast when inference can be paused.
Two things learned building the workflow, both now in the code:
Gitea's workflow concurrency group does not serialise runs on one host. The pause, measure, resume sequence runs on the host under a flock in bench-on-host.sh, with per-run file names.
Binaries for the hosts must be built on the cuda-13.0 runner (Fedora 43). The rust runner is Fedora 44 and its binaries fail on the hosts with GLIBC_2.43 not found. Same constraint applies to #8.
Not done in #11: the per-device hash metric from this issue's scope. It goes in with miner_build_info in #9.
## Baseline: origin v4.0.2 wgpu engine, native u64 kernel (2026-09-03)
Harness from #11, run twice on benjy by the `bench` workflow with the miner paused. This is the number every perf workstream is measured against on the reference card.
| run | commit | card | power limit | median MH/s | spread | parity |
| --- | --- | --- | --- | --- | --- | --- |
| actions/runs/4 | bad02fb | RTX 4090 | 250 W | 144.13 | 0.6% | 25/25 OK |
| actions/runs/9 | ca66429 | RTX 4090 | 250 W | 144.39 | 0.2% | 25/25 OK |
Both at batch 1M, 5 x 30 s windows after 5 s warm-up, one worker. Run-to-run agreement 0.2%. The live miner on the same host reported `miner_hash_rate` 144.5 MH/s at the same time, so harness and production agree within noise.
**The deploy repo's readme is stale on this card.** lair/quantus `readme.md` has the 4090 at 180.3 MH/s at 250 W and 183.2 at 300 W. On v4.0.2 it is 144 MH/s at 250 W. Either those figures were taken on an earlier kernel (the u64 kernel selection changed across 4.0.0, 4.0.1 and 4.0.2) or at a different limit; either way the harness record is now the authority and the readme table should be regenerated from harness records, not carried forward. Likely the 5090 figures (368 MH/s at 600 W, 281 at 400 W) need re-measuring too before anyone treats them as a target; beast is not mining until the mainnet call, so that is a manual dispatch of the workflow with `host=beast` when inference can be paused.
Two things learned building the workflow, both now in the code:
- Gitea's workflow concurrency group does not serialise runs on one host. The pause, measure, resume sequence runs on the host under a `flock` in `bench-on-host.sh`, with per-run file names.
- Binaries for the hosts must be built on the `cuda-13.0` runner (Fedora 43). The `rust` runner is Fedora 44 and its binaries fail on the hosts with `GLIBC_2.43 not found`. Same constraint applies to #8.
Not done in #11: the per-device hash metric from this issue's scope. It goes in with `miner_build_info` in #9.
Closing: delivered by #11 (harness, bench workflow, host-side lock) and #14 (per-device metric). Baseline 144.4 MH/s recorded above; the harness has since gated #14, #16 and #17 and produced the 5090 numbers on #3. Run-to-run noise on benjy is about ±1.5%; the interleaving discipline is documented on #3.
Closing: delivered by #11 (harness, `bench` workflow, host-side lock) and #14 (per-device metric). Baseline 144.4 MH/s recorded above; the harness has since gated #14, #16 and #17 and produced the 5090 numbers on #3. Run-to-run noise on benjy is about ±1.5%; the interleaving discipline is documented on #3.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Part of #1. Prerequisite for every other
perfissue: none of them merges on an estimate, and every origin merge has to be shown not to regress us.Problem
Origin's measurement tooling is three things that do not compose: the
benchmarkCLI subcommand,crates/engine-gpu/examples/hashrate.rs, and the criterion bench. None of them pins the variables that move the number (power limit, batch size, clocks, which kernel was selected), none writes a machine-readable result, and none runs in CI. Origin's own history shows the cost: PR 87 was measured on Apple only and turned out 37% slower on NVIDIA, which PR 92 then had to unpick.Deliverable
A
benchworkflow in this repo's.gitea/workflowsthat runs on a GPU host via the fleet's Gitea runner (see~/git/architecture/gitea-runners.md; benjy or beast, whichever is not serving inference at the time) and produces one JSON record per run. It runs on every PR here and on everyorigin-mainmerge PR.What it records
u64,u64-apple,u32, latercuda)power.limit,clocks.smat start and endnvidia-smi --query-gpugpu_cpu_parityexampleProtocol
nvidia-power-limit.servicesets it; the harness reads and refuses to run if it is not what the matrix says). Power limit is the single biggest confound on these cards.Where it lives
New crate
crates/bench-harness(ours, never touched by origin merges) that drivesMinerEngineimplementations through the trait, so a new engine crate gets benchmarked by adding one match arm. It reuses origin'sgpu_cpu_paritylogic by calling it, not copying it, so a change topow_coreat origin flows through.Also in scope: per-device visibility
The miner exports only aggregate
miner_hashes_total. On a two-card host a worker that exits on device-lost halves the hashrate and nothing in the metrics says which card. Add adevicelabel on the hash counter (new metric name, origin's aggregate stays as-is so the fleet dashboard keeps working). Small, but without it the harness cannot tell a two-card result from a one-card result on a two-card host.Origin coupling
Zero. New crate, new workflow, one additional metric. The only origin file touched is the metrics registration, a few lines behind a
// lair:marker.Definition of done
A PR that changes nothing in the kernel shows the same MH/s within spread on two consecutive runs, and a PR that sets
--gpu-batch-size 16000000shows a different number. Both records are in the closing comment.Constraint found while checking readiness (gongfoo
asset/manifest.yml): the only hosts that accept thecuda-13.0label are beast, benjy and quadbrat, the mining hosts themselves. There is no separate GPU box. So a harness job runs on a card thatquantus-miner.serviceis already driving to full utilisation, and the number is meaningless unless the miner is paused.Design change for the harness workflow:
quantus-miner.serviceon its own host before the run and starts it after, undertrapso a failed run cannot leave the host idle. Two sudoers lines (systemctl stop/start quantus-miner.service) forgitea_cion the GPU hosts; the runner container runs as the gitea agent, not asgitea_ci, so the job does it over loopback ssh exactly like a deploy does.systemctl is-active quantus-miner.serviceandnvidia_gpu_utilization_ratioat start and refuse to run if anything else is on the card (on beast that means inference; check the helexa model server's slot count is zero or run on a card it is not using).engine-gpu,engine-cudaorminer-service; skip the harness for PRs that do not (path filter), so a docs change does not idle a card.architecture/generic.mdand should not run on every push tomain; run it on PRs and on manual dispatch, and rely on #9 for post-merge measurement.Baseline: origin v4.0.2 wgpu engine, native u64 kernel (2026-09-03)
Harness from #11, run twice on benjy by the
benchworkflow with the miner paused. This is the number every perf workstream is measured against on the reference card.bad02fbca66429Both at batch 1M, 5 x 30 s windows after 5 s warm-up, one worker. Run-to-run agreement 0.2%. The live miner on the same host reported
miner_hash_rate144.5 MH/s at the same time, so harness and production agree within noise.The deploy repo's readme is stale on this card. lair/quantus
readme.mdhas the 4090 at 180.3 MH/s at 250 W and 183.2 at 300 W. On v4.0.2 it is 144 MH/s at 250 W. Either those figures were taken on an earlier kernel (the u64 kernel selection changed across 4.0.0, 4.0.1 and 4.0.2) or at a different limit; either way the harness record is now the authority and the readme table should be regenerated from harness records, not carried forward. Likely the 5090 figures (368 MH/s at 600 W, 281 at 400 W) need re-measuring too before anyone treats them as a target; beast is not mining until the mainnet call, so that is a manual dispatch of the workflow withhost=beastwhen inference can be paused.Two things learned building the workflow, both now in the code:
flockinbench-on-host.sh, with per-run file names.cuda-13.0runner (Fedora 43). Therustrunner is Fedora 44 and its binaries fail on the hosts withGLIBC_2.43 not found. Same constraint applies to #8.Not done in #11: the per-device hash metric from this issue's scope. It goes in with
miner_build_infoin #9.Closing: delivered by #11 (harness,
benchworkflow, host-side lock) and #14 (per-device metric). Baseline 144.4 MH/s recorded above; the harness has since gated #14, #16 and #17 and produced the 5090 numbers on #3. Run-to-run noise on benjy is about ±1.5%; the interleaving discipline is documented on #3.