blackbeard/miner
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Benchmark and parity harness: the gate every perf change and every origin merge passes #2

New Issue
Closed
opened 2026-09-03 09:07:59 +00:00 by grenade · 3 comments
grenade commented 2026-09-03 09:07:59 +00:00
Owner
Copy Link

Part of #1. Prerequisite for every other perf issue: none of them merges on an estimate, and every origin merge has to be shown not to regress us.

Problem

Origin's measurement tooling is three things that do not compose: the benchmark CLI subcommand, crates/engine-gpu/examples/hashrate.rs, and the criterion bench. None of them pins the variables that move the number (power limit, batch size, clocks, which kernel was selected), none writes a machine-readable result, and none runs in CI. Origin's own history shows the cost: PR 87 was measured on Apple only and turned out 37% slower on NVIDIA, which PR 92 then had to unpick.

Deliverable

A bench workflow in this repo's .gitea/workflows that runs on a GPU host via the fleet's Gitea runner (see ~/git/architecture/gitea-runners.md; benjy or beast, whichever is not serving inference at the time) and produces one JSON record per run. It runs on every PR here and on every origin-main merge PR.

What it records

Field Source
commit, kernel id (u64, u64-apple, u32, later cuda) binary
GPU name, driver, power.limit, clocks.sm at start and end nvidia-smi --query-gpu
batch size, workers per device, device count CLI flags under test
MH/s, median of N runs of fixed duration after warm-up harness
parity: pass/fail, jobs checked gpu_cpu_parity example

Protocol

  • Fixed power limit for the run, asserted at start (the fleet's nvidia-power-limit.service sets it; the harness reads and refuses to run if it is not what the matrix says). Power limit is the single biggest confound on these cards.
  • Warm-up batch, then N timed windows of 30 s, report median and spread. Reject a run whose spread exceeds 2%.
  • Same batch size across the compared pair unless batch size is the variable under test.
  • Parity runs on the same build, every time. A faster kernel that is not bit-exact is a consensus failure, not a win.

Where it lives

New crate crates/bench-harness (ours, never touched by origin merges) that drives MinerEngine implementations through the trait, so a new engine crate gets benchmarked by adding one match arm. It reuses origin's gpu_cpu_parity logic by calling it, not copying it, so a change to pow_core at origin flows through.

Also in scope: per-device visibility

The miner exports only aggregate miner_hashes_total. On a two-card host a worker that exits on device-lost halves the hashrate and nothing in the metrics says which card. Add a device label on the hash counter (new metric name, origin's aggregate stays as-is so the fleet dashboard keeps working). Small, but without it the harness cannot tell a two-card result from a one-card result on a two-card host.

Origin coupling

Zero. New crate, new workflow, one additional metric. The only origin file touched is the metrics registration, a few lines behind a // lair: marker.

Definition of done

A PR that changes nothing in the kernel shows the same MH/s within spread on two consecutive runs, and a PR that sets --gpu-batch-size 16000000 shows a different number. Both records are in the closing comment.

Part of #1. Prerequisite for every other `perf` issue: none of them merges on an estimate, and every origin merge has to be shown not to regress us. ## Problem Origin's measurement tooling is three things that do not compose: the `benchmark` CLI subcommand, `crates/engine-gpu/examples/hashrate.rs`, and the criterion bench. None of them pins the variables that move the number (power limit, batch size, clocks, which kernel was selected), none writes a machine-readable result, and none runs in CI. Origin's own history shows the cost: PR 87 was measured on Apple only and turned out 37% slower on NVIDIA, which PR 92 then had to unpick. ## Deliverable A `bench` workflow in this repo's `.gitea/workflows` that runs on a GPU host via the fleet's Gitea runner (see `~/git/architecture/gitea-runners.md`; benjy or beast, whichever is not serving inference at the time) and produces one JSON record per run. It runs on every PR here and on every `origin-main` merge PR. ### What it records | Field | Source | | --- | --- | | commit, kernel id (`u64`, `u64-apple`, `u32`, later `cuda`) | binary | | GPU name, driver, `power.limit`, `clocks.sm` at start and end | `nvidia-smi --query-gpu` | | batch size, workers per device, device count | CLI flags under test | | MH/s, median of N runs of fixed duration after warm-up | harness | | parity: pass/fail, jobs checked | `gpu_cpu_parity` example | ### Protocol - Fixed power limit for the run, asserted at start (the fleet's `nvidia-power-limit.service` sets it; the harness reads and refuses to run if it is not what the matrix says). Power limit is the single biggest confound on these cards. - Warm-up batch, then N timed windows of 30 s, report median and spread. Reject a run whose spread exceeds 2%. - Same batch size across the compared pair unless batch size is the variable under test. - Parity runs on the same build, every time. A faster kernel that is not bit-exact is a consensus failure, not a win. ### Where it lives New crate `crates/bench-harness` (ours, never touched by origin merges) that drives `MinerEngine` implementations through the trait, so a new engine crate gets benchmarked by adding one match arm. It reuses origin's `gpu_cpu_parity` logic by calling it, not copying it, so a change to `pow_core` at origin flows through. ## Also in scope: per-device visibility The miner exports only aggregate `miner_hashes_total`. On a two-card host a worker that exits on device-lost halves the hashrate and nothing in the metrics says which card. Add a `device` label on the hash counter (new metric name, origin's aggregate stays as-is so the fleet dashboard keeps working). Small, but without it the harness cannot tell a two-card result from a one-card result on a two-card host. ## Origin coupling Zero. New crate, new workflow, one additional metric. The only origin file touched is the metrics registration, a few lines behind a `// lair:` marker. ## Definition of done A PR that changes nothing in the kernel shows the same MH/s within spread on two consecutive runs, and a PR that sets `--gpu-batch-size 16000000` shows a different number. Both records are in the closing comment.
grenade added the origin-couplingperf labels 2026-09-03 09:10:14 +00:00
grenade commented 2026-09-03 09:34:27 +00:00
Author
Owner
Copy Link

Constraint found while checking readiness (gongfoo asset/manifest.yml): the only hosts that accept the cuda-13.0 label are beast, benjy and quadbrat, the mining hosts themselves. There is no separate GPU box. So a harness job runs on a card that quantus-miner.service is already driving to full utilisation, and the number is meaningless unless the miner is paused.

Design change for the harness workflow:

  • The job stops quantus-miner.service on its own host before the run and starts it after, under trap so a failed run cannot leave the host idle. Two sudoers lines (systemctl stop / start quantus-miner.service) for gitea_ci on the GPU hosts; the runner container runs as the gitea agent, not as gitea_ci, so the job does it over loopback ssh exactly like a deploy does.
  • Record systemctl is-active quantus-miner.service and nvidia_gpu_utilization_ratio at start and refuse to run if anything else is on the card (on beast that means inference; check the helexa model server's slot count is zero or run on a card it is not using).
  • The reference card for merge decisions is benjy's 4090: single card, no inference, dedicated. Beast's 5090s are the target, so run there too, but treat it as confirmation, not the gate, while it also serves inference.
  • Cost: every bench run is a few minutes of lost hashing on one host. Acceptable for PRs that touch engine-gpu, engine-cuda or miner-service; skip the harness for PRs that do not (path filter), so a docs change does not idle a card.
  • Because the harness stops production, it is production-touching in the sense of architecture/generic.md and should not run on every push to main; run it on PRs and on manual dispatch, and rely on #9 for post-merge measurement.
Constraint found while checking readiness (gongfoo `asset/manifest.yml`): the only hosts that accept the `cuda-13.0` label are **beast, benjy and quadbrat**, the mining hosts themselves. There is no separate GPU box. So a harness job runs on a card that `quantus-miner.service` is already driving to full utilisation, and the number is meaningless unless the miner is paused. Design change for the harness workflow: - The job stops `quantus-miner.service` on its own host before the run and starts it after, under `trap` so a failed run cannot leave the host idle. Two sudoers lines (`systemctl stop` / `start quantus-miner.service`) for `gitea_ci` on the GPU hosts; the runner container runs as the gitea agent, not as `gitea_ci`, so the job does it over loopback ssh exactly like a deploy does. - Record `systemctl is-active quantus-miner.service` and `nvidia_gpu_utilization_ratio` at start and refuse to run if anything else is on the card (on beast that means inference; check the helexa model server's slot count is zero or run on a card it is not using). - The reference card for merge decisions is **benjy's 4090**: single card, no inference, dedicated. Beast's 5090s are the target, so run there too, but treat it as confirmation, not the gate, while it also serves inference. - Cost: every bench run is a few minutes of lost hashing on one host. Acceptable for PRs that touch `engine-gpu`, `engine-cuda` or `miner-service`; skip the harness for PRs that do not (path filter), so a docs change does not idle a card. - Because the harness stops production, it is production-touching in the sense of `architecture/generic.md` and should not run on every push to `main`; run it on PRs and on manual dispatch, and rely on #9 for post-merge measurement.
grenade commented 2026-09-03 10:15:14 +00:00
Author
Owner
Copy Link

Baseline: origin v4.0.2 wgpu engine, native u64 kernel (2026-09-03)

Harness from #11, run twice on benjy by the bench workflow with the miner paused. This is the number every perf workstream is measured against on the reference card.

run commit card power limit median MH/s spread parity
actions/runs/4 bad02fb RTX 4090 250 W 144.13 0.6% 25/25 OK
actions/runs/9 ca66429 RTX 4090 250 W 144.39 0.2% 25/25 OK

Both at batch 1M, 5 x 30 s windows after 5 s warm-up, one worker. Run-to-run agreement 0.2%. The live miner on the same host reported miner_hash_rate 144.5 MH/s at the same time, so harness and production agree within noise.

The deploy repo's readme is stale on this card. lair/quantus readme.md has the 4090 at 180.3 MH/s at 250 W and 183.2 at 300 W. On v4.0.2 it is 144 MH/s at 250 W. Either those figures were taken on an earlier kernel (the u64 kernel selection changed across 4.0.0, 4.0.1 and 4.0.2) or at a different limit; either way the harness record is now the authority and the readme table should be regenerated from harness records, not carried forward. Likely the 5090 figures (368 MH/s at 600 W, 281 at 400 W) need re-measuring too before anyone treats them as a target; beast is not mining until the mainnet call, so that is a manual dispatch of the workflow with host=beast when inference can be paused.

Two things learned building the workflow, both now in the code:

  • Gitea's workflow concurrency group does not serialise runs on one host. The pause, measure, resume sequence runs on the host under a flock in bench-on-host.sh, with per-run file names.
  • Binaries for the hosts must be built on the cuda-13.0 runner (Fedora 43). The rust runner is Fedora 44 and its binaries fail on the hosts with GLIBC_2.43 not found. Same constraint applies to #8.

Not done in #11: the per-device hash metric from this issue's scope. It goes in with miner_build_info in #9.

## Baseline: origin v4.0.2 wgpu engine, native u64 kernel (2026-09-03) Harness from #11, run twice on benjy by the `bench` workflow with the miner paused. This is the number every perf workstream is measured against on the reference card. | run | commit | card | power limit | median MH/s | spread | parity | | --- | --- | --- | --- | --- | --- | --- | | actions/runs/4 | bad02fb | RTX 4090 | 250 W | 144.13 | 0.6% | 25/25 OK | | actions/runs/9 | ca66429 | RTX 4090 | 250 W | 144.39 | 0.2% | 25/25 OK | Both at batch 1M, 5 x 30 s windows after 5 s warm-up, one worker. Run-to-run agreement 0.2%. The live miner on the same host reported `miner_hash_rate` 144.5 MH/s at the same time, so harness and production agree within noise. **The deploy repo's readme is stale on this card.** lair/quantus `readme.md` has the 4090 at 180.3 MH/s at 250 W and 183.2 at 300 W. On v4.0.2 it is 144 MH/s at 250 W. Either those figures were taken on an earlier kernel (the u64 kernel selection changed across 4.0.0, 4.0.1 and 4.0.2) or at a different limit; either way the harness record is now the authority and the readme table should be regenerated from harness records, not carried forward. Likely the 5090 figures (368 MH/s at 600 W, 281 at 400 W) need re-measuring too before anyone treats them as a target; beast is not mining until the mainnet call, so that is a manual dispatch of the workflow with `host=beast` when inference can be paused. Two things learned building the workflow, both now in the code: - Gitea's workflow concurrency group does not serialise runs on one host. The pause, measure, resume sequence runs on the host under a `flock` in `bench-on-host.sh`, with per-run file names. - Binaries for the hosts must be built on the `cuda-13.0` runner (Fedora 43). The `rust` runner is Fedora 44 and its binaries fail on the hosts with `GLIBC_2.43 not found`. Same constraint applies to #8. Not done in #11: the per-device hash metric from this issue's scope. It goes in with `miner_build_info` in #9.
grenade commented 2026-09-03 15:22:51 +00:00
Author
Owner
Copy Link

Closing: delivered by #11 (harness, bench workflow, host-side lock) and #14 (per-device metric). Baseline 144.4 MH/s recorded above; the harness has since gated #14, #16 and #17 and produced the 5090 numbers on #3. Run-to-run noise on benjy is about ±1.5%; the interleaving discipline is documented on #3.

Closing: delivered by #11 (harness, `bench` workflow, host-side lock) and #14 (per-device metric). Baseline 144.4 MH/s recorded above; the harness has since gated #14, #16 and #17 and produced the 5090 numbers on #3. Run-to-run noise on benjy is about ±1.5%; the interleaving discipline is documented on #3.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
needs-benchmark
Gain is estimated, not measured; needs a run on a 5090 host
 needs-chain
May require a matching change in quantus/chain
 origin-coupling
Touches how the lair fork tracks Quantus-Network/quantus-miner
 perf
Mining throughput workstream
No Label origin-coupling perf
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
grenade
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: blackbeard/miner#2
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?