bench: quantus-bench harness and a workflow that measures on a mining host #11

Merged

grenade merged 3 commits from bench/harness into main 2026-09-03 10:19:34 +00:00

Conversation 2 Commits 3 Files Changed 7 +746

grenade commented 2026-09-03 10:00:18 +00:00

Owner

Copy Link

Implements #2, the measurement gate. Stacked on #10 (its two commits are included; merge #10 first).

What changed

crates/bench-harness (lair-owned; origin merges never touch it), binary quantus-bench:

• Drives GpuEngine through the MinerEngine trait, one worker thread per card, fixed timed windows after a warm-up. Reports per-worker median MH/s and spread, exit 5 when spread exceeds 2%.
• Captures nvidia-smi state (driver, power limit, draw, SM clock, temperature) before and after. --expect-power-limit refuses the run (exit 3) if the enforced limit is not the expected one; that is the largest confound on these cards.
• GPU/CPU parity on random jobs, same shape as engine-gpu's gpu_cpu_parity example, exit 4 on any mismatch.
• One JSON record per run (schema: 1) and a markdown summary on stdout.
• build.rs embeds the commit like miner-cli's.

.gitea/workflows/bench.yaml:

• Runner containers on the GPU hosts have no device passthrough (gongfoo devices: None), so the binary is built on cuda-13.0 and executed on the host over ssh as gitea_ci. cuda-13.0 rather than rust because the rust image is Fedora 44 and the hosts are Fedora 43: a binary from the rust runner fails with GLIBC_2.43 not found.
• Stops quantus-miner.service for the window and starts it again under a trap. Refuses to measure a card that is still busy with the miner stopped (on beast that would be inference).
• Benjy by default. Host, duration, runs, batch size, workers are dispatch inputs. PRs touching engine or service crates trigger it and get the summary as a PR comment; one measurement per host at a time.
• Sudoers stop/start grants: lair/quantus#5, already applied to benjy and quadbrat.

Workspace: one line in members, marked # lair:. Cargo.lock gains the crate.

Validation

• fmt, clippy -D warnings, build: clean locally.
• Smoke run on quadbrat (Fedora 43, built there, miner left running so the number is not meaningful): parity 3/3 OK, record written, GPU state captured, exit 0.
• This PR triggers the workflow itself; its comment below is the first real measurement on benjy at 250 W.

Not in this PR: the per-device hash metric from #2's scope. It goes with the build-info gauge in #9.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5

Implements #2, the measurement gate. Stacked on #10 (its two commits are included; merge #10 first). ## What changed **`crates/bench-harness`** (lair-owned; origin merges never touch it), binary `quantus-bench`: - Drives `GpuEngine` through the `MinerEngine` trait, one worker thread per card, fixed timed windows after a warm-up. Reports per-worker median MH/s and spread, exit 5 when spread exceeds 2%. - Captures `nvidia-smi` state (driver, power limit, draw, SM clock, temperature) before and after. `--expect-power-limit` refuses the run (exit 3) if the enforced limit is not the expected one; that is the largest confound on these cards. - GPU/CPU parity on random jobs, same shape as `engine-gpu`'s `gpu_cpu_parity` example, exit 4 on any mismatch. - One JSON record per run (`schema: 1`) and a markdown summary on stdout. - `build.rs` embeds the commit like `miner-cli`'s. **`.gitea/workflows/bench.yaml`**: - Runner containers on the GPU hosts have no device passthrough (gongfoo `devices: None`), so the binary is built on `cuda-13.0` and executed on the host over ssh as `gitea_ci`. `cuda-13.0` rather than `rust` because the `rust` image is Fedora 44 and the hosts are Fedora 43: a binary from the `rust` runner fails with `GLIBC_2.43 not found`. - Stops `quantus-miner.service` for the window and starts it again under a `trap`. Refuses to measure a card that is still busy with the miner stopped (on beast that would be inference). - Benjy by default. Host, duration, runs, batch size, workers are dispatch inputs. PRs touching engine or service crates trigger it and get the summary as a PR comment; one measurement per host at a time. - Sudoers `stop`/`start` grants: lair/quantus#5, already applied to benjy and quadbrat. **Workspace**: one line in `members`, marked `# lair:`. `Cargo.lock` gains the crate. ## Validation - fmt, clippy `-D warnings`, build: clean locally. - Smoke run on quadbrat (Fedora 43, built there, miner left running so the number is not meaningful): parity 3/3 OK, record written, GPU state captured, exit 0. - This PR triggers the workflow itself; its comment below is the first real measurement on benjy at 250 W. Not in this PR: the per-device hash metric from #2's scope. It goes with the build-info gauge in #9. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5

grenade added 3 commits 2026-09-03 10:00:19 +00:00

ci: gitea workflow, drop inherited github workflows, embed commit in --version ... 094536a341

First lair-only change on top of origin v4.0.2 (quantus/miner#8, #1).

- Delete .github/ (workflows and the disk action). Gitea Actions reads
  .github/workflows too, so origin's GitHub-hosted jobs would queue forever
  for runners that do not exist here. Standing resolution on every
  origin-main merge: `git rm -r .github/workflows`.
- Add .gitea/workflows/ci.yml on the `rust` runner: fmt, clippy with
  -D warnings (same invocation as clippy.sh), build, test, doc, and a check
  that the built binary reports a commit in --version.
- crates/miner-cli/build.rs embeds MINER_BUILD_SHA (env override, else
  git rev-parse with a -dirty marker, else "unknown") and --version prints
  "<semver> (<sha>)". A deploy that follows main needs to assert the running
  binary is the commit it shipped; the semver alone cannot do that.

Origin coupling: .gitea/ and build.rs are new files; main.rs gains one const
and one attribute argument behind a `// lair:` marker.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5

ci: add rustfmt and clippy components explicitly ... db6316713a

rustup auto-installs the pinned toolchain from rust-toolchain on the runner
but not its listed components; the fmt job failed with "cargo-fmt is not
installed for the toolchain". Add the component in the two jobs that need it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5

bench: quantus-bench harness and a workflow that measures on a mining host ... bad02fbc33

quantus/miner#2. The measurement gate every kernel and submission change
passes before merge.

crates/bench-harness (lair-owned, never touched by origin merges):
- drives GpuEngine through MinerEngine, one worker thread per card, with
  fixed timed windows after a warm-up; reports per-worker median MH/s and
  spread and refuses (exit 5) when spread exceeds 2%
- captures nvidia-smi state (driver, power limit, draw, clock, temperature)
  before and after, and refuses to run (exit 3) when the enforced power limit
  is not the expected one, since that is the largest confound on these cards
- GPU/CPU parity on random jobs, same shape as engine-gpu's gpu_cpu_parity
  example, exit 4 on any mismatch
- one JSON record per run (schema 1) plus a markdown summary on stdout
- build.rs embeds the commit like miner-cli's

.gitea/workflows/bench.yaml:
- runner containers on the GPU hosts have no device passthrough, so the
  binary is built on cuda-13.0 (Fedora 43, matching the hosts; the rust
  image is Fedora 44 and its binaries need a newer glibc) and executed on
  the host over ssh as gitea_ci
- stops quantus-miner.service for the window and starts it again under a
  trap; refuses to measure a card that is busy with the miner stopped
- benjy (4090, dedicated) by default; host, duration, runs, batch size and
  workers are dispatch inputs; PRs touching engine or service crates trigger
  it and get the summary as a comment
- one measurement per host at a time (concurrency group)

The stop/start sudoers grants were added to lair/quantus infra-setup.sh
(lair/quantus#5) and applied to benjy and quadbrat.

Origin coupling: one line in the workspace members list, marked.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5

grenade added 1 commit 2026-09-03 10:03:27 +00:00

bench: allow more worker threads than cards to measure submission overlap ... 5f3c51c5cb

Two threads on one card overlap one thread's readback with the other's
dispatch, which is the bubble quantus/miner#4 and #5 target. Measuring it in
the harness first means #4 lands with a number instead of a hypothesis.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5

gitea-actionsbot commented 2026-09-03 10:05:21 +00:00

Copy Link

quantus-bench bad02fbc33 on benjy.hanzalova.internal

worker	median MH/s	spread	windows
0	144.13	0.6%	144.7, 144.1, 144.2, 143.9, 144.1

total median: 144.13 MH/s (batch 1000000, 5 x 30s, engine gpu-wgpu)

• gpu 0 NVIDIA GeForce RTX 4090: driver 580.173.02, limit 250 W, draw 250 W, sm 2190 MHz, 54 C
• parity: OK (25/25 jobs found solutions, all verified against CPU)

## quantus-bench bad02fbc33720e6cb6114ddfb93b2b78be19b0d3 on benjy.hanzalova.internal | worker | median MH/s | spread | windows | | --- | --- | --- | --- | | 0 | 144.13 | 0.6% | 144.7, 144.1, 144.2, 143.9, 144.1 | total median: **144.13 MH/s** (batch 1000000, 5 x 30s, engine gpu-wgpu) - gpu 0 NVIDIA GeForce RTX 4090: driver 580.173.02, limit 250 W, draw 250 W, sm 2190 MHz, 54 C - parity: OK (25/25 jobs found solutions, all verified against CPU)

grenade added 1 commit 2026-09-03 10:06:51 +00:00

bench: serialise measurements per host with a flock, per-run file names ... ca6642960c

Gitea's concurrency group did not serialise two runs on benjy: the second
run's scp failed with ETXTBSY on the binary the first was executing (safe,
it happened before the miner was touched, but the run was lost). The pause,
lock, measure and resume sequence now lives in bench-on-host.sh, executed
over one ssh call under flock on the host, with per-run binary and record
names so staging never collides.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5

gitea-actionsbot commented 2026-09-03 10:12:19 +00:00

Copy Link

quantus-bench ca6642960c on benjy.hanzalova.internal

worker	median MH/s	spread	windows
0	144.39	0.2%	144.3, 144.4, 144.4, 144.5, 144.1

total median: 144.39 MH/s (batch 1000000, 5 x 30s, engine gpu-wgpu)

• gpu 0 NVIDIA GeForce RTX 4090: driver 580.173.02, limit 250 W, draw 250 W, sm 1200 MHz, 57 C
• parity: OK (25/25 jobs found solutions, all verified against CPU)
  --- resume miner ---
  active

## quantus-bench ca6642960c40bd936d8a8f0d9697cd6e9ba46c40 on benjy.hanzalova.internal | worker | median MH/s | spread | windows | | --- | --- | --- | --- | | 0 | 144.39 | 0.2% | 144.3, 144.4, 144.4, 144.5, 144.1 | total median: **144.39 MH/s** (batch 1000000, 5 x 30s, engine gpu-wgpu) - gpu 0 NVIDIA GeForce RTX 4090: driver 580.173.02, limit 250 W, draw 250 W, sm 1200 MHz, 57 C - parity: OK (25/25 jobs found solutions, all verified against CPU) --- resume miner --- active

grenade referenced this pull request 2026-09-03 10:15:14 +00:00

Benchmark and parity harness: the gate every perf change and every origin merge passes #2

grenade merged commit ad846af984 into main 2026-09-03 10:19:34 +00:00

grenade deleted branch bench/harness 2026-09-03 10:19:35 +00:00

grenade referenced this issue from a commit 2026-09-03 10:19:35 +00:00

Merge pull request 'bench: quantus-bench harness and a workflow that measures on a mining host' (#11) from bench/harness into main

grenade referenced this pull request 2026-09-03 15:22:51 +00:00

Benchmark and parity harness: the gate every perf change and every origin merge passes #2

grenade referenced this pull request 2026-09-03 15:23:01 +00:00

Allow more than one worker thread per GPU device #4

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

needs-benchmark

Gain is estimated, not measured; needs a run on a 5090 host

needs-chain

May require a matching change in quantus/chain

origin-coupling

Touches how the lair fork tracks Quantus-Network/quantus-miner

perf

Mining throughput workstream

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

grenade

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: blackbeard/miner#11