Part of #1. Companion to #2: the harness measures a change before merge on one card under controlled conditions; this measures every deployed build after merge on every card under real conditions, continuously, so that an improvement or regression is attributable to a commit without anyone running anything. The two numbers should agree, and when they do not, that is itself a finding (thermal, driver, node latency, contention with inference).
What exists
miner_hashes_total, miner_hash_rate (aggregate per host), nvidia_gpu_* per card from the node-exporter textfile in lair/quantus, and a Grafana dashboard with an aggregate hashrate panel. Nothing ties a hashrate to a build, nothing is per device, and the miner's --version is a semver that does not change between commits.
Miner-side (this repo)
miner_build_info gauge, value 1, labels version, commit, cuda_arch, kernel_set. Populated from the build-time env the CI issue adds. This is the join key for everything below. A change in the label set is the deploy marker; no Grafana annotation API and no credentials needed, a Prometheus query renders it.
Per-device counters from #2: miner_device_hashes_total{device, gpu_name, kernel}. kernel is the id the engine logs (u64, cuda, ...), so a host that silently fell back from CUDA to wgpu shows up as a label change, not as a mystery 40% drop.
Per-job efficiency counters, since batch-size and pipelining work (#4, #5, #6) trade hashrate against these and nobody can tune a trade-off with only one side measured:
miner_stale_hashes_total{device}: hashes reported for a job after a newer job was issued (the worker loop already knows; it logs "interrupted by new block").
miner_seal_latency_seconds histogram: found on device to JobResult sent.
miner_job_idle_seconds_total: time between sending a result and receiving the next job, which is node-attributable and feeds quantus/chain#1.
All new metric names, origin's metrics untouched, registration behind a // lair: marker in crates/metrics.
Monitoring-side (lair/quantus, which keeps the fleet's Prometheus and Grafana)
quantus:device_efficiency:rate5m = quantus:device_hashrate:rate5m / on(instance, device) nvidia_gpu_power_watts (MH/s per watt; the honest number when a power limit changes between builds)
Dashboard row "Performance by build": MH/s and MH/s per watt per device, series split by commit, with build changes as annotations from changes(miner_build_info[5m]) > 0. A table of median MH/s per (host, device, commit) over each commit's lifetime is the one that answers "did that change help".
Regression rule
For each (instance, device), compare the median hashrate over the first 6 hours of the current commit (after a 15-minute warm-up) against the same window of the previous commit at the same power limit. Below 0.97x is a regression, above 1.03x an improvement, anything in between is noise. Implemented as a recording rule plus a dashboard stat; wired to alerting only if the fleet has an Alertmanager path already (lair/quantus knows; do not build one for this).
Power limit is a label on the comparison, not something to normalise away: a build measured at 400 W is not comparable to one at 600 W, and nvidia_gpu_power_limit_watts is already exported.
What this does not replace
The harness in #2. Production numbers are noisy: two miners on one node, inference on beast, ambient temperature. The harness is where a 3% kernel change is decided; this is where a 3% regression that slipped through is caught within a day.
lair/quantus#1 (authored-block share). That is chain-level efficiency and needs the node metrics; this issue stops at the miner process.
Origin coupling
Metric additions only. If origin later adds its own build-info or per-device metrics, adopt theirs and drop ours in that merge.
Part of #1. Companion to #2: the harness measures a change **before** merge on one card under controlled conditions; this measures every deployed build **after** merge on every card under real conditions, continuously, so that an improvement or regression is attributable to a commit without anyone running anything. The two numbers should agree, and when they do not, that is itself a finding (thermal, driver, node latency, contention with inference).
## What exists
`miner_hashes_total`, `miner_hash_rate` (aggregate per host), `nvidia_gpu_*` per card from the node-exporter textfile in lair/quantus, and a Grafana dashboard with an aggregate hashrate panel. Nothing ties a hashrate to a build, nothing is per device, and the miner's `--version` is a semver that does not change between commits.
## Miner-side (this repo)
1. **`miner_build_info`** gauge, value 1, labels `version`, `commit`, `cuda_arch`, `kernel_set`. Populated from the build-time env the CI issue adds. This is the join key for everything below. A change in the label set is the deploy marker; no Grafana annotation API and no credentials needed, a Prometheus query renders it.
2. **Per-device counters** from #2: `miner_device_hashes_total{device, gpu_name, kernel}`. `kernel` is the id the engine logs (`u64`, `cuda`, ...), so a host that silently fell back from CUDA to wgpu shows up as a label change, not as a mystery 40% drop.
3. **Per-job efficiency counters**, since batch-size and pipelining work (#4, #5, #6) trade hashrate against these and nobody can tune a trade-off with only one side measured:
- `miner_stale_hashes_total{device}`: hashes reported for a job after a newer job was issued (the worker loop already knows; it logs "interrupted by new block").
- `miner_seal_latency_seconds` histogram: found on device to `JobResult` sent.
- `miner_job_idle_seconds_total`: time between sending a result and receiving the next job, which is node-attributable and feeds quantus/chain#1.
All new metric names, origin's metrics untouched, registration behind a `// lair:` marker in `crates/metrics`.
## Monitoring-side (lair/quantus, which keeps the fleet's Prometheus and Grafana)
Recording rules, per `(instance, device)`:
- `quantus:device_hashrate:rate5m = rate(miner_device_hashes_total[5m])`
- `quantus:device_efficiency:rate5m = quantus:device_hashrate:rate5m / on(instance, device) nvidia_gpu_power_watts` (MH/s per watt; the honest number when a power limit changes between builds)
- `quantus:device_hashrate_by_commit = quantus:device_hashrate:rate5m * on(instance) group_left(commit, cuda_arch) miner_build_info`
- `quantus:stale_fraction:rate1h = rate(miner_stale_hashes_total[1h]) / rate(miner_device_hashes_total[1h])`
Dashboard row "Performance by build": MH/s and MH/s per watt per device, series split by `commit`, with build changes as annotations from `changes(miner_build_info[5m]) > 0`. A table of median MH/s per `(host, device, commit)` over each commit's lifetime is the one that answers "did that change help".
## Regression rule
For each `(instance, device)`, compare the median hashrate over the first 6 hours of the current commit (after a 15-minute warm-up) against the same window of the previous commit at the same power limit. Below 0.97x is a regression, above 1.03x an improvement, anything in between is noise. Implemented as a recording rule plus a dashboard stat; wired to alerting only if the fleet has an Alertmanager path already (lair/quantus knows; do not build one for this).
Power limit is a label on the comparison, not something to normalise away: a build measured at 400 W is not comparable to one at 600 W, and `nvidia_gpu_power_limit_watts` is already exported.
## What this does not replace
- The harness in #2. Production numbers are noisy: two miners on one node, inference on beast, ambient temperature. The harness is where a 3% kernel change is decided; this is where a 3% regression that slipped through is caught within a day.
- lair/quantus#1 (authored-block share). That is chain-level efficiency and needs the node metrics; this issue stops at the miner process.
## Origin coupling
Metric additions only. If origin later adds its own build-info or per-device metrics, adopt theirs and drop ours in that merge.
Miner side: #14 (all series listed there), correction #15 (stale semantics, see below). Deployed to benjy and quadbrat by actions/runs/23; validate now asserts miner_build_info carries the deployed commit.
Monitoring side: lair/quantus#7 (recording rules under rule_files, "Performance by build" dashboard row with a deploy annotation from miner_build_info, nvidia_gpu_event_reason_active throttle reasons) and #8 (host label fix for the GPU join). Rule group quantus-miner loaded and producing values.
Instrumentation cost: the bench on #14 measured 143.89 MH/s against the 144.1 to 144.4 baseline, inside spread.
So the host-side bubble that #4 and #5 target is 1.9% on the 4090, not the double-digit figure the audit estimated. The estimate used 2.7 ms batches, which is a 5090 number; on the 4090 a 1M batch is 6.8 ms and the fixed overhead is 130 µs. On a 5090 at 368 MH/s the same overhead would be about 5%. Revised expectations: #4 and #5 together are worth at most 2% on the 4090 and about 5% on the 5090 at the default batch; #6 (bigger batches) captures most of that on its own by dividing the overhead. None of the three is a large lever; #3 remains the only one.
Stale work as first implemented was wrong (74% flagged); #15 records only the batch in flight at cancellation. Expect about one batch per job switch, so roughly 7 ms per job on the 4090 at 1M.
Throttle reasons: both mining cards report sw_power_cap active continuously, as expected under their limits. Thermal and brake reasons are 0.
Job cadence on testnet: 11 jobs in the first ~4 minutes on benjy, one result submitted. Idle-after-result was 0.26 s for that one. Too early to read more into.
## Landed (2026-09-03)
- Miner side: #14 (all series listed there), correction #15 (stale semantics, see below). Deployed to benjy and quadbrat by actions/runs/23; validate now asserts `miner_build_info` carries the deployed commit.
- Monitoring side: lair/quantus#7 (recording rules under `rule_files`, "Performance by build" dashboard row with a deploy annotation from `miner_build_info`, `nvidia_gpu_event_reason_active` throttle reasons) and #8 (host label fix for the GPU join). Rule group `quantus-miner` loaded and producing values.
- Instrumentation cost: the bench on #14 measured 143.89 MH/s against the 144.1 to 144.4 baseline, inside spread.
## First live numbers worth acting on
**Batch phase split, benjy 4090 at 250 W, 1M batches, u64 kernel** (`miner_gpu_batch_seconds`, 21,432 batches):
| phase | total | per batch | share |
| --- | --- | --- | --- |
| gpu | 145.55 s | 6.79 ms | 98.1% |
| host | 2.78 s | 0.13 ms | 1.9% |
So the host-side bubble that #4 and #5 target is **1.9% on the 4090**, not the double-digit figure the audit estimated. The estimate used 2.7 ms batches, which is a 5090 number; on the 4090 a 1M batch is 6.8 ms and the fixed overhead is 130 µs. On a 5090 at 368 MH/s the same overhead would be about 5%. Revised expectations: #4 and #5 together are worth at most 2% on the 4090 and about 5% on the 5090 at the default batch; #6 (bigger batches) captures most of that on its own by dividing the overhead. None of the three is a large lever; #3 remains the only one.
**Stale work** as first implemented was wrong (74% flagged); #15 records only the batch in flight at cancellation. Expect about one batch per job switch, so roughly 7 ms per job on the 4090 at 1M.
**Throttle reasons**: both mining cards report `sw_power_cap` active continuously, as expected under their limits. Thermal and brake reasons are 0.
**Job cadence on testnet**: 11 jobs in the first ~4 minutes on benjy, one result submitted. Idle-after-result was 0.26 s for that one. Too early to read more into.
Closing: miner side (#14, #15) and monitoring side (lair/quantus#7, #8) are live, the "Performance by build" row showed today's three deploys as distinct series with annotations, and the per-build comparison answered its first real question (the rival miner's ramp at 13:30 UTC was theirs, not our regression: our 497 MH/s was flat and our observed share tracked expected within noise). The regression rule remains a dashboard comparison rather than an alert, since the fleet has no Alertmanager path to hook; that can be its own issue if one appears.
Closing: miner side (#14, #15) and monitoring side (lair/quantus#7, #8) are live, the "Performance by build" row showed today's three deploys as distinct series with annotations, and the per-build comparison answered its first real question (the rival miner's ramp at 13:30 UTC was theirs, not our regression: our 497 MH/s was flat and our observed share tracked expected within noise). The regression rule remains a dashboard comparison rather than an alert, since the fleet has no Alertmanager path to hook; that can be its own issue if one appears.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Part of #1. Companion to #2: the harness measures a change before merge on one card under controlled conditions; this measures every deployed build after merge on every card under real conditions, continuously, so that an improvement or regression is attributable to a commit without anyone running anything. The two numbers should agree, and when they do not, that is itself a finding (thermal, driver, node latency, contention with inference).
What exists
miner_hashes_total,miner_hash_rate(aggregate per host),nvidia_gpu_*per card from the node-exporter textfile in lair/quantus, and a Grafana dashboard with an aggregate hashrate panel. Nothing ties a hashrate to a build, nothing is per device, and the miner's--versionis a semver that does not change between commits.Miner-side (this repo)
miner_build_infogauge, value 1, labelsversion,commit,cuda_arch,kernel_set. Populated from the build-time env the CI issue adds. This is the join key for everything below. A change in the label set is the deploy marker; no Grafana annotation API and no credentials needed, a Prometheus query renders it.miner_device_hashes_total{device, gpu_name, kernel}.kernelis the id the engine logs (u64,cuda, ...), so a host that silently fell back from CUDA to wgpu shows up as a label change, not as a mystery 40% drop.miner_stale_hashes_total{device}: hashes reported for a job after a newer job was issued (the worker loop already knows; it logs "interrupted by new block").miner_seal_latency_secondshistogram: found on device toJobResultsent.miner_job_idle_seconds_total: time between sending a result and receiving the next job, which is node-attributable and feeds quantus/chain#1.All new metric names, origin's metrics untouched, registration behind a
// lair:marker incrates/metrics.Monitoring-side (lair/quantus, which keeps the fleet's Prometheus and Grafana)
Recording rules, per
(instance, device):quantus:device_hashrate:rate5m = rate(miner_device_hashes_total[5m])quantus:device_efficiency:rate5m = quantus:device_hashrate:rate5m / on(instance, device) nvidia_gpu_power_watts(MH/s per watt; the honest number when a power limit changes between builds)quantus:device_hashrate_by_commit = quantus:device_hashrate:rate5m * on(instance) group_left(commit, cuda_arch) miner_build_infoquantus:stale_fraction:rate1h = rate(miner_stale_hashes_total[1h]) / rate(miner_device_hashes_total[1h])Dashboard row "Performance by build": MH/s and MH/s per watt per device, series split by
commit, with build changes as annotations fromchanges(miner_build_info[5m]) > 0. A table of median MH/s per(host, device, commit)over each commit's lifetime is the one that answers "did that change help".Regression rule
For each
(instance, device), compare the median hashrate over the first 6 hours of the current commit (after a 15-minute warm-up) against the same window of the previous commit at the same power limit. Below 0.97x is a regression, above 1.03x an improvement, anything in between is noise. Implemented as a recording rule plus a dashboard stat; wired to alerting only if the fleet has an Alertmanager path already (lair/quantus knows; do not build one for this).Power limit is a label on the comparison, not something to normalise away: a build measured at 400 W is not comparable to one at 600 W, and
nvidia_gpu_power_limit_wattsis already exported.What this does not replace
Origin coupling
Metric additions only. If origin later adds its own build-info or per-device metrics, adopt theirs and drop ours in that merge.
Landed (2026-09-03)
miner_build_infocarries the deployed commit.rule_files, "Performance by build" dashboard row with a deploy annotation fromminer_build_info,nvidia_gpu_event_reason_activethrottle reasons) and #8 (host label fix for the GPU join). Rule groupquantus-minerloaded and producing values.First live numbers worth acting on
Batch phase split, benjy 4090 at 250 W, 1M batches, u64 kernel (
miner_gpu_batch_seconds, 21,432 batches):So the host-side bubble that #4 and #5 target is 1.9% on the 4090, not the double-digit figure the audit estimated. The estimate used 2.7 ms batches, which is a 5090 number; on the 4090 a 1M batch is 6.8 ms and the fixed overhead is 130 µs. On a 5090 at 368 MH/s the same overhead would be about 5%. Revised expectations: #4 and #5 together are worth at most 2% on the 4090 and about 5% on the 5090 at the default batch; #6 (bigger batches) captures most of that on its own by dividing the overhead. None of the three is a large lever; #3 remains the only one.
Stale work as first implemented was wrong (74% flagged); #15 records only the batch in flight at cancellation. Expect about one batch per job switch, so roughly 7 ms per job on the 4090 at 1M.
Throttle reasons: both mining cards report
sw_power_capactive continuously, as expected under their limits. Thermal and brake reasons are 0.Job cadence on testnet: 11 jobs in the first ~4 minutes on benjy, one result submitted. Idle-after-result was 0.26 s for that one. Too early to read more into.
Closing: miner side (#14, #15) and monitoring side (lair/quantus#7, #8) are live, the "Performance by build" row showed today's three deploys as distinct series with annotations, and the per-build comparison answered its first real question (the rival miner's ramp at 13:30 UTC was theirs, not our regression: our 497 MH/s was flat and our observed share tracked expected within noise). The regression rule remains a dashboard comparison rather than an alert, since the fleet has no Alertmanager path to hook; that can be its own issue if one appears.