Continuous performance telemetry: attribute production hashrate to builds #9
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of #1. Companion to #2: the harness measures a change before merge on one card under controlled conditions; this measures every deployed build after merge on every card under real conditions, continuously, so that an improvement or regression is attributable to a commit without anyone running anything. The two numbers should agree, and when they do not, that is itself a finding (thermal, driver, node latency, contention with inference).
What exists
miner_hashes_total,miner_hash_rate(aggregate per host),nvidia_gpu_*per card from the node-exporter textfile in lair/quantus, and a Grafana dashboard with an aggregate hashrate panel. Nothing ties a hashrate to a build, nothing is per device, and the miner's--versionis a semver that does not change between commits.Miner-side (this repo)
miner_build_infogauge, value 1, labelsversion,commit,cuda_arch,kernel_set. Populated from the build-time env the CI issue adds. This is the join key for everything below. A change in the label set is the deploy marker; no Grafana annotation API and no credentials needed, a Prometheus query renders it.miner_device_hashes_total{device, gpu_name, kernel}.kernelis the id the engine logs (u64,cuda, ...), so a host that silently fell back from CUDA to wgpu shows up as a label change, not as a mystery 40% drop.miner_stale_hashes_total{device}: hashes reported for a job after a newer job was issued (the worker loop already knows; it logs "interrupted by new block").miner_seal_latency_secondshistogram: found on device toJobResultsent.miner_job_idle_seconds_total: time between sending a result and receiving the next job, which is node-attributable and feeds quantus/chain#1.All new metric names, origin's metrics untouched, registration behind a
// lair:marker incrates/metrics.Monitoring-side (lair/quantus, which keeps the fleet's Prometheus and Grafana)
Recording rules, per
(instance, device):quantus:device_hashrate:rate5m = rate(miner_device_hashes_total[5m])quantus:device_efficiency:rate5m = quantus:device_hashrate:rate5m / on(instance, device) nvidia_gpu_power_watts(MH/s per watt; the honest number when a power limit changes between builds)quantus:device_hashrate_by_commit = quantus:device_hashrate:rate5m * on(instance) group_left(commit, cuda_arch) miner_build_infoquantus:stale_fraction:rate1h = rate(miner_stale_hashes_total[1h]) / rate(miner_device_hashes_total[1h])Dashboard row "Performance by build": MH/s and MH/s per watt per device, series split by
commit, with build changes as annotations fromchanges(miner_build_info[5m]) > 0. A table of median MH/s per(host, device, commit)over each commit's lifetime is the one that answers "did that change help".Regression rule
For each
(instance, device), compare the median hashrate over the first 6 hours of the current commit (after a 15-minute warm-up) against the same window of the previous commit at the same power limit. Below 0.97x is a regression, above 1.03x an improvement, anything in between is noise. Implemented as a recording rule plus a dashboard stat; wired to alerting only if the fleet has an Alertmanager path already (lair/quantus knows; do not build one for this).Power limit is a label on the comparison, not something to normalise away: a build measured at 400 W is not comparable to one at 600 W, and
nvidia_gpu_power_limit_wattsis already exported.What this does not replace
Origin coupling
Metric additions only. If origin later adds its own build-info or per-device metrics, adopt theirs and drop ours in that merge.
Landed (2026-09-03)
miner_build_infocarries the deployed commit.rule_files, "Performance by build" dashboard row with a deploy annotation fromminer_build_info,nvidia_gpu_event_reason_activethrottle reasons) and #8 (host label fix for the GPU join). Rule groupquantus-minerloaded and producing values.First live numbers worth acting on
Batch phase split, benjy 4090 at 250 W, 1M batches, u64 kernel (
miner_gpu_batch_seconds, 21,432 batches):So the host-side bubble that #4 and #5 target is 1.9% on the 4090, not the double-digit figure the audit estimated. The estimate used 2.7 ms batches, which is a 5090 number; on the 4090 a 1M batch is 6.8 ms and the fixed overhead is 130 µs. On a 5090 at 368 MH/s the same overhead would be about 5%. Revised expectations: #4 and #5 together are worth at most 2% on the 4090 and about 5% on the 5090 at the default batch; #6 (bigger batches) captures most of that on its own by dividing the overhead. None of the three is a large lever; #3 remains the only one.
Stale work as first implemented was wrong (74% flagged); #15 records only the batch in flight at cancellation. Expect about one batch per job switch, so roughly 7 ms per job on the 4090 at 1M.
Throttle reasons: both mining cards report
sw_power_capactive continuously, as expected under their limits. Thermal and brake reasons are 0.Job cadence on testnet: 11 jobs in the first ~4 minutes on benjy, one result submitted. Idle-after-result was 0.26 s for that one. Too early to read more into.
Closing: miner side (#14, #15) and monitoring side (lair/quantus#7, #8) are live, the "Performance by build" row showed today's three deploys as distinct series with annotations, and the per-build comparison answered its first real question (the rival miner's ramp at 13:30 UTC was theirs, not our regression: our 497 MH/s was flat and our observed share tracked expected within noise). The regression rule remains a dashboard comparison rather than an alert, since the fleet has no Alertmanager path to hook; that can be its own issue if one appears.