metrics: performance by build — recording rules, dashboard row, throttle reasons #7

grenade · 2026-09-03T11:13:51Z

grenade commented

2026-09-03 11:13:51 +00:00

Monitoring side of quantus/miner#9; the miner-side series arrive with quantus/miner#14. Either can merge first: rules over absent series simply produce nothing until the miners are redeployed.

What changed

asset/prometheus/quantus-rules.yml: eight recording rules (promtool check rules: SUCCESS). They join the miner's series (instance host:9900) with the GPU series from node-exporter's textfile on a derived host label and device = gpu index: hashrate by device and commit, hashrate per watt, batch time off the GPU, stale work, seal latency p95, idle waiting for the node.
script/infra-setup.sh metrics role: prometheus.yml gains rule_files: for /etc/prometheus/rules.d, guarded and backed up exactly like the scrape_config_files include; the directory; a sudoers grant for the rules file. Already applied to the metrics host from this branch; Prometheus reloaded. The gpu role was re-run too, so all three GPU hosts ship the new exporter script.
deploy.yaml deploy-metrics: pushes the rules file (reload already gated on content change) and validate asserts the quantus-miner rule group is loaded, with the same "is it wired into prometheus.yml" hint as the scrape check.
asset/grafana/quantus.json: a "Performance by build" row of eight panels, and a deploy annotation driven by changes in miner_build_info, so no Grafana credentials are needed.
asset/gpu/nvidia-textfile.sh: five nvidia_gpu_event_reason_active{reason=...} gauges from nvidia-smi's clock event reasons (sw_power_cap, hw_slowdown, sw_thermal_slowdown, hw_thermal_slowdown, hw_power_brake_slowdown). Tested on benjy: power cap active, the rest 0, as expected at a 250 W limit.

Known limitation

The device join assumes the miner's engine index equals nvidia-smi's index. True on the single-card hosts; on beast it depends on Vulkan's enumeration order matching, which is likely but not guaranteed. Noted in the rules file.

Merging triggers this repo's deploy (workflow and assets changed): node untouched, metrics pushed and reloaded, validate checks targets and the rule group.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5

Monitoring side of quantus/miner#9; the miner-side series arrive with quantus/miner#14. Either can merge first: rules over absent series simply produce nothing until the miners are redeployed. ## What changed - **`asset/prometheus/quantus-rules.yml`**: eight recording rules (`promtool check rules`: SUCCESS). They join the miner's series (instance `host:9900`) with the GPU series from node-exporter's textfile on a derived `host` label and `device` = `gpu` index: hashrate by device and commit, hashrate per watt, batch time off the GPU, stale work, seal latency p95, idle waiting for the node. - **`script/infra-setup.sh`** metrics role: `prometheus.yml` gains `rule_files:` for `/etc/prometheus/rules.d`, guarded and backed up exactly like the `scrape_config_files` include; the directory; a sudoers grant for the rules file. **Already applied** to the metrics host from this branch; Prometheus reloaded. The gpu role was re-run too, so all three GPU hosts ship the new exporter script. - **`deploy.yaml`** `deploy-metrics`: pushes the rules file (reload already gated on content change) and validate asserts the `quantus-miner` rule group is loaded, with the same "is it wired into prometheus.yml" hint as the scrape check. - **`asset/grafana/quantus.json`**: a "Performance by build" row of eight panels, and a deploy annotation driven by changes in `miner_build_info`, so no Grafana credentials are needed. - **`asset/gpu/nvidia-textfile.sh`**: five `nvidia_gpu_event_reason_active{reason=...}` gauges from `nvidia-smi`'s clock event reasons (sw_power_cap, hw_slowdown, sw_thermal_slowdown, hw_thermal_slowdown, hw_power_brake_slowdown). Tested on benjy: power cap active, the rest 0, as expected at a 250 W limit. ## Known limitation The device join assumes the miner's engine index equals nvidia-smi's index. True on the single-card hosts; on beast it depends on Vulkan's enumeration order matching, which is likely but not guaranteed. Noted in the rules file. Merging triggers this repo's deploy (workflow and assets changed): node untouched, metrics pushed and reloaded, validate checks targets and the rule group. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5

grenade added 1 commit 2026-09-03 11:13:52 +00:00

metrics: performance by build — recording rules, dashboard row, throttle reasons aa6bd855f4

Monitoring side of quantus/miner#9. The miner now exports build identity,
per-device and per-job series (quantus/miner#14); this makes them readable.

- asset/prometheus/quantus-rules.yml: recording rules joining the miner's
  series with the GPU series on a derived `host` and device index —
  hashrate by device and commit, hashrate per watt, batch time off the GPU,
  stale work, seal latency p95, idle waiting for the node.
- script/infra-setup.sh (metrics role): prometheus.yml gains `rule_files:`
  for /etc/prometheus/rules.d, guarded like scrape_config_files; the dir;
  a sudoers grant for the rules file.
- deploy.yaml deploy-metrics: pushes the rules (reload is already gated on
  content change); validate asserts the quantus-miner rule group loaded.
- asset/grafana/quantus.json: "Performance by build" row, eight panels, and
  a deploy annotation from changes in miner_build_info. No credentials
  involved; Prometheus is the annotation source.
- asset/gpu/nvidia-textfile.sh (gpu role, operator-shipped): five
  nvidia_gpu_event_reason_active{reason=...} gauges from nvidia-smi's clock
  event reasons, so a regression that coincides with thermal or power-brake
  capping can be told from a code regression.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5