metrics: performance by build — recording rules, dashboard row, throttle reasons #7

Merged
grenade merged 1 commits from metrics/performance-by-build into main 2026-09-03 11:16:28 +00:00

1 Commits

Author SHA1 Message Date
Rob Thijssen
aa6bd855f4 metrics: performance by build — recording rules, dashboard row, throttle reasons
Monitoring side of quantus/miner#9. The miner now exports build identity,
per-device and per-job series (quantus/miner#14); this makes them readable.

- asset/prometheus/quantus-rules.yml: recording rules joining the miner's
  series with the GPU series on a derived `host` and device index —
  hashrate by device and commit, hashrate per watt, batch time off the GPU,
  stale work, seal latency p95, idle waiting for the node.
- script/infra-setup.sh (metrics role): prometheus.yml gains `rule_files:`
  for /etc/prometheus/rules.d, guarded like scrape_config_files; the dir;
  a sudoers grant for the rules file.
- deploy.yaml deploy-metrics: pushes the rules (reload is already gated on
  content change); validate asserts the quantus-miner rule group loaded.
- asset/grafana/quantus.json: "Performance by build" row, eight panels, and
  a deploy annotation from changes in miner_build_info. No credentials
  involved; Prometheus is the annotation source.
- asset/gpu/nvidia-textfile.sh (gpu role, operator-shipped): five
  nvidia_gpu_event_reason_active{reason=...} gauges from nvidia-smi's clock
  event reasons, so a regression that coincides with thermal or power-brake
  capping can be told from a code regression.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
2026-09-03 14:13:15 +03:00