Commit Graph
3 Commits
Author SHA1 Message Date
Rob ThijssenandClaude Fable 5.1 aa6bd855f4 metrics: performance by build — recording rules, dashboard row, throttle reasons
Monitoring side of quantus/miner#9. The miner now exports build identity,
per-device and per-job series (quantus/miner#14); this makes them readable.

- asset/prometheus/quantus-rules.yml: recording rules joining the miner's
  series with the GPU series on a derived `host` and device index —
  hashrate by device and commit, hashrate per watt, batch time off the GPU,
  stale work, seal latency p95, idle waiting for the node.
- script/infra-setup.sh (metrics role): prometheus.yml gains `rule_files:`
  for /etc/prometheus/rules.d, guarded like scrape_config_files; the dir;
  a sudoers grant for the rules file.
- deploy.yaml deploy-metrics: pushes the rules (reload is already gated on
  content change); validate asserts the quantus-miner rule group loaded.
- asset/grafana/quantus.json: "Performance by build" row, eight panels, and
  a deploy annotation from changes in miner_build_info. No credentials
  involved; Prometheus is the annotation source.
- asset/gpu/nvidia-textfile.sh (gpu role, operator-shipped): five
  nvidia_gpu_event_reason_active{reason=...} gauges from nvidia-smi's clock
  event reasons, so a regression that coincides with thermal or power-brake
  capping can be told from a code regression.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
2026-09-03 14:13:15 +03:00
Rob Thijssen e178bcbdf9 feat: boot-time GPU power limits, clamped to hardware floors
deploy / fetch (push) Successful in 17s
deploy / deploy-node (bob.hanzalova.internal, 0x134e73f06fa9bdb1dbfa909e149c563f5860ceb71a0e7307918f7033970edf59, benjy.hanzalova.internal quadbrat.hanzalova.internal, --public-addr /dns4/nh.thgttg.com/tcp/30333, --unsafe-rpc-external --rpc-methods safe --rpc-c… (push) Successful in 34s
deploy / deploy-miner (1, quadbrat.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 40s
deploy / deploy-miner (1, benjy.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 43s
deploy / deploy-metrics (push) Successful in 8s
nvidia-smi -pl survives neither a reboot nor a driver unload, and both
failures are silent -- the card returns to its default limit and nobody
notices until a connector runs warm. Hence a boot-time unit, which also
enables persistence mode so the limit is not lost when the last GPU
client detaches.

Per-host limits configured as host=watts in infra-setup.sh, clamped to
each card's own min/max. Verified: a 300W request on a 5090 whose floor
is 400W is corrected with an explicit log line rather than failing or
being ignored.

Measured on a 4090 mining Poseidon2:
  450W (stock)  185.5 MH/s  70C  0.413 MH/s/W
  300W          183.2 MH/s  61C  0.611
  250W          180.3 MH/s  58C  0.721

44% less power and 12C cooler for 2.8% less hashrate -- this workload is
register-resident integer math and does not need the top of the clock
curve. It also cuts current through the 12VHPWR connector, which is the
one component on these cards with a documented melting failure mode.

Ordered Before=quantus-miner.service so a GPU is never briefly at its
default limit under full mining load.
2026-09-01 06:42:32 +03:00
Rob Thijssen 598aea28f8 feat: GPU metrics via the node-exporter textfile collector
deploy / fetch (push) Successful in 16s
deploy / deploy-node (bob.hanzalova.internal, 0x134e73f06fa9bdb1dbfa909e149c563f5860ceb71a0e7307918f7033970edf59, benjy.hanzalova.internal quadbrat.hanzalova.internal, --public-addr /dns4/nh.thgttg.com/tcp/30333, --unsafe-rpc-external --rpc-methods safe --rpc-c… (push) Successful in 33s
deploy / deploy-miner (1, benjy.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 41s
deploy / deploy-miner (1, quadbrat.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 40s
deploy / deploy-metrics (push) Successful in 9s
node-exporter already runs on every host, is already scraped (job
'node', targets from cichlid's inventory, labelled by host), and already
has the textfile collector enabled with a spool at
/var/lib/prometheus/node-exporter. So a textfile drop needs no new
exporter, no port allocation, no firewalld service and no scrape config
-- which is why it is one-time provisioning rather than a deploy job.

DCGM was the alternative and is the wrong tool: it targets datacenter
cards and its GeForce support is partial. nvidia-smi reports everything
wanted.

The script writes to a temp file and renames atomically, because the
collector reads whole files and a half-written one surfaces as
node_textfile_scrape_error. When nvidia-smi fails it emits an explicit
nvidia_gpu_scrape_ok 0 rather than leaving a stale file, so a broken
driver reads as broken instead of the graphs quietly flatlining.

Verified on the dual-GPU host: both cards reporting temperature, power,
utilisation, memory, clocks and fan through node-exporter with
node_textfile_scrape_error 0.

Fleet-general rather than quantus-specific -- it lives here because
there is no monitoring repo yet, and the dashboard says so.
2026-09-01 06:10:10 +03:00