Commit Graph
2 Commits
Author SHA1 Message Date
Rob Thijssen e178bcbdf9 feat: boot-time GPU power limits, clamped to hardware floors
deploy / fetch (push) Successful in 17s
deploy / deploy-node (bob.hanzalova.internal, 0x134e73f06fa9bdb1dbfa909e149c563f5860ceb71a0e7307918f7033970edf59, benjy.hanzalova.internal quadbrat.hanzalova.internal, --public-addr /dns4/nh.thgttg.com/tcp/30333, --unsafe-rpc-external --rpc-methods safe --rpc-c… (push) Successful in 34s
deploy / deploy-miner (1, quadbrat.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 40s
deploy / deploy-miner (1, benjy.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 43s
deploy / deploy-metrics (push) Successful in 8s
nvidia-smi -pl survives neither a reboot nor a driver unload, and both
failures are silent -- the card returns to its default limit and nobody
notices until a connector runs warm. Hence a boot-time unit, which also
enables persistence mode so the limit is not lost when the last GPU
client detaches.

Per-host limits configured as host=watts in infra-setup.sh, clamped to
each card's own min/max. Verified: a 300W request on a 5090 whose floor
is 400W is corrected with an explicit log line rather than failing or
being ignored.

Measured on a 4090 mining Poseidon2:
  450W (stock)  185.5 MH/s  70C  0.413 MH/s/W
  300W          183.2 MH/s  61C  0.611
  250W          180.3 MH/s  58C  0.721

44% less power and 12C cooler for 2.8% less hashrate -- this workload is
register-resident integer math and does not need the top of the clock
curve. It also cuts current through the 12VHPWR connector, which is the
one component on these cards with a documented melting failure mode.

Ordered Before=quantus-miner.service so a GPU is never briefly at its
default limit under full mining load.
2026-09-01 06:42:32 +03:00
Rob Thijssen 598aea28f8 feat: GPU metrics via the node-exporter textfile collector
deploy / fetch (push) Successful in 16s
deploy / deploy-node (bob.hanzalova.internal, 0x134e73f06fa9bdb1dbfa909e149c563f5860ceb71a0e7307918f7033970edf59, benjy.hanzalova.internal quadbrat.hanzalova.internal, --public-addr /dns4/nh.thgttg.com/tcp/30333, --unsafe-rpc-external --rpc-methods safe --rpc-c… (push) Successful in 33s
deploy / deploy-miner (1, benjy.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 41s
deploy / deploy-miner (1, quadbrat.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 40s
deploy / deploy-metrics (push) Successful in 9s
node-exporter already runs on every host, is already scraped (job
'node', targets from cichlid's inventory, labelled by host), and already
has the textfile collector enabled with a spool at
/var/lib/prometheus/node-exporter. So a textfile drop needs no new
exporter, no port allocation, no firewalld service and no scrape config
-- which is why it is one-time provisioning rather than a deploy job.

DCGM was the alternative and is the wrong tool: it targets datacenter
cards and its GeForce support is partial. nvidia-smi reports everything
wanted.

The script writes to a temp file and renames atomically, because the
collector reads whole files and a half-written one surfaces as
node_textfile_scrape_error. When nvidia-smi fails it emits an explicit
nvidia_gpu_scrape_ok 0 rather than leaving a stale file, so a broken
driver reads as broken instead of the graphs quietly flatlining.

Verified on the dual-GPU host: both cards reporting temperature, power,
utilisation, memory, clocks and fan through node-exporter with
node_textfile_scrape_error 0.

Fleet-general rather than quantus-specific -- it lives here because
there is no monitoring repo yet, and the dashboard says so.
2026-09-01 06:10:10 +03:00