598aea28f8
All checks were successful
deploy / fetch (push) Successful in 16s
deploy / deploy-node (bob.hanzalova.internal, 0x134e73f06fa9bdb1dbfa909e149c563f5860ceb71a0e7307918f7033970edf59, benjy.hanzalova.internal quadbrat.hanzalova.internal, --public-addr /dns4/nh.thgttg.com/tcp/30333, --unsafe-rpc-external --rpc-methods safe --rpc-c… (push) Successful in 33s
deploy / deploy-miner (1, benjy.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 41s
deploy / deploy-miner (1, quadbrat.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 40s
deploy / deploy-metrics (push) Successful in 9s
node-exporter already runs on every host, is already scraped (job 'node', targets from cichlid's inventory, labelled by host), and already has the textfile collector enabled with a spool at /var/lib/prometheus/node-exporter. So a textfile drop needs no new exporter, no port allocation, no firewalld service and no scrape config -- which is why it is one-time provisioning rather than a deploy job. DCGM was the alternative and is the wrong tool: it targets datacenter cards and its GeForce support is partial. nvidia-smi reports everything wanted. The script writes to a temp file and renames atomically, because the collector reads whole files and a half-written one surfaces as node_textfile_scrape_error. When nvidia-smi fails it emits an explicit nvidia_gpu_scrape_ok 0 rather than leaving a stale file, so a broken driver reads as broken instead of the graphs quietly flatlining. Verified on the dual-GPU host: both cards reporting temperature, power, utilisation, memory, clocks and fan through node-exporter with node_textfile_scrape_error 0. Fleet-general rather than quantus-specific -- it lives here because there is no monitoring repo yet, and the dashboard says so.
13 lines
296 B
SYSTEMD
13 lines
296 B
SYSTEMD
[Unit]
|
|
Description=Refresh NVIDIA GPU metrics for node-exporter
|
|
|
|
[Timer]
|
|
OnBootSec=30s
|
|
# Matched to Prometheus' 15s scrape interval: slower would show stale values,
|
|
# faster would just burn nvidia-smi invocations nobody reads.
|
|
OnUnitActiveSec=15s
|
|
AccuracySec=1s
|
|
|
|
[Install]
|
|
WantedBy=timers.target
|