Rob Thijssen
598aea28f8
deploy / fetch (push) Successful in 16s
deploy / deploy-node (bob.hanzalova.internal, 0x134e73f06fa9bdb1dbfa909e149c563f5860ceb71a0e7307918f7033970edf59, benjy.hanzalova.internal quadbrat.hanzalova.internal, --public-addr /dns4/nh.thgttg.com/tcp/30333, --unsafe-rpc-external --rpc-methods safe --rpc-c… (push) Successful in 33s
deploy / deploy-miner (1, benjy.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 41s
deploy / deploy-miner (1, quadbrat.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 40s
deploy / deploy-metrics (push) Successful in 9s
node-exporter already runs on every host, is already scraped (job 'node', targets from cichlid's inventory, labelled by host), and already has the textfile collector enabled with a spool at /var/lib/prometheus/node-exporter. So a textfile drop needs no new exporter, no port allocation, no firewalld service and no scrape config -- which is why it is one-time provisioning rather than a deploy job. DCGM was the alternative and is the wrong tool: it targets datacenter cards and its GeForce support is partial. nvidia-smi reports everything wanted. The script writes to a temp file and renames atomically, because the collector reads whole files and a half-written one surfaces as node_textfile_scrape_error. When nvidia-smi fails it emits an explicit nvidia_gpu_scrape_ok 0 rather than leaving a stale file, so a broken driver reads as broken instead of the graphs quietly flatlining. Verified on the dual-GPU host: both cards reporting temperature, power, utilisation, memory, clocks and fan through node-exporter with node_textfile_scrape_error 0. Fleet-general rather than quantus-specific -- it lives here because there is no monitoring repo yet, and the dashboard says so.
19 lines
564 B
Desktop File
19 lines
564 B
Desktop File
[Unit]
|
|
Description=Write NVIDIA GPU metrics for the node-exporter textfile collector
|
|
Documentation=https://git.lair.cafe/lair/quantus
|
|
ConditionPathExists=/usr/bin/nvidia-smi
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
ExecStart=/usr/local/bin/nvidia-textfile.sh
|
|
# Runs as root only to write into node-exporter's spool; it reads nothing
|
|
# sensitive and holds no credentials.
|
|
NoNewPrivileges=true
|
|
ProtectHome=true
|
|
ProtectKernelTunables=true
|
|
ProtectKernelModules=false
|
|
ProtectControlGroups=true
|
|
RestrictSUIDSGID=true
|
|
LockPersonality=true
|
|
ReadWritePaths=/var/lib/prometheus/node-exporter
|