feat: boot-time GPU power limits, clamped to hardware floors
deploy / fetch (push) Successful in 17s
deploy / deploy-node (bob.hanzalova.internal, 0x134e73f06fa9bdb1dbfa909e149c563f5860ceb71a0e7307918f7033970edf59, benjy.hanzalova.internal quadbrat.hanzalova.internal, --public-addr /dns4/nh.thgttg.com/tcp/30333, --unsafe-rpc-external --rpc-methods safe --rpc-c… (push) Successful in 34s
deploy / deploy-miner (1, quadbrat.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 40s
deploy / deploy-miner (1, benjy.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 43s
deploy / deploy-metrics (push) Successful in 8s

nvidia-smi -pl survives neither a reboot nor a driver unload, and both
failures are silent -- the card returns to its default limit and nobody
notices until a connector runs warm. Hence a boot-time unit, which also
enables persistence mode so the limit is not lost when the last GPU
client detaches.

Per-host limits configured as host=watts in infra-setup.sh, clamped to
each card's own min/max. Verified: a 300W request on a 5090 whose floor
is 400W is corrected with an explicit log line rather than failing or
being ignored.

Measured on a 4090 mining Poseidon2:
  450W (stock)  185.5 MH/s  70C  0.413 MH/s/W
  300W          183.2 MH/s  61C  0.611
  250W          180.3 MH/s  58C  0.721

44% less power and 12C cooler for 2.8% less hashrate -- this workload is
register-resident integer math and does not need the top of the clock
curve. It also cuts current through the 12VHPWR connector, which is the
one component on these cards with a documented melting failure mode.

Ordered Before=quantus-miner.service so a GPU is never briefly at its
default limit under full mining load.
This commit is contained in:
Rob Thijssen
2026-09-01 06:42:32 +03:00
parent 298dff9343
commit e178bcbdf9
4 changed files with 139 additions and 0 deletions
+30
View File
@@ -56,6 +56,13 @@ EDGE_HOSTS="${EDGE_HOSTS-hanzalova.internal}"
# quantus-specific — it lives here because there is no monitoring repo yet, and
# should move to one if that changes.
GPU_HOSTS="${GPU_HOSTS-beast.hanzalova.internal benjy.hanzalova.internal quadbrat.hanzalova.internal}"
# Per-host GPU power limits, as `host=watts`. Applied at boot by
# nvidia-power-limit.service and clamped to each card's own floor/ceiling, so a
# value the hardware refuses is corrected loudly rather than silently ignored.
# Conservative on purpose: a 4090 measured 183.2 MH/s at 300W against 185.5 at
# 450W, so the hashrate cost is ~1% and the thermal and connector-current
# margin is large. A 5090's floor is 400W; requests below it are clamped.
GPU_POWER_LIMITS="${GPU_POWER_LIMITS-beast.hanzalova.internal=300 benjy.hanzalova.internal=250 quadbrat.hanzalova.internal=150}"
# Vanity name served by the edge proxy. Dot-free short name: the step@ renewal
# unit is a template instance and %i cannot carry dots (internal-tls.md §1).
EDGE_NAME="${EDGE_NAME:-quantus}"
@@ -74,6 +81,7 @@ while [[ $# -gt 0 ]]; do
--metrics-hosts) METRICS_HOSTS="$2"; shift 2 ;;
--edge-hosts) EDGE_HOSTS="$2"; shift 2 ;;
--gpu-hosts) GPU_HOSTS="$2"; shift 2 ;;
--gpu-power-limits) GPU_POWER_LIMITS="$2"; shift 2 ;;
--admin) ADMIN_USER="$2"; shift 2 ;;
--node-version) NODE_VERSION="$2"; shift 2 ;;
*) echo "unknown arg: $1" >&2; exit 2 ;;
@@ -383,15 +391,37 @@ provision_gpu() {
rsync -az --rsync-path='sudo rsync' --chmod=F0755 \
asset/gpu/nvidia-textfile.sh "${ADMIN_USER}@${host}:/usr/local/bin/nvidia-textfile.sh"
rsync -az --rsync-path='sudo rsync' --chmod=F0755 \
asset/gpu/nvidia-power-limit.sh "${ADMIN_USER}@${host}:/usr/local/bin/nvidia-power-limit.sh"
rsync -az --rsync-path='sudo rsync' --chmod=F0644 \
asset/systemd/nvidia-textfile.service asset/systemd/nvidia-textfile.timer \
asset/systemd/nvidia-power-limit.service \
"${ADMIN_USER}@${host}:/etc/systemd/system/"
# Per-host power limit, if one is configured for this host.
local watts=""
for pair in $GPU_POWER_LIMITS; do
case "$pair" in "${host}="*) watts="${pair#*=}" ;; esac
done
if [ -n "$watts" ]; then
printf 'NVIDIA_POWER_LIMIT_WATTS=%s\n' "$watts" \
| ssh "${ADMIN_USER}@${host}" "sudo tee /etc/default/nvidia-power-limit >/dev/null"
echo " = requested power limit ${watts}W"
else
echo " = no power limit configured for this host — cards left at default"
fi
ssh "${ADMIN_USER}@${host}" "
sudo install -d -m 0755 /var/lib/prometheus/node-exporter
sudo systemctl daemon-reload
sudo systemctl enable --now nvidia-textfile.timer
sudo systemctl start nvidia-textfile.service"
if [ -n "$watts" ]; then
ssh "${ADMIN_USER}@${host}" "sudo systemctl enable nvidia-power-limit.service \
&& sudo systemctl restart nvidia-power-limit.service"
ssh "${ADMIN_USER}@${host}" \
"nvidia-smi --query-gpu=index,power.limit --format=csv,noheader" | sed "s/^/ now: /"
fi
local n
n=$(ssh "${ADMIN_USER}@${host}" \