feat: boot-time GPU power limits, clamped to hardware floors
deploy / fetch (push) Successful in 17s
deploy / deploy-node (bob.hanzalova.internal, 0x134e73f06fa9bdb1dbfa909e149c563f5860ceb71a0e7307918f7033970edf59, benjy.hanzalova.internal quadbrat.hanzalova.internal, --public-addr /dns4/nh.thgttg.com/tcp/30333, --unsafe-rpc-external --rpc-methods safe --rpc-c… (push) Successful in 34s
deploy / deploy-miner (1, quadbrat.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 40s
deploy / deploy-miner (1, benjy.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 43s
deploy / deploy-metrics (push) Successful in 8s
deploy / fetch (push) Successful in 17s
deploy / deploy-node (bob.hanzalova.internal, 0x134e73f06fa9bdb1dbfa909e149c563f5860ceb71a0e7307918f7033970edf59, benjy.hanzalova.internal quadbrat.hanzalova.internal, --public-addr /dns4/nh.thgttg.com/tcp/30333, --unsafe-rpc-external --rpc-methods safe --rpc-c… (push) Successful in 34s
deploy / deploy-miner (1, quadbrat.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 40s
deploy / deploy-miner (1, benjy.hanzalova.internal, bob.hanzalova.internal) (push) Successful in 43s
deploy / deploy-metrics (push) Successful in 8s
nvidia-smi -pl survives neither a reboot nor a driver unload, and both failures are silent -- the card returns to its default limit and nobody notices until a connector runs warm. Hence a boot-time unit, which also enables persistence mode so the limit is not lost when the last GPU client detaches. Per-host limits configured as host=watts in infra-setup.sh, clamped to each card's own min/max. Verified: a 300W request on a 5090 whose floor is 400W is corrected with an explicit log line rather than failing or being ignored. Measured on a 4090 mining Poseidon2: 450W (stock) 185.5 MH/s 70C 0.413 MH/s/W 300W 183.2 MH/s 61C 0.611 250W 180.3 MH/s 58C 0.721 44% less power and 12C cooler for 2.8% less hashrate -- this workload is register-resident integer math and does not need the top of the clock curve. It also cuts current through the 12VHPWR connector, which is the one component on these cards with a documented melting failure mode. Ordered Before=quantus-miner.service so a GPU is never briefly at its default limit under full mining load.
This commit is contained in:
@@ -56,6 +56,13 @@ EDGE_HOSTS="${EDGE_HOSTS-hanzalova.internal}"
|
||||
# quantus-specific — it lives here because there is no monitoring repo yet, and
|
||||
# should move to one if that changes.
|
||||
GPU_HOSTS="${GPU_HOSTS-beast.hanzalova.internal benjy.hanzalova.internal quadbrat.hanzalova.internal}"
|
||||
# Per-host GPU power limits, as `host=watts`. Applied at boot by
|
||||
# nvidia-power-limit.service and clamped to each card's own floor/ceiling, so a
|
||||
# value the hardware refuses is corrected loudly rather than silently ignored.
|
||||
# Conservative on purpose: a 4090 measured 183.2 MH/s at 300W against 185.5 at
|
||||
# 450W, so the hashrate cost is ~1% and the thermal and connector-current
|
||||
# margin is large. A 5090's floor is 400W; requests below it are clamped.
|
||||
GPU_POWER_LIMITS="${GPU_POWER_LIMITS-beast.hanzalova.internal=300 benjy.hanzalova.internal=250 quadbrat.hanzalova.internal=150}"
|
||||
# Vanity name served by the edge proxy. Dot-free short name: the step@ renewal
|
||||
# unit is a template instance and %i cannot carry dots (internal-tls.md §1).
|
||||
EDGE_NAME="${EDGE_NAME:-quantus}"
|
||||
@@ -74,6 +81,7 @@ while [[ $# -gt 0 ]]; do
|
||||
--metrics-hosts) METRICS_HOSTS="$2"; shift 2 ;;
|
||||
--edge-hosts) EDGE_HOSTS="$2"; shift 2 ;;
|
||||
--gpu-hosts) GPU_HOSTS="$2"; shift 2 ;;
|
||||
--gpu-power-limits) GPU_POWER_LIMITS="$2"; shift 2 ;;
|
||||
--admin) ADMIN_USER="$2"; shift 2 ;;
|
||||
--node-version) NODE_VERSION="$2"; shift 2 ;;
|
||||
*) echo "unknown arg: $1" >&2; exit 2 ;;
|
||||
@@ -383,15 +391,37 @@ provision_gpu() {
|
||||
|
||||
rsync -az --rsync-path='sudo rsync' --chmod=F0755 \
|
||||
asset/gpu/nvidia-textfile.sh "${ADMIN_USER}@${host}:/usr/local/bin/nvidia-textfile.sh"
|
||||
rsync -az --rsync-path='sudo rsync' --chmod=F0755 \
|
||||
asset/gpu/nvidia-power-limit.sh "${ADMIN_USER}@${host}:/usr/local/bin/nvidia-power-limit.sh"
|
||||
rsync -az --rsync-path='sudo rsync' --chmod=F0644 \
|
||||
asset/systemd/nvidia-textfile.service asset/systemd/nvidia-textfile.timer \
|
||||
asset/systemd/nvidia-power-limit.service \
|
||||
"${ADMIN_USER}@${host}:/etc/systemd/system/"
|
||||
|
||||
# Per-host power limit, if one is configured for this host.
|
||||
local watts=""
|
||||
for pair in $GPU_POWER_LIMITS; do
|
||||
case "$pair" in "${host}="*) watts="${pair#*=}" ;; esac
|
||||
done
|
||||
if [ -n "$watts" ]; then
|
||||
printf 'NVIDIA_POWER_LIMIT_WATTS=%s\n' "$watts" \
|
||||
| ssh "${ADMIN_USER}@${host}" "sudo tee /etc/default/nvidia-power-limit >/dev/null"
|
||||
echo " = requested power limit ${watts}W"
|
||||
else
|
||||
echo " = no power limit configured for this host — cards left at default"
|
||||
fi
|
||||
|
||||
ssh "${ADMIN_USER}@${host}" "
|
||||
sudo install -d -m 0755 /var/lib/prometheus/node-exporter
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now nvidia-textfile.timer
|
||||
sudo systemctl start nvidia-textfile.service"
|
||||
if [ -n "$watts" ]; then
|
||||
ssh "${ADMIN_USER}@${host}" "sudo systemctl enable nvidia-power-limit.service \
|
||||
&& sudo systemctl restart nvidia-power-limit.service"
|
||||
ssh "${ADMIN_USER}@${host}" \
|
||||
"nvidia-smi --query-gpu=index,power.limit --format=csv,noheader" | sed "s/^/ now: /"
|
||||
fi
|
||||
|
||||
local n
|
||||
n=$(ssh "${ADMIN_USER}@${host}" \
|
||||
|
||||
Reference in New Issue
Block a user