gpu: stop the miner profile failing on a supported memory clock #35

2026-10-12T13:32:56Z

grenade commented

2026-10-12 13:32:56 +00:00

nvidia-power-limit.sh miner intermittently rejects 810 MHz as unsupported while listing it among the supported clocks:

gpu 0 (NVIDIA GeForce RTX 4090): memory clock 810 MHz is not a supported state; supported: 405 MHz 810 MHz 5001 MHz 10251 MHz 10501 MHz

Cause: nvidia-smi -q -d SUPPORTED_CLOCKS | grep -qE ... under set -o pipefail. grep -q exits on the first match, nvidia-smi dies of SIGPIPE writing the rest (exit 141), and pipefail fails the check. Confirmed on benjy: PIPESTATUS = 141 0 with -q, 0 0 without.

Impact: since #34 the miner runs this as ExecStartPre, so a lost race blocks the miner from starting until a Restart=always retry wins. It passed on beast at its 2026-10-08 start and failed on every manual run on beast and benjy on 2026-10-12.

Fix: drop -q, send the match to /dev/null; grep reads all input and nvidia-smi exits 0. The only grep -q reading from a pipe in the repo.

Tested: deployed to benjy via infra-setup.sh --gpu-hosts benjy.hanzalova.internal: 10/10 runs pass (5/5 failed before), and a miner stop/start cycled the 4090 from 450 W / 10501 MHz to 250 W / 810 MHz and back. Not yet deployed to beast or quadbrat.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Mi5JWvH5YyhcsMwfEFowxK

`nvidia-power-limit.sh miner` intermittently rejects 810 MHz as unsupported while listing it among the supported clocks: ``` gpu 0 (NVIDIA GeForce RTX 4090): memory clock 810 MHz is not a supported state; supported: 405 MHz 810 MHz 5001 MHz 10251 MHz 10501 MHz ``` **Cause:** `nvidia-smi -q -d SUPPORTED_CLOCKS | grep -qE ...` under `set -o pipefail`. `grep -q` exits on the first match, nvidia-smi dies of SIGPIPE writing the rest (exit 141), and pipefail fails the check. Confirmed on benjy: `PIPESTATUS` = `141 0` with `-q`, `0 0` without. **Impact:** since #34 the miner runs this as `ExecStartPre`, so a lost race blocks the miner from starting until a `Restart=always` retry wins. It passed on beast at its 2026-10-08 start and failed on every manual run on beast and benjy on 2026-10-12. **Fix:** drop `-q`, send the match to `/dev/null`; grep reads all input and nvidia-smi exits 0. The only `grep -q` reading from a pipe in the repo. **Tested:** deployed to benjy via `infra-setup.sh --gpu-hosts benjy.hanzalova.internal`: 10/10 runs pass (5/5 failed before), and a miner stop/start cycled the 4090 from 450 W / 10501 MHz to 250 W / 810 MHz and back. Not yet deployed to beast or quadbrat. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Mi5JWvH5YyhcsMwfEFowxK

grenade added 1 commit 2026-10-12 13:32:58 +00:00

gpu: stop the miner profile failing on a supported memory clock 2797838bea

`nvidia-power-limit.sh miner` checks the requested memory clock against
`nvidia-smi -q -d SUPPORTED_CLOCKS | grep -qE ...`. grep -q exits on the
first match; nvidia-smi then dies of SIGPIPE writing the rest of its
output (800+ lines on a 4090, 1600+ on a 5090), and pipefail turns its
141 into a failed check: "810 MHz is not a supported state; supported:
... 810 MHz ...", exit 1.

It is timing-dependent. It passed when beast's miner started on
2026-10-08 and failed on every manual run on beast and benjy on
2026-10-12. Since the miner runs this as ExecStartPre, a failure blocks
the miner from starting until a Restart=always retry happens to win the
race.

grep without -q reads its whole input, so nvidia-smi exits 0. Verified
on benjy: 10/10 runs pass with the fix, 5/5 failed without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mi5JWvH5YyhcsMwfEFowxK

grenade merged commit a0ca61b028 into main

2026-10-12 13:34:07 +00:00

grenade deleted branch fix/gpu-profile-sigpipe

2026-10-12 13:34:21 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lair/quantus#35