deploy: tie the GPU profile to the miner's lifecycle #33

Merged
grenade merged 1 commits from feat/gpu-profile-with-lifecycle into main 2026-09-22 13:14:30 +00:00
Owner

Problem

The miner's GPU profile (lower power limit + memory locked at 810 MHz) was applied at boot as a host setting, so it outlived the miner. A host switched back to LLM inference kept the lock and decoded at about a sixth of its measured speed (3.8 vs 21.9 tok/s for a 27B model on 2× 5090). Nothing reported it as throttling.

Change

  • Unit: ExecStartPre=+/usr/local/bin/nvidia-power-limit.sh miner and ExecStopPost=+… default. ExecStopPost runs on stop, crash and failed start. The script and its per-host config come from lair/quantus.
  • Deploy: on a host where neuron.service is enabled, ship the files and daemon-reload, but do not enable or start the miner. Validate passes there without checking hashing. neuron now Conflicts= the miner (lair/quantus drop-in), so starting the miner would stop inference behind the operator's back. Switching a host is a deliberate systemctl disable --now … && enable --now ….
  • Bench: stopping the unit now resets the clocks, so bench-on-host.sh:
    • re-applies the miner profile for the measurement window;
    • reads the asserted power limit only after applying it;
    • restores the prior profile afterwards;
    • refuses to measure where neuron is active.
  • Sudoers: deploy/infra-setup.sh grants the two script invocations the bench needs. Re-run it on the mining hosts before the next deploy. Until then the preflight will name the missing path.

Depends on

lair/quantus feat/gpu-profile-follows-workload, which provides the moded script, the boot-time reset and the neuron drop-in. Either can land first: the old script ignores the mode argument and applies the profile, which is the same behaviour as today.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA

## Problem The miner's GPU profile (lower power limit + memory locked at 810 MHz) was applied at boot as a host setting, so it outlived the miner. A host switched back to LLM inference kept the lock and decoded at about a sixth of its measured speed (3.8 vs 21.9 tok/s for a 27B model on 2× 5090). Nothing reported it as throttling. ## Change - **Unit**: `ExecStartPre=+/usr/local/bin/nvidia-power-limit.sh miner` and `ExecStopPost=+… default`. `ExecStopPost` runs on stop, crash and failed start. The script and its per-host config come from lair/quantus. - **Deploy**: on a host where `neuron.service` is enabled, ship the files and `daemon-reload`, but do **not** enable or start the miner. Validate passes there without checking hashing. neuron now `Conflicts=` the miner (lair/quantus drop-in), so starting the miner would stop inference behind the operator's back. Switching a host is a deliberate `systemctl disable --now … && enable --now …`. - **Bench**: stopping the unit now resets the clocks, so `bench-on-host.sh`: - re-applies the miner profile for the measurement window; - reads the asserted power limit only after applying it; - restores the prior profile afterwards; - refuses to measure where neuron is active. - **Sudoers**: `deploy/infra-setup.sh` grants the two script invocations the bench needs. **Re-run it on the mining hosts before the next deploy.** Until then the preflight will name the missing path. ## Depends on lair/quantus `feat/gpu-profile-follows-workload`, which provides the moded script, the boot-time reset and the neuron drop-in. Either can land first: the old script ignores the mode argument and applies the profile, which is the same behaviour as today. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA
grenade added 1 commit 2026-09-22 13:02:21 +00:00
deploy: tie the GPU profile to the miner's lifecycle
ci / fmt (pull_request) Successful in 23s
bench / build (pull_request) Successful in 1m19s
ci / clippy (pull_request) Successful in 2m23s
ci / doc (pull_request) Successful in 2m29s
bench / measure (pull_request) Failing after 2m23s
ci / test (pull_request) Successful in 8m30s
97073b0971
The power limit and 810 MHz memory lock were applied at boot as a host
setting, so they outlived the miner: a host switched back to LLM
inference kept the lock and decoded at a sixth of its measured speed,
with nothing reporting as throttled.

- quantus-miner.service applies the profile in ExecStartPre and resets
  to driver defaults in ExecStopPost (nvidia-power-limit.sh, installed
  by lair/quantus). ExecStopPost runs on stop, crash and failed start.
- deploy: on a host with neuron.service enabled, ship the files but do
  not enable or start the miner; validate passes there without checking
  hashing. neuron conflicts with the miner, so starting it would stop
  inference behind the operator's back.
- bench-on-host.sh: stopping the unit now resets the clocks, so apply
  the miner profile for the measurement window (and read the asserted
  power limit after it), restore the prior profile afterwards, and
  refuse to measure on a host where neuron is active.
- infra-setup.sh: sudoers for the two script invocations the bench
  needs. Re-run it on the mining hosts before the next deploy; the
  preflight will name the missing path until then.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA
grenade merged commit 1520e19f2a into main 2026-09-22 13:14:30 +00:00
grenade deleted branch feat/gpu-profile-with-lifecycle 2026-09-22 13:14:31 +00:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: blackbeard/miner#33