lair/quantus
2
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

gpu: apply the miner's clock profile only while the miner runs #34

Merged
grenade merged 1 commits from feat/gpu-profile-follows-workload into main 2026-09-22 13:10:59 +00:00
Conversation 0 Commits 1 Files Changed 5
+233 -113
grenade commented 2026-09-22 13:02:12 +00:00
Owner
Copy Link
Copy Source

Problem

nvidia-power-limit.service applied the miner profile (power limit + -lmc 810,810) at every boot as a host setting. A host switched from mining back to LLM inference kept the lock with the miner stopped. Inference is memory-bandwidth bound: a 27B model on 2× RTX 5090 decoded at 3.8 tok/s on a build that had measured 21.9 tok/s. Nothing reported it. A -lmc lock is not a clocks event reason, utilisation reads ~98%, and the only signs are P5, the 810 MHz memory clock and ~125 W of power draw.

Change

The profile now follows the workload instead of the host:

  • nvidia-power-limit.sh takes a mode. miner applies /etc/default/nvidia-power-limit exactly as before. default resets each card to its own power.default_limit and runs -rmc.
  • nvidia-power-limit.service runs default at boot, ordered before both quantus-miner and neuron.
  • New asset/systemd/neuron-gpu-exclusive.conf → neuron.service.d/50-gpu-exclusive.conf. It sets Conflicts=quantus-miner.service, After=quantus-miner.service so the miner's ExecStopPost reset finishes first, and ExecStartPre=+… default. Starting either service stops the other, and the clocks follow.
  • infra-setup.sh:
    • installs the drop-in on every GPU host;
    • always writes the profile config, so removing a host from the lists clears its profile;
    • applies the profile that matches the running workload, instead of restarting the boot unit under a live miner.
  • readme: a new GPU profiles and inference section, and a rewritten Known conflict section.

Companion PR

quantus/miner (feat/gpu-profile-with-lifecycle) adds the other half:

  • ExecStartPre=+nvidia-power-limit.sh miner / ExecStopPost=+… default on the unit;
  • a deploy guard that ships files but won't enable or start the miner on a host where neuron.service is enabled;
  • bench changes that re-apply the profile for the measurement window.

Rollout order

  1. Merge this PR, then run script/infra-setup.sh against the GPU hosts. It installs the new script and drop-in, and resets any host not currently mining.
  2. Re-run quantus/miner's deploy/infra-setup.sh, which adds new sudoers lines for the bench.
  3. Merge the miner PR.

The two halves can land in either order. The old script ignores the mode argument, so it keeps working in the meantime, just without the reset on stop.

Tested

  • nvidia-power-limit.sh was exercised against a stub nvidia-smi: both modes, clamping, a bad mode (exit 2), a failing -rmc (exit 1, loud), and an empty profile.
  • systemd-analyze verify passes on neuron.service with the drop-in and on the boot unit.
  • Not yet run on real cards.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA

## Problem `nvidia-power-limit.service` applied the miner profile (power limit + `-lmc 810,810`) at every boot as a *host* setting. A host switched from mining back to LLM inference kept the lock with the miner stopped. Inference is memory-bandwidth bound: a 27B model on 2× RTX 5090 decoded at **3.8 tok/s** on a build that had measured **21.9 tok/s**. Nothing reported it. A `-lmc` lock is not a clocks event reason, utilisation reads ~98%, and the only signs are P5, the 810 MHz memory clock and ~125 W of power draw. ## Change The profile now follows the workload instead of the host: - `nvidia-power-limit.sh` takes a mode. `miner` applies `/etc/default/nvidia-power-limit` exactly as before. `default` resets each card to its own `power.default_limit` and runs `-rmc`. - `nvidia-power-limit.service` runs `default` at boot, ordered before both `quantus-miner` and `neuron`. - New `asset/systemd/neuron-gpu-exclusive.conf` → `neuron.service.d/50-gpu-exclusive.conf`. It sets `Conflicts=quantus-miner.service`, `After=quantus-miner.service` so the miner's `ExecStopPost` reset finishes first, and `ExecStartPre=+… default`. Starting either service stops the other, and the clocks follow. - `infra-setup.sh`: - installs the drop-in on every GPU host; - always writes the profile config, so removing a host from the lists clears its profile; - applies the profile that matches the running workload, instead of restarting the boot unit under a live miner. - readme: a new *GPU profiles and inference* section, and a rewritten *Known conflict* section. ## Companion PR quantus/miner (`feat/gpu-profile-with-lifecycle`) adds the other half: - `ExecStartPre=+nvidia-power-limit.sh miner` / `ExecStopPost=+… default` on the unit; - a deploy guard that ships files but won't enable or start the miner on a host where `neuron.service` is enabled; - bench changes that re-apply the profile for the measurement window. ## Rollout order 1. Merge this PR, then run `script/infra-setup.sh` against the GPU hosts. It installs the new script and drop-in, and resets any host not currently mining. 2. Re-run quantus/miner's `deploy/infra-setup.sh`, which adds new sudoers lines for the bench. 3. Merge the miner PR. The two halves can land in either order. The old script ignores the mode argument, so it keeps working in the meantime, just without the reset on stop. ## Tested - `nvidia-power-limit.sh` was exercised against a stub `nvidia-smi`: both modes, clamping, a bad mode (exit 2), a failing `-rmc` (exit 1, loud), and an empty profile. - `systemd-analyze verify` passes on `neuron.service` with the drop-in and on the boot unit. - Not yet run on real cards. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA
grenade added 1 commit 2026-09-22 13:02:13 +00:00
gpu: apply the miner's clock profile only while the miner runs 80bbe12455 
The power limit and 810 MHz memory lock were a host setting, applied at
every boot by nvidia-power-limit.service regardless of what the GPU was
doing. A host switched from mining back to LLM inference kept the lock
with the miner stopped, and inference is memory-bandwidth bound: a 27B
model on 2x RTX 5090 decoded at 3.8 tok/s on a build that had measured
21.9 tok/s. Nothing reported it — a -lmc lock is not a clocks event
reason, and utilisation still reads ~98%.

The profile now follows the workload:

- nvidia-power-limit.sh takes a mode. `miner` applies the configured
  profile as before; `default` resets each card to its own default power
  limit and memory clock.
- nvidia-power-limit.service runs `default` at boot. The miner applies
  `miner` in ExecStartPre and `default` in ExecStopPost (quantus/miner).
- A drop-in for neuron.service makes it conflict with the miner and
  reset to default before starting, so either one can take the host with
  a single systemctl start and the clocks follow.
- infra-setup.sh installs the drop-in, always writes the profile config
  (so removing a host clears it), and applies whichever profile matches
  the running workload instead of restarting the boot unit under a live
  miner.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA
grenade merged commit f7337fe9a1 into main 2026-09-22 13:10:59 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
agent-bot (agent credential bot) grenade
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lair/quantus#34