quanpool-miner 6.2.4 measured on beast: 1234 MH/s per 5090, what it does differently, what was tried #27

Closed
opened 2026-09-13 21:48:55 +00:00 by grenade · 5 comments
Owner

Fetched https://download.quanpool.com/quanpool-miner-linux-x86_64 (sha256 1e398a83…f5aaf, "quanpool-miner 6.2.4 (2026-09-11)", a fork of the open-source miner: same metrics names and benchmark subcommand, plus a devfee crate with a licence fetch from license.quanpool.com and telemetry to quanpool.com; 5% dev fee). Run in benchmark mode only, inside a network-less namespace, on beast GPU 0 at the 400 W cap with our miner paused.

Result

kernel MH/s per 5090 per-nonce instructions (ncu) registers threads/SM
ours after #26 1104 22,913 110 512
quantusminer.com qpow-cuda 1.0.7 1171 22,233 128 512
quanpool-miner 6.2.4 1234 20,675 64 1024

quanpool: 1020 blocks x 1024 threads x 8 nonces per launch, 2 streams in flight, autotuned geometry cached in ~/.config/quanpool-miner/tuning.json, "early abort off (measured: costs more than it saves)". 279 SM-cycles per nonce against our 354; issue-active 57.8% vs our 50.6%. After their 5% fee the user-visible rate is ~1172, about 6% above ours.

What the SASS shows (ncu --print-source sass; the cubin is not stored raw in the ELF)

3,014 static instructions, three loops (external-initial 741, internal 397 per two rounds, external-terminal 662), 12 LDG, 14 LDS, 56 LDCU; round constants and the diagonal come from constant memory with computed offsets. Same 32-bit-limb Goldilocks arithmetic as ours and quantusminer's, with two differences:

  1. The 2^64 = EPS fold in the reduction is four ALU ops (IADD, IADD.X, IADD.X, IADD3.X with a -2 immediate) and no multiply; emulated in Python it is bit-for-bit identical to our fused mad.lo.cc/madc.hi.cc reduction on 2M random products including the dropped-borrow cases. Same count per multiply, one fewer IMAD.WIDE.
  2. Carries are deferred in predicates across neighbouring operations (the row-sum carries flow straight into the next multiply-add's IADD3.X; the round-constant add is two instructions with its carry consumed later). That is where their internal round is ~199 instructions to our ~235, i.e. most of the 10% per-nonce difference.

The second point is a ptxas scheduling result over one long carry chain, not something the PTX carry model (a single CC flag) lets hand-written asm express; our asm blocks fence it.

Tried against it (three interleaved rounds each, beast)

  • 64 registers by launch bounds (1024 threads; 256 x 4; 512 x 2): 216-264 bytes of spill, -5%. 80 registers (256 x 3): neutral. Their register fit comes from the loop structure, not from a compiler flag.
  • 8 nonces per thread (MINER_CUDA_BATCH_WAVES=8): -1.2%; 4: neutral.
  • The whole arithmetic in C over unsigned __int128 (to let ptxas schedule carries itself): 40,368 static instructions on sm_120 against 21,120, nvcc's 128-bit lowering is far worse than the asm. Not committed.

Where that leaves it

Both remaining differences (predicate-carried adds across operations, 64-register loop-structured kernel) need the kernel restructured as rolled loops with the arithmetic as one continuous chain per round, so ptxas can schedule carries globally and the register footprint drops. That is a rewrite of permute, with the #25 ptxas anomaly as a standing warning to parity-test every step. Expected gain if it fully lands: ~10% per nonce plus whatever the occupancy buys, i.e. roughly their 1234.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ue5ZZm1Hiv5zPnucykKKuF

Fetched https://download.quanpool.com/quanpool-miner-linux-x86_64 (sha256 1e398a83…f5aaf, "quanpool-miner 6.2.4 (2026-09-11)", a fork of the open-source miner: same metrics names and `benchmark` subcommand, plus a `devfee` crate with a licence fetch from license.quanpool.com and telemetry to quanpool.com; 5% dev fee). Run in benchmark mode only, inside a network-less namespace, on beast GPU 0 at the 400 W cap with our miner paused. ## Result | kernel | MH/s per 5090 | per-nonce instructions (ncu) | registers | threads/SM | |---|---|---|---|---| | ours after #26 | 1104 | 22,913 | 110 | 512 | | quantusminer.com qpow-cuda 1.0.7 | 1171 | 22,233 | 128 | 512 | | **quanpool-miner 6.2.4** | **1234** | **20,675** | 64 | 1024 | quanpool: 1020 blocks x 1024 threads x 8 nonces per launch, 2 streams in flight, autotuned geometry cached in `~/.config/quanpool-miner/tuning.json`, "early abort off (measured: costs more than it saves)". 279 SM-cycles per nonce against our 354; issue-active 57.8% vs our 50.6%. After their 5% fee the user-visible rate is ~1172, about 6% above ours. ## What the SASS shows (ncu --print-source sass; the cubin is not stored raw in the ELF) 3,014 static instructions, three loops (external-initial 741, internal 397 per two rounds, external-terminal 662), 12 LDG, 14 LDS, 56 LDCU; round constants and the diagonal come from constant memory with computed offsets. Same 32-bit-limb Goldilocks arithmetic as ours and quantusminer's, with two differences: 1. The 2^64 = EPS fold in the reduction is four ALU ops (IADD, IADD.X, IADD.X, IADD3.X with a -2 immediate) and no multiply; emulated in Python it is bit-for-bit identical to our fused `mad.lo.cc`/`madc.hi.cc` reduction on 2M random products including the dropped-borrow cases. Same count per multiply, one fewer IMAD.WIDE. 2. Carries are deferred in predicates across neighbouring operations (the row-sum carries flow straight into the next multiply-add's IADD3.X; the round-constant add is two instructions with its carry consumed later). That is where their internal round is ~199 instructions to our ~235, i.e. most of the 10% per-nonce difference. The second point is a ptxas scheduling result over one long carry chain, not something the PTX carry model (a single CC flag) lets hand-written asm express; our asm blocks fence it. ## Tried against it (three interleaved rounds each, beast) - 64 registers by launch bounds (1024 threads; 256 x 4; 512 x 2): 216-264 bytes of spill, **-5%**. 80 registers (256 x 3): neutral. Their register fit comes from the loop structure, not from a compiler flag. - 8 nonces per thread (`MINER_CUDA_BATCH_WAVES=8`): -1.2%; 4: neutral. - The whole arithmetic in C over `unsigned __int128` (to let ptxas schedule carries itself): 40,368 static instructions on sm_120 against 21,120, nvcc's 128-bit lowering is far worse than the asm. Not committed. ## Where that leaves it Both remaining differences (predicate-carried adds across operations, 64-register loop-structured kernel) need the kernel restructured as rolled loops with the arithmetic as one continuous chain per round, so ptxas can schedule carries globally and the register footprint drops. That is a rewrite of `permute`, with the #25 ptxas anomaly as a standing warning to parity-test every step. Expected gain if it fully lands: ~10% per nonce plus whatever the occupancy buys, i.e. roughly their 1234. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Ue5ZZm1Hiv5zPnucykKKuF
Author
Owner

The LAIR_MINBLOCKS launch-bounds knob used for the register experiments is on branch perf/minblocks-knob (c1cac91), pushed but not merged: it changes no default, and merging would only trigger a rebuild-and-redeploy of the same kernel. Merge it with the next real kernel change.

The `LAIR_MINBLOCKS` launch-bounds knob used for the register experiments is on branch `perf/minblocks-knob` (c1cac91), pushed but not merged: it changes no default, and merging would only trigger a rebuild-and-redeploy of the same kernel. Merge it with the next real kernel change.
Author
Owner

Upstream's own engine measured for comparison. origin/main 0ba8ddf (v4.2.0, "CUDA: software-pipelined internal rounds, free-addend products, no early exit"), built on beast, quantus-miner benchmark --cuda-gpu --cpu-workers 0 --gpu-devices 1 --duration 30 on GPU 0 at the 400 W cap, miner paused, NVRTC from CUDA 13.0:

engine MH/s per 5090
upstream wgpu (its default) 278
upstream --cuda-gpu 944
ours (#26), same window, one card 1079
quanpool-miner 6.2.4 1234

So upstream's CUDA engine is 12.5% slower than ours on this card, and quanpool's lead is their own work on top of it, not something we can merge from origin. Upstream CUDA commit titles are not performance evidence; measure before evaluating any of them for the fork.

Upstream's own engine measured for comparison. origin/main 0ba8ddf (v4.2.0, "CUDA: software-pipelined internal rounds, free-addend products, no early exit"), built on beast, `quantus-miner benchmark --cuda-gpu --cpu-workers 0 --gpu-devices 1 --duration 30` on GPU 0 at the 400 W cap, miner paused, NVRTC from CUDA 13.0: | engine | MH/s per 5090 | |---|---| | upstream wgpu (its default) | 278 | | upstream `--cuda-gpu` | 944 | | ours (#26), same window, one card | 1079 | | quanpool-miner 6.2.4 | 1234 | So upstream's CUDA engine is 12.5% slower than ours on this card, and quanpool's lead is their own work on top of it, not something we can merge from origin. Upstream CUDA commit titles are not performance evidence; measure before evaluating any of them for the fork.
Author
Owner

Round 2 (2026-09-14): the loop kernel's internal round, itemised

With mining_loop in (#29), every candidate is now screened by the static size of its internal-round loop body on sm_120 before card time (screen.sh / kbody.py on beast:/tmp; branch perf/int-round-chain carries the variants). Reference: ours 239 instructions at 128 registers, quanpool 199.

Opcode difference per round (ours minus theirs): LDC +12, IADD-family +24, IMAD +11, MOV +6, SEL -8. Broken down:

  • 12 constant reloads. We fetch the twelve diagonal constants from the constant bank every round; they hold them in uniform registers across the loop and fetch only the round constant. This is ptxas's decision, not pressure: at 192 registers it still issues 13 LDC per round, and hoisting the loads in C or delivering the constants as 32-bit halves both compile to byte-identical SASS.
  • ~12 in the product's carry assembly (three IADD3 per multiply-add to their two).
  • ~8 in the reductions: our sixteen per round each use an IMAD.WIDE for the EPS fold; their S-box reductions are ALU-only (count-neutral, different pipe).
  • ~5 in the sum-plus-round-constant fold (two reductions and an accumulator add to their reduce plus a two-op add whose carry rides into the product).
  • ~6 moves.

Screened and rejected (all static, none beat the shipped form): five multiply-add formulations, LAIR_MULADD 0..4: 239 / 260 / 283 / 275 / 265 (1 = whole 64-bit addend with carry-out, the shape of their SASS; 2 = nvcc's product with the addend in C; 3 = addend added to the 128-bit product as a limb chain; 4 = 32-bit carry assembly of the middle products). Constant hoisting: identical. 32-bit-halves diagonal: identical. Internal rounds paired per iteration: worse on hardware (1976 vs 2105).

Registers. 128 with 88 B spill is the floor for this arithmetic in the loop kernel; 64 spills 632 B whatever the block shape. The 4090 nonetheless prefers the loop kernel (+1.7%, deployed on benjy via #29); the 5090 and 3060 do not.

What is left that could move it, in order of plausibility: (1) get the diagonal into uniform registers, which needs a delivery ptxas will not sink back into the loop, e.g. kernel parameters (parameter bank, uniform by construction) or asm volatile loads before the loop; (2) an ALU-only EPS fold in the S-box reductions to rebalance pipes (count-neutral, needs a bench); (3) the round-constant carry riding into the product's high half instead of a second reduction. Each is small; together they are roughly the 40. The register half of the target (64, no spill, 1024 threads) is not reachable by any shape tried.

## Round 2 (2026-09-14): the loop kernel's internal round, itemised With `mining_loop` in (#29), every candidate is now screened by the static size of its internal-round loop body on sm_120 before card time (`screen.sh` / `kbody.py` on beast:/tmp; branch `perf/int-round-chain` carries the variants). Reference: ours 239 instructions at 128 registers, quanpool 199. **Opcode difference per round (ours minus theirs):** LDC +12, IADD-family +24, IMAD +11, MOV +6, SEL -8. Broken down: - **12 constant reloads.** We fetch the twelve diagonal constants from the constant bank every round; they hold them in uniform registers across the loop and fetch only the round constant. This is ptxas's decision, not pressure: at 192 registers it still issues 13 LDC per round, and hoisting the loads in C or delivering the constants as 32-bit halves both compile to byte-identical SASS. - **~12 in the product's carry assembly** (three IADD3 per multiply-add to their two). - **~8 in the reductions**: our sixteen per round each use an IMAD.WIDE for the EPS fold; their S-box reductions are ALU-only (count-neutral, different pipe). - **~5 in the sum-plus-round-constant fold** (two reductions and an accumulator add to their reduce plus a two-op add whose carry rides into the product). - **~6 moves.** **Screened and rejected (all static, none beat the shipped form):** five multiply-add formulations, `LAIR_MULADD` 0..4: 239 / 260 / 283 / 275 / 265 (1 = whole 64-bit addend with carry-out, the shape of their SASS; 2 = nvcc's product with the addend in C; 3 = addend added to the 128-bit product as a limb chain; 4 = 32-bit carry assembly of the middle products). Constant hoisting: identical. 32-bit-halves diagonal: identical. Internal rounds paired per iteration: worse on hardware (1976 vs 2105). **Registers.** 128 with 88 B spill is the floor for this arithmetic in the loop kernel; 64 spills 632 B whatever the block shape. The 4090 nonetheless prefers the loop kernel (+1.7%, deployed on benjy via #29); the 5090 and 3060 do not. **What is left that could move it**, in order of plausibility: (1) get the diagonal into uniform registers, which needs a delivery ptxas will not sink back into the loop, e.g. kernel parameters (parameter bank, uniform by construction) or `asm volatile` loads before the loop; (2) an ALU-only EPS fold in the S-box reductions to rebalance pipes (count-neutral, needs a bench); (3) the round-constant carry riding into the product's high half instead of a second reduction. Each is small; together they are roughly the 40. The register half of the target (64, no spill, 1024 threads) is not reachable by any shape tried.
Author
Owner

Lead (1) from the round-2 list, measured: the diagonal delivered through the launch parameters (MiningUniforms::diag, LAIR_LOOP_PARAM_DIAG). It does not remove the per-round loads (ptxas issues 13 LDC either way, +2 instructions) but the loop kernel's spill drops from 88 to 32 bytes, and that shows on the card: beast loop 2135 -> 2162 MH/s (+1.3%), benjy 725 -> 730 (+0.7%), parity clean. Loop kernel on the 5090 is now 2.0% behind the unrolled one (2162 vs 2206). On branch perf/int-round-chain with the screening variants; PR to follow, benjy gets it through its existing cuda_kernel: loop row.

Lead (1) from the round-2 list, measured: the diagonal delivered through the launch parameters (`MiningUniforms::diag`, `LAIR_LOOP_PARAM_DIAG`). It does not remove the per-round loads (ptxas issues 13 LDC either way, +2 instructions) but the loop kernel's spill drops from 88 to 32 bytes, and that shows on the card: beast loop 2135 -> 2162 MH/s (+1.3%), benjy 725 -> 730 (+0.7%), parity clean. Loop kernel on the 5090 is now 2.0% behind the unrolled one (2162 vs 2206). On branch `perf/int-round-chain` with the screening variants; PR to follow, benjy gets it through its existing `cuda_kernel: loop` row.
Author
Owner

Close-out (2026-09-14)

Where we stand against the quanpool kernel on the same 5090 at 400 W: ours 1107 MH/s per card (unrolled, deployed), theirs 1234; 1172 after their 5% fee. Instruction count per nonce 22.9k vs 20.7k.

What this issue produced:

  • The quanpool kernel fully characterised from its SASS: same 32-bit-limb Goldilocks arithmetic and a bit-identical dropped-borrow reduction, rolled round loops with constants in uniform registers, carries deferred in predicates, 64 registers at 1024 threads per SM, ~199 instructions per internal round to our 239.
  • Upstream's own CUDA engine measured for comparison: 944 MH/s, 15% behind ours, so nothing to take from origin.
  • A gated loop-structured kernel (mining_loop, MINER_CUDA_KERNEL=loop, #29, #31), parity-clean, deployed where it wins: benjy's 4090 +1.7%. On the 5090 it is 2% behind our unrolled kernel.
  • Static screening tooling (loop-body size per candidate, no card time) and a list of what does not work, so nobody repeats it: 64 registers by launch bounds (spills 600-900 B, -5% to collapse), 80 registers (neutral), 8 nonces per thread (-1.2%), whole-addend and three other multiply-add formulations (+20 to +45 instructions per round), C-side constant hoisting and 32-bit-halves delivery (byte-identical SASS), 128-bit C arithmetic (2x the instructions), INT8 tensor-core multiplies (#24, 15x slower).
  • One real gain from the analysis: the diagonal in the launch parameters, +1.3% on the loop kernel.

Why it stops here. The remaining ~40 instructions per internal round are five small items, the largest of which (12 constant reloads per round) is a ptxas scheduling decision that survives every delivery expressible from source, and the register half of the target (64 without spill) is unreachable with this arithmetic in any shape tried. The two ideas left, an ALU-only EPS fold in the S-box reductions and the round-constant carry riding into the product's high half, are worth about a percent each and would not reach instruction parity together. The win condition set for this issue (their instruction count at 1024 threads per SM without spill) is not reachable along this line, and further effort here buys less than the ~1% per step it costs.

Standing guard, unchanged: every kernel change passes parity on all three architectures before merge (#25 remains open for the ptxas anomaly).

Foreign binaries used for the measurements are still on beast under /tmp/kregs (quanpool) and /tmp/kregs/qpow-cuda; delete them if the residual risk of having them there is unwanted.

## Close-out (2026-09-14) **Where we stand against the quanpool kernel on the same 5090 at 400 W:** ours 1107 MH/s per card (unrolled, deployed), theirs 1234; 1172 after their 5% fee. Instruction count per nonce 22.9k vs 20.7k. **What this issue produced:** - The quanpool kernel fully characterised from its SASS: same 32-bit-limb Goldilocks arithmetic and a bit-identical dropped-borrow reduction, rolled round loops with constants in uniform registers, carries deferred in predicates, 64 registers at 1024 threads per SM, ~199 instructions per internal round to our 239. - Upstream's own CUDA engine measured for comparison: 944 MH/s, 15% behind ours, so nothing to take from origin. - A gated loop-structured kernel (`mining_loop`, `MINER_CUDA_KERNEL=loop`, #29, #31), parity-clean, deployed where it wins: benjy's 4090 +1.7%. On the 5090 it is 2% behind our unrolled kernel. - Static screening tooling (loop-body size per candidate, no card time) and a list of what does not work, so nobody repeats it: 64 registers by launch bounds (spills 600-900 B, -5% to collapse), 80 registers (neutral), 8 nonces per thread (-1.2%), whole-addend and three other multiply-add formulations (+20 to +45 instructions per round), C-side constant hoisting and 32-bit-halves delivery (byte-identical SASS), 128-bit C arithmetic (2x the instructions), INT8 tensor-core multiplies (#24, 15x slower). - One real gain from the analysis: the diagonal in the launch parameters, +1.3% on the loop kernel. **Why it stops here.** The remaining ~40 instructions per internal round are five small items, the largest of which (12 constant reloads per round) is a ptxas scheduling decision that survives every delivery expressible from source, and the register half of the target (64 without spill) is unreachable with this arithmetic in any shape tried. The two ideas left, an ALU-only EPS fold in the S-box reductions and the round-constant carry riding into the product's high half, are worth about a percent each and would not reach instruction parity together. The win condition set for this issue (their instruction count at 1024 threads per SM without spill) is not reachable along this line, and further effort here buys less than the ~1% per step it costs. **Standing guard, unchanged:** every kernel change passes parity on all three architectures before merge (#25 remains open for the ptxas anomaly). Foreign binaries used for the measurements are still on beast under /tmp/kregs (quanpool) and /tmp/kregs/qpow-cuda; delete them if the residual risk of having them there is unwanted.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: blackbeard/miner#27