blackbeard/miner
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 2 Wiki Activity

engine-cuda: native CUDA kernel behind MinerEngine #3

New Issue
Closed
opened 2026-09-03 09:08:29 +00:00 by grenade · 5 comments
grenade commented 2026-09-03 09:08:29 +00:00
Owner
Copy Link

Part of #1. Largest single lever identified in the audit. Estimated 1.5x to 2.5x on a 5090 over the current 368 MH/s; the estimate is from instruction counts and needs the harness in the benchmark issue to confirm.

Why WGSL is the ceiling

The PoW is two Poseidon2 permutations per nonce over Goldilocks (p = 2^64 - 2^32 + 1), roughly 1,500 field multiplies per nonce, and nothing else. WGSL has no 64x64 to 128-bit multiply and no multiply-high. gf64_mul in crates/engine-gpu/src/kernels/mining_u64.wgsl therefore splits both operands into 32-bit halves, forms four 64-bit partial products, and reconstructs carries with compare-and-select. Each of those four partials is itself a 64-bit multiply as far as the driver's compiler knows, so it becomes several IMAD.WIDE instructions. The net is on the order of 30 to 40 SASS instructions per field multiply.

In CUDA the same multiply is __umul64hi plus a plain 64-bit multiply, or a mad.lo.cc.u32 / madc.hi.u32 chain in inline PTX, followed by the same 2^64 = 2^32 - 1 reduction. That is roughly 12 to 15 instructions. The permutation is otherwise adds and the same reduction, so the multiply count is the whole story.

Secondary CUDA-only wins, each small: round constants in __constant__ memory instead of whatever naga emits for const arrays indexed by loop variable; explicit #pragma unroll control; PTX slct/predication instead of WGSL select on u64, which naga may lower to a branch.

Design

New crate crates/engine-cuda. Implements MinerEngine from engine-cpu. Nothing in engine-gpu changes.

  • Bindings. cudarc (pure Rust, dlopens libcuda, no build-time CUDA toolkit needed for the Rust side). The kernel is compiled to PTX by nvcc in build.rs when CUDA_HOME is set, else the crate builds with the PTX checked in as a fallback artifact for the CI image. The cuda-builder.cf container origin already ships is the build environment.
  • Kernel. Same bindings as the WGSL kernel: midstate (12 felts), start nonce (16 u32), target (16 u32), results (33 u32 with atomic flag). Same host contract: a batch never carries into the high 256 bits of the nonce, host precomputes the midstate via pow_core::mining_midstate. Same lazy second squeeze. Keeping the contract identical means the parity tests and the batch loop are shared, not duplicated.
  • Constants generated, not copied. build.rs emits the round constants and MDS diagonal from qp-poseidon-constants into a generated header. Origin's WGSL hand-copies them; if the hash ever changes at origin, our CUDA kernel updates by dependency bump. This is the coupling rule from #1 applied to the kernel.
  • Selection. resolve_gpu_configuration in miner-service tries engine-cuda first when --gpu-engine auto|cuda and libcuda loads, falls back to engine-gpu otherwise. The flag defaults to auto. Against an origin node nothing changes on the wire.
  • Multi-GPU. One CudaDevice per card, one stream per worker, same thread-local worker-to-device assignment pattern as engine-gpu so WorkerPool needs no change.
  • Submission. Build it pipelined from the start: two result buffers, cuMemcpyDtoHAsync plus cuEventRecord on stream, poll the event for batch N while batch N+1 runs. The wgpu double-buffering issue does the same for the origin engine; here it is free.

Steps

  1. Crate skeleton, cudarc device enumeration, PTX load, MinerEngine impl that returns Exhausted { 0 }. Wire into resolution behind the flag. Bench harness sees it.
  2. Straight port of mining_u64.wgsl to CUDA C, schoolbook multiply kept. Parity passes. Establishes the port is correct before any arithmetic changes.
  3. Replace gf64_mul/gf64_sqr with __umul64hi. Measure.
  4. Inline PTX mad chains. Measure. Keep whichever of 3 and 4 wins.
  5. __constant__ round constants, unroll tuning, occupancy check with --ptxas-options=-v. Measure each.
  6. Pipelined submission. Measure.

Each step is a PR with its harness numbers in the description.

Risks

  • SHADER_INT64 parity tests are wgpu-specific. The CUDA parity path calls pow_core on the CPU for the same nonces; the component tests for individual layers do not port and are not needed once end-to-end parity holds at scale (origin's reverted "fuzz GPU/CPU parity at scale" commit b0cfc30 is the template).
  • Driver dependency on the fleet: libcuda.so is present wherever the NVIDIA driver is. No toolkit needed at runtime.
  • Blackwell PTX ISA: target sm_100/sm_120 explicitly; check what the CUDA 12.9 builder image supports.

Origin coupling

New crate plus the selection arm and one flag. The generated-constants rule means a hash change at origin is a Cargo.lock bump here. A mining-protocol change at origin does not touch this crate at all.

Part of #1. Largest single lever identified in the audit. Estimated 1.5x to 2.5x on a 5090 over the current 368 MH/s; the estimate is from instruction counts and needs the harness in the benchmark issue to confirm. ## Why WGSL is the ceiling The PoW is two Poseidon2 permutations per nonce over Goldilocks (`p = 2^64 - 2^32 + 1`), roughly 1,500 field multiplies per nonce, and nothing else. WGSL has no 64x64 to 128-bit multiply and no multiply-high. `gf64_mul` in `crates/engine-gpu/src/kernels/mining_u64.wgsl` therefore splits both operands into 32-bit halves, forms four 64-bit partial products, and reconstructs carries with compare-and-select. Each of those four partials is itself a 64-bit multiply as far as the driver's compiler knows, so it becomes several IMAD.WIDE instructions. The net is on the order of 30 to 40 SASS instructions per field multiply. In CUDA the same multiply is `__umul64hi` plus a plain 64-bit multiply, or a `mad.lo.cc.u32` / `madc.hi.u32` chain in inline PTX, followed by the same `2^64 = 2^32 - 1` reduction. That is roughly 12 to 15 instructions. The permutation is otherwise adds and the same reduction, so the multiply count is the whole story. Secondary CUDA-only wins, each small: round constants in `__constant__` memory instead of whatever naga emits for `const` arrays indexed by loop variable; explicit `#pragma unroll` control; PTX `slct`/predication instead of WGSL `select` on u64, which naga may lower to a branch. ## Design New crate `crates/engine-cuda`. Implements `MinerEngine` from `engine-cpu`. Nothing in `engine-gpu` changes. - **Bindings.** `cudarc` (pure Rust, dlopens `libcuda`, no build-time CUDA toolkit needed for the Rust side). The kernel is compiled to PTX by `nvcc` in `build.rs` when `CUDA_HOME` is set, else the crate builds with the PTX checked in as a fallback artifact for the CI image. The `cuda-builder.cf` container origin already ships is the build environment. - **Kernel.** Same bindings as the WGSL kernel: midstate (12 felts), start nonce (16 u32), target (16 u32), results (33 u32 with atomic flag). Same host contract: a batch never carries into the high 256 bits of the nonce, host precomputes the midstate via `pow_core::mining_midstate`. Same lazy second squeeze. Keeping the contract identical means the parity tests and the batch loop are shared, not duplicated. - **Constants generated, not copied.** `build.rs` emits the round constants and MDS diagonal from `qp-poseidon-constants` into a generated header. Origin's WGSL hand-copies them; if the hash ever changes at origin, our CUDA kernel updates by dependency bump. This is the coupling rule from #1 applied to the kernel. - **Selection.** `resolve_gpu_configuration` in `miner-service` tries `engine-cuda` first when `--gpu-engine auto|cuda` and `libcuda` loads, falls back to `engine-gpu` otherwise. The flag defaults to `auto`. Against an origin node nothing changes on the wire. - **Multi-GPU.** One `CudaDevice` per card, one stream per worker, same thread-local worker-to-device assignment pattern as `engine-gpu` so `WorkerPool` needs no change. - **Submission.** Build it pipelined from the start: two result buffers, `cuMemcpyDtoHAsync` plus `cuEventRecord` on stream, poll the event for batch N while batch N+1 runs. The wgpu double-buffering issue does the same for the origin engine; here it is free. ## Steps 1. Crate skeleton, `cudarc` device enumeration, PTX load, `MinerEngine` impl that returns `Exhausted { 0 }`. Wire into resolution behind the flag. Bench harness sees it. 2. Straight port of `mining_u64.wgsl` to CUDA C, schoolbook multiply kept. Parity passes. Establishes the port is correct before any arithmetic changes. 3. Replace `gf64_mul`/`gf64_sqr` with `__umul64hi`. Measure. 4. Inline PTX mad chains. Measure. Keep whichever of 3 and 4 wins. 5. `__constant__` round constants, unroll tuning, occupancy check with `--ptxas-options=-v`. Measure each. 6. Pipelined submission. Measure. Each step is a PR with its harness numbers in the description. ## Risks - `SHADER_INT64` parity tests are wgpu-specific. The CUDA parity path calls `pow_core` on the CPU for the same nonces; the component tests for individual layers do not port and are not needed once end-to-end parity holds at scale (origin's reverted "fuzz GPU/CPU parity at scale" commit b0cfc30 is the template). - Driver dependency on the fleet: `libcuda.so` is present wherever the NVIDIA driver is. No toolkit needed at runtime. - Blackwell PTX ISA: target `sm_100`/`sm_120` explicitly; check what the CUDA 12.9 builder image supports. ## Origin coupling New crate plus the selection arm and one flag. The generated-constants rule means a hash change at origin is a `Cargo.lock` bump here. A mining-protocol change at origin does not touch this crate at all.
grenade added the needs-benchmarkorigin-couplingperf labels 2026-09-03 09:10:15 +00:00
grenade commented 2026-09-03 09:23:57 +00:00
Author
Owner
Copy Link

Origin history has a CUDA engine crate that was removed when wgpu arrived: crates/engine-gpu-cuda at 5a6de31^ (deleted in "add wgpu support (#28)"), with .github/workflows/cuda-build.yml and a Containerfile from d2c39bb ("ci: build cuda binaries").

Its kernel is not reusable: it was written for the earlier proof of work (512-bit Montgomery CIOS multiply via __umul64hi, Keccak constants, "G1" bring-up with Poseidon2 on the host). The scaffolding is:

  • build.rs: nvcc discovery via NVCC / CUDA_HOME / PATH, CUDA_ARCH=sm_NN normalised to -arch compute_NN -code sm_NN, cubin plus PTX both embedded with a header preflight that fails the build rather than emit a GPU-broken binary, MINER_NVCC_CCBIN and MINER_CUDA_ALLOW_UNSUPPORTED_COMPILER knobs.
  • mul64wide / add64_carry helpers using __umul64hi, which is exactly step 3 of this issue's plan.
  • scripts/local-cuda-build.sh (still in tree) with the fleet's arch list 86 / 89 / 120 and CUDA 13.0.

Start step 1 from that build.rs rather than from cudarc's examples; it already encodes the fleet's build constraints. The arch matrix in the CI issue assumes its CUDA_ARCH contract.

Origin history has a CUDA engine crate that was removed when wgpu arrived: `crates/engine-gpu-cuda` at `5a6de31^` (deleted in "add wgpu support (#28)"), with `.github/workflows/cuda-build.yml` and a `Containerfile` from `d2c39bb` ("ci: build cuda binaries"). Its kernel is not reusable: it was written for the earlier proof of work (512-bit Montgomery CIOS multiply via `__umul64hi`, Keccak constants, "G1" bring-up with Poseidon2 on the host). The scaffolding is: - `build.rs`: `nvcc` discovery via `NVCC` / `CUDA_HOME` / `PATH`, `CUDA_ARCH=sm_NN` normalised to `-arch compute_NN -code sm_NN`, cubin plus PTX both embedded with a header preflight that fails the build rather than emit a GPU-broken binary, `MINER_NVCC_CCBIN` and `MINER_CUDA_ALLOW_UNSUPPORTED_COMPILER` knobs. - `mul64wide` / `add64_carry` helpers using `__umul64hi`, which is exactly step 3 of this issue's plan. - `scripts/local-cuda-build.sh` (still in tree) with the fleet's arch list `86 / 89 / 120` and CUDA 13.0. Start step 1 from that `build.rs` rather than from `cudarc`'s examples; it already encodes the fleet's build constraints. The arch matrix in the CI issue assumes its `CUDA_ARCH` contract.
grenade commented 2026-09-03 11:57:51 +00:00
Author
Owner
Copy Link

Steps 1 to 3 landed: #16 (2026-09-03)

Harness on benjy via the bench workflow (actions/runs/28), miner paused, 5 x 30 s windows, batch 1M:

engine card power limit median MH/s spread parity
wgpu u64 (baseline, #2) RTX 4090 250 W 144.4 0.2% 25/25
CUDA RTX 4090 250 W 297.7 0.1% 25/25

2.06x on the reference card. On quadbrat's 3060 at 130 W: 37.3 to 59.5 MH/s, 1.59x. Both bit-exact against pow_core on every job.

What the port does differently from the WGSL: the field multiply is __umul64hi plus a plain 64-bit multiply and one Goldilocks reduction, and the round loops are unrolled with the constants in __constant__ memory. Nothing else; the sponge schedule and host contract are identical, which is why parity held first time.

Grid sizing (MINER_CUDA_THREADS_PER_SM): 8192 and above flat on the 3060, 2048 loses 10%. Default 8192. A 4M batch measured slower than 1M on the 3060 (52.6 vs 59.5), which is the per-thread nonce loop at that grid cap; worth a sweep on the 4090 with the harness before #6 touches batch size for the CUDA engine.

Remaining from this issue's plan: step 5 (constant handling and unroll tuning, --ptxas-options=-v occupancy check), step 6 (pipelined submission; worth at most the host share of a batch, which #9 measures live), and the 5090 number, which needs beast and a manual dispatch.

Deployed to both hosts by the merge; validate asserts kernel="cuda" on device 0.

## Steps 1 to 3 landed: #16 (2026-09-03) Harness on benjy via the `bench` workflow (actions/runs/28), miner paused, 5 x 30 s windows, batch 1M: | engine | card | power limit | median MH/s | spread | parity | | --- | --- | --- | --- | --- | --- | | wgpu u64 (baseline, #2) | RTX 4090 | 250 W | 144.4 | 0.2% | 25/25 | | **CUDA** | RTX 4090 | 250 W | **297.7** | 0.1% | 25/25 | **2.06x** on the reference card. On quadbrat's 3060 at 130 W: 37.3 to 59.5 MH/s, 1.59x. Both bit-exact against `pow_core` on every job. What the port does differently from the WGSL: the field multiply is `__umul64hi` plus a plain 64-bit multiply and one Goldilocks reduction, and the round loops are unrolled with the constants in `__constant__` memory. Nothing else; the sponge schedule and host contract are identical, which is why parity held first time. Grid sizing (`MINER_CUDA_THREADS_PER_SM`): 8192 and above flat on the 3060, 2048 loses 10%. Default 8192. A 4M batch measured slower than 1M on the 3060 (52.6 vs 59.5), which is the per-thread nonce loop at that grid cap; worth a sweep on the 4090 with the harness before #6 touches batch size for the CUDA engine. Remaining from this issue's plan: step 5 (constant handling and unroll tuning, `--ptxas-options=-v` occupancy check), step 6 (pipelined submission; worth at most the host share of a batch, which #9 measures live), and the 5090 number, which needs beast and a manual dispatch. Deployed to both hosts by the merge; validate asserts `kernel="cuda"` on device 0.
grenade commented 2026-09-03 13:01:58 +00:00
Author
Owner
Copy Link

Step 5 landed: #17 (2026-09-03)

Every variant parity-checked on quadbrat, then timed on benjy (RTX 4090, 250 W, miner paused) in interleaved rounds. Full-protocol confirmation by the bench workflow: 408.27 MH/s, spread 0.2%, parity 25/25 (actions/runs/33). Origin's wgpu build on the same card and limit: 144.4. 2.83x.

change MH/s note
straight port (#16) 297 to 305
grid cap 32768 threads/SM 304 one nonce per thread up to 4M; batch size then irrelevant (1M to 16M all 304 to 306)
deferred-carry linear layers 344 +13%
carry-flag gf_add / gf_reduce (inline PTX) 385 +12%; clocks rose 1920 to 1995 MHz under the power cap
carry-flag accumulators 414 +7%
fold round constants into the preceding layer 408 to 418 within noise; kept for the instruction count

Rejected: __noinline__ permute (-13%), -maxrregcount 80 and 64 (flat), internal-round unroll 1 or 22 (flat). Code size and occupancy are not the limit on Ada; instruction count per hash is, and the adds were a larger share of it than the audit assumed.

Measurement notes for whoever continues: run-to-run noise on benjy is about ±1.5% with the SM clock moving 1935 to 2025 MHz at the cap, so anything under 3% needs interleaved rounds and is not resolvable in a single pass. Ad-hoc runs over ssh do not take the workflow's host lock; do not run them while a bench job is measuring on the same host.

Still open on this issue: step 6 (pipelined submission; ceiling is the host share of a batch, 1.9% on the 4090 per #9, less now that batches are shorter), the sm_120 codegen question (255 registers with spills on Blackwell against 128 on Ada), and the 5090 number itself. Both of the last two need beast.

## Step 5 landed: #17 (2026-09-03) Every variant parity-checked on quadbrat, then timed on benjy (RTX 4090, 250 W, miner paused) in interleaved rounds. Full-protocol confirmation by the `bench` workflow: **408.27 MH/s, spread 0.2%, parity 25/25** (actions/runs/33). Origin's wgpu build on the same card and limit: 144.4. **2.83x.** | change | MH/s | note | | --- | --- | --- | | straight port (#16) | 297 to 305 | | | grid cap 32768 threads/SM | 304 | one nonce per thread up to 4M; batch size then irrelevant (1M to 16M all 304 to 306) | | deferred-carry linear layers | 344 | +13% | | carry-flag `gf_add` / `gf_reduce` (inline PTX) | 385 | +12%; clocks rose 1920 to 1995 MHz under the power cap | | carry-flag accumulators | 414 | +7% | | fold round constants into the preceding layer | 408 to 418 | within noise; kept for the instruction count | Rejected: `__noinline__` permute (-13%), `-maxrregcount` 80 and 64 (flat), internal-round unroll 1 or 22 (flat). Code size and occupancy are not the limit on Ada; instruction count per hash is, and the adds were a larger share of it than the audit assumed. Measurement notes for whoever continues: run-to-run noise on benjy is about ±1.5% with the SM clock moving 1935 to 2025 MHz at the cap, so anything under 3% needs interleaved rounds and is not resolvable in a single pass. Ad-hoc runs over ssh do not take the workflow's host lock; do not run them while a `bench` job is measuring on the same host. Still open on this issue: step 6 (pipelined submission; ceiling is the host share of a batch, 1.9% on the 4090 per #9, less now that batches are shorter), the `sm_120` codegen question (255 registers with spills on Blackwell against 128 on Ada), and the 5090 number itself. Both of the last two need beast.
grenade commented 2026-09-03 15:20:03 +00:00
Author
Owner
Copy Link

The 5090 number (2026-09-03, beast, neuron stopped for the run)

Both RTX 5090s at their 400 W floor, driver 580.178.04, binary built on beast from main (ad66f13), harness with two workers (one per card), parity 12/12 verified against CPU on each engine.

engine card 0 card 1 host vs origin
wgpu u64 (origin) 282.9 283.3 566 MH/s (matches lair/quantus's old 281 MH/s at 400 W)
CUDA, tuned (#17) 549.7 559.2 1,109 MH/s 1.96x

Single card alone: 553.7. Interleaved rounds put the host at 1,109 to 1,134 MH/s (clock 2265 to 2317 MHz at the cap).

Checked and settled:

  • sm_120 codegen: with the tuned kernel Blackwell compiles at 128 registers with a 104-byte spill (the earlier 255-register figure was the pre-tuning kernel). LAIR_INT_UNROLL=1 removes the spill but measures 1 to 1.5% slower (1,118 / 1,109 vs 1,134 / 1,122 interleaved). Default stays.
  • Batch 4M: 1,121, flat with 1M. Grid cap 131072: 1,114, flat. Defaults stay.

The 5090's gain over origin (1.96x) is smaller than the 4090's (2.83x) because origin's wgpu kernel is relatively better on Blackwell, not because the CUDA kernel is worse: per card the CUDA kernel does 554 MH/s at 400 W against 414 on the 4090 at 250 W, which is the same 1.4 MH/s per watt.

Fleet total on the tuned engine, if beast mined: 414 + 81 + 1,109 = about 1.6 GH/s.

## The 5090 number (2026-09-03, beast, neuron stopped for the run) Both RTX 5090s at their 400 W floor, driver 580.178.04, binary built on beast from `main` (ad66f13), harness with two workers (one per card), parity 12/12 verified against CPU on each engine. | engine | card 0 | card 1 | host | vs origin | | --- | --- | --- | --- | --- | | wgpu u64 (origin) | 282.9 | 283.3 | 566 MH/s | (matches lair/quantus's old 281 MH/s at 400 W) | | CUDA, tuned (#17) | 549.7 | 559.2 | **1,109 MH/s** | **1.96x** | Single card alone: 553.7. Interleaved rounds put the host at 1,109 to 1,134 MH/s (clock 2265 to 2317 MHz at the cap). Checked and settled: - **sm_120 codegen**: with the tuned kernel Blackwell compiles at 128 registers with a 104-byte spill (the earlier 255-register figure was the pre-tuning kernel). `LAIR_INT_UNROLL=1` removes the spill but measures 1 to 1.5% slower (1,118 / 1,109 vs 1,134 / 1,122 interleaved). Default stays. - **Batch 4M**: 1,121, flat with 1M. **Grid cap 131072**: 1,114, flat. Defaults stay. The 5090's gain over origin (1.96x) is smaller than the 4090's (2.83x) because origin's wgpu kernel is relatively better on Blackwell, not because the CUDA kernel is worse: per card the CUDA kernel does 554 MH/s at 400 W against 414 on the 4090 at 250 W, which is the same 1.4 MH/s per watt. Fleet total on the tuned engine, if beast mined: 414 + 81 + 1,109 = about 1.6 GH/s.
grenade commented 2026-09-03 15:22:57 +00:00
Author
Owner
Copy Link

Closing: the engine is deployed on every mining host and measured on every card in the fleet.

card origin wgpu lair CUDA gain
RTX 3060 at 130 W 37.3 80.9 (live) 2.17x
RTX 4090 at 250 W 144.4 408 to 414 2.83x
RTX 5090 at 400 W, per card 283 550 to 559 1.96x

Step 6 (pipelined submission) is not worth its own issue on this engine: the host share of a batch is under 2% (#9), so it is folded into #5, which stays open as the one pipelining issue for both engines. Constants are generated from qp-poseidon-constants at build time, the fat binary carries sm_86/89/120 plus PTX, and every kernel knob remains a -D macro reachable through MINER_NVCC_FLAGS for the harness.

Closing: the engine is deployed on every mining host and measured on every card in the fleet. | card | origin wgpu | lair CUDA | gain | | --- | --- | --- | --- | | RTX 3060 at 130 W | 37.3 | 80.9 (live) | 2.17x | | RTX 4090 at 250 W | 144.4 | 408 to 414 | 2.83x | | RTX 5090 at 400 W, per card | 283 | 550 to 559 | 1.96x | Step 6 (pipelined submission) is not worth its own issue on this engine: the host share of a batch is under 2% (#9), so it is folded into #5, which stays open as the one pipelining issue for both engines. Constants are generated from `qp-poseidon-constants` at build time, the fat binary carries sm_86/89/120 plus PTX, and every kernel knob remains a `-D` macro reachable through `MINER_NVCC_FLAGS` for the harness.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
needs-benchmark
Gain is estimated, not measured; needs a run on a 5090 host
 needs-chain
May require a matching change in quantus/chain
 origin-coupling
Touches how the lair fork tracks Quantus-Network/quantus-miner
 perf
Mining throughput workstream
No Label needs-benchmark origin-coupling perf
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
grenade
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: blackbeard/miner#3
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?