Batch size: raise the default and derive it from measured hashrate and a stale-work budget #6

Closed
opened 2026-09-03 09:09:22 +00:00 by grenade · 1 comment
Owner

Part of #1. Available today by flag, so the deployment repo (lair/quantus) can set it before any code changes. This issue is about making the miner pick well on its own.

Current behaviour

DEFAULT_GPU_BATCH_SIZE = 1_000_000 in crates/miner-cli/src/main.rs. Origin's commit 58615ed measured micro-optimisations at 16M and saw gains that were noise at 1M, so they know larger batches help and kept 1M anyway, presumably for cancellation latency on small cards.

Two things happen at 1M on a 5090 that do not happen at 16M:

  • The tier hint does not bind. get_vendor_specific_dispatch computes 10,922 workgroups for a Blackwell flagship (2.8M threads), but run_single_batch clamps logical threads to the batch, so 1M nonces means ~3,900 workgroups and exactly one nonce per thread. Every thread pays the per-thread loads (midstate, target, nonce base, ~44 storage reads) for one hash. At 16M the hint binds, each thread loops six nonces, and those loads amortise. That is what the 1.3% in 58615ed was.
  • The per-batch fixed cost is a large fraction. 1M nonces is ~2.7 ms of kernel; the submit/fence/map turnaround is hundreds of microseconds. The pipelining issues attack the turnaround; a bigger batch divides it.

Cost of a bigger batch

A running dispatch cannot be cancelled; the cancel flag is checked between batches only. On a new job the in-flight batch is wasted. That waste per job is one batch (two, once double-buffered):

batch 5090 at 368 MH/s wasted per new job
1M 2.7 ms negligible
16M 43 ms negligible
64M 174 ms ~1% at a 20 s block interval
256M 700 ms too much

Second cost: results[0] early-exit means a found solution stops the rest of the batch, but the worker still waits for the whole dispatch to retire. Latency from found to submitted grows with batch size. At 64M that is up to 174 ms added to seal propagation, which is a real orphan-risk cost on a fast chain. This is the actual constraint, not cancellation.

Design

  1. Immediately, in the deploy repo: --gpu-batch-size 16000000 on the 5090 host, 8000000 on the 4090. Measured, not guessed, once the harness exists.
  2. Auto mode in the miner. New flag value --gpu-batch-size auto. The engine measures its own MH/s over the first few batches (it already computes hash_rate for logging) and sizes subsequent batches to a target duration, --gpu-batch-target-ms, default 20 ms. On a 5090 that is ~7M; on a 3060 ~1M; on Apple ~1M. The stale-work and found-latency costs then stay bounded by wall-clock on every card, which is the right invariant, instead of by a nonce count that means 2.7 ms on one card and 100 ms on another.
  3. Clamp to [256k, 64M]. Keep the high-256-bit carry clamp that already exists.

Where: batch sizing lives in the worker's search_range loop. For engine-cuda it is ours. For engine-gpu it goes in the pipelined module from the double-buffering issue, so origin's run_single_batch still receives a plain batch_size: u32 and is untouched.

Measure

Harness sweep on one 5090: 1M, 4M, 16M, 64M, fixed workers = 1, pipelining off. Then the same with pipelining on to see how much of the batch-size gain pipelining already captures. Record found-to-submitted latency as well as MH/s; the sweet spot is where MH/s flattens, not the top.

Origin coupling

Flag plus a sizing function in our modules. Origin's default stays origin's.

Part of #1. Available today by flag, so the deployment repo (`lair/quantus`) can set it before any code changes. This issue is about making the miner pick well on its own. ## Current behaviour `DEFAULT_GPU_BATCH_SIZE = 1_000_000` in `crates/miner-cli/src/main.rs`. Origin's commit 58615ed measured micro-optimisations at 16M and saw gains that were noise at 1M, so they know larger batches help and kept 1M anyway, presumably for cancellation latency on small cards. Two things happen at 1M on a 5090 that do not happen at 16M: - **The tier hint does not bind.** `get_vendor_specific_dispatch` computes 10,922 workgroups for a Blackwell flagship (2.8M threads), but `run_single_batch` clamps logical threads to the batch, so 1M nonces means ~3,900 workgroups and exactly one nonce per thread. Every thread pays the per-thread loads (midstate, target, nonce base, ~44 storage reads) for one hash. At 16M the hint binds, each thread loops six nonces, and those loads amortise. That is what the 1.3% in 58615ed was. - **The per-batch fixed cost is a large fraction.** 1M nonces is ~2.7 ms of kernel; the submit/fence/map turnaround is hundreds of microseconds. The pipelining issues attack the turnaround; a bigger batch divides it. ## Cost of a bigger batch A running dispatch cannot be cancelled; the cancel flag is checked between batches only. On a new job the in-flight batch is wasted. That waste per job is one batch (two, once double-buffered): | batch | 5090 at 368 MH/s | wasted per new job | | --- | --- | --- | | 1M | 2.7 ms | negligible | | 16M | 43 ms | negligible | | 64M | 174 ms | ~1% at a 20 s block interval | | 256M | 700 ms | too much | Second cost: `results[0]` early-exit means a found solution stops the rest of the batch, but the worker still waits for the whole dispatch to retire. Latency from found to submitted grows with batch size. At 64M that is up to 174 ms added to seal propagation, which is a real orphan-risk cost on a fast chain. This is the actual constraint, not cancellation. ## Design 1. **Immediately, in the deploy repo:** `--gpu-batch-size 16000000` on the 5090 host, `8000000` on the 4090. Measured, not guessed, once the harness exists. 2. **Auto mode in the miner.** New flag value `--gpu-batch-size auto`. The engine measures its own MH/s over the first few batches (it already computes `hash_rate` for logging) and sizes subsequent batches to a target duration, `--gpu-batch-target-ms`, default 20 ms. On a 5090 that is ~7M; on a 3060 ~1M; on Apple ~1M. The stale-work and found-latency costs then stay bounded by wall-clock on every card, which is the right invariant, instead of by a nonce count that means 2.7 ms on one card and 100 ms on another. 3. Clamp to `[256k, 64M]`. Keep the high-256-bit carry clamp that already exists. Where: batch sizing lives in the worker's `search_range` loop. For `engine-cuda` it is ours. For `engine-gpu` it goes in the pipelined module from the double-buffering issue, so origin's `run_single_batch` still receives a plain `batch_size: u32` and is untouched. ## Measure Harness sweep on one 5090: 1M, 4M, 16M, 64M, fixed workers = 1, pipelining off. Then the same with pipelining on to see how much of the batch-size gain pipelining already captures. Record found-to-submitted latency as well as MH/s; the sweet spot is where MH/s flattens, not the top. ## Origin coupling Flag plus a sizing function in our modules. Origin's default stays origin's.
grenade added the needs-benchmarkneeds-chainperf labels 2026-09-03 09:10:19 +00:00
Author
Owner

Closing: settled by the sweep on the 4090 (#3, #17). Batch size only appeared to matter because the grid cap forced multiple nonces per thread above 1M; with the cap raised to keep one nonce per thread, 1M to 16M all measure 304 to 306 MH/s on the wgpu-era kernel and 1,109 to 1,121 on the 5090s with the tuned one. The default stays at 1M, which is the smallest stale cost (one batch per job switch, now about 2.4 ms on the 4090). The auto-sizing idea has nothing left to optimise.

Closing: settled by the sweep on the 4090 (#3, #17). Batch size only appeared to matter because the grid cap forced multiple nonces per thread above 1M; with the cap raised to keep one nonce per thread, 1M to 16M all measure 304 to 306 MH/s on the wgpu-era kernel and 1,109 to 1,121 on the 5090s with the tuned one. The default stays at 1M, which is the smallest stale cost (one batch per job switch, now about 2.4 ms on the 4090). The auto-sizing idea has nothing left to optimise.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: blackbeard/miner#6