miner_stale_hashes_total was recorded in the QUIC loop for every result whose
job id no longer matched, which is every cancelled search: 74% of all hashes
on benjy flagged as stale on the first hour of data. That is the loop's
notion of a stale *result*, not wasted work; the hashes were done while the
job was current.
The wasted work is the batch that completes after the job was superseded.
Record that in the engine at the cancellation check, per device and kernel,
and drop the loop-level accounting. Expect one batch per job switch.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
quantus/miner#9. Additive metrics in a lair-owned module of the metrics
crate; origin's metrics are untouched so the fleet dashboard keeps working
across origin merges.
Identity:
- miner_build_info{version, commit}: the join key for everything below.
- miner_config_info{engine, gpu_batch_size, gpu_devices, cpu_workers,
gpu_throttle_ms}: two deploys of one commit with different flags are
different experiments.
Per device (handles resolved once at engine init, no label lookups in the
batch path):
- miner_device_hashes_total{device, kernel}, miner_device_solutions_total,
miner_device_lost_total.
- miner_gpu_batch_seconds{device, kernel, phase=gpu|host}: submit-to-mapped
on the device versus everything else in the batch. This is the bubble
#4, #5 and #6 attack, measured directly.
Jobs and results:
- miner_jobs_received_total, miner_stale_hashes_total{engine},
miner_job_pickup_seconds{engine} (issued to picked up; the cancel latency
of a busy worker), miner_results_submitted_total,
miner_results_send_failed_total, miner_seal_latency_seconds (found to
sent), miner_job_idle_seconds_total (sent to next job, node-attributable).
Connection: miner_connects_total, miner_connect_failures_total,
miner_disconnects_total, miner_connected, miner_disconnected_seconds_total.
Origin-owned code touched, each block marked lair: engine-gpu gains a
metrics dependency, a DeviceMetrics handle on GpuContext and timing points
in run_single_batch; miner-service gains found_at on WorkerResult,
created_at on MiningJob and the recording calls; miner-cli sets build info
at startup. Deploy validate now asserts miner_build_info carries the
deployed commit.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
* Select Apple Metal 4.0.1 kernel; keep 4.0.0 u64 elsewhere
NVIDIA was ~37% slower on the 4.0.1 Metal-tuned kernel. Split
Poseidon2 kernels and pick by backend:
- Metal + SHADER_INT64: v4.0.1 u64
- other SHADER_INT64: v4.0.0 u64
- no SHADER_INT64: 32-bit fallback (same as 4.0.0)
CI uploads the Linux release binary as an artifact so we can
bench NVIDIA without building on the GPU box.
* Drop version numbers from GPU kernel labels
Log native-u64 vs native-u64 Apple Metal without tying
the strings to release tags.
Apple M5 Pro, CLI benchmark: 24.0 -> 43.2 MH/s. Kernel output is
unchanged and bit-exact against the CPU reference.
Field arithmetic:
- Deferred-carry additions (Acc): the external layer and the internal
row sum accumulate unreduced with a carry counter and fold once per
output instead of two epsilon corrections per add.
- gf64_mul_add folds the row sum into the 128-bit product before a
single plain reduction.
- mul_wide / gf64_sqr assemble the 128-bit product from 32-bit-valued
partial sums with no carry compares.
Kernel structure (code size dominates on Apple's compiler; every
unrolled variant measured slower):
- One loop drives all 30 rounds so each layer is emitted once, with
round constants added by the linear layer preceding each S-box
(RC_EXT table, RC_INTERNAL padded with a trailing zero).
- One S-box site with a runtime lane count (12 external, 1 internal).
- mining_main runs pad / first squeeze / second squeeze through a
single inlined permute64 call in a phase loop.
Measured and rejected: uniform buffers and unhoisted loads, workgroup
sizes 64/128/512, 8-add MDS, one-compare reduce, fully unrolled
layers, two nonces per thread.
Investigating reported seal submission failures. Root cause analysis:
the miner's job-id staleness chain (worker check, quic-loop filter,
node-side job_id compare) cannot mislabel a candidate — internal and
node job ids are updated atomically together. The remaining benign race
is node-side: a block-template rebuild (new pre_hash, same parent) does
not bump the node's job counter, so an in-flight seal mined on the old
template passes the job-id check and fails seal verification, logging
"Failed to submit seal". Fixing that belongs in the chain repo (bump the
job counter on template rebuild).
Miner-side hardening in this commit:
- gpu_cpu_parity is now a two-phase fuzzer. Phase 1 dumps FULL 512-bit
hashes for randomized (header, nonce) pairs via the real
midstate-resume datapath and verifies each against the canonical CPU
implementation, multi-threaded (defaults to 1M hashes, ~5s; 10M runs
in ~15s). Phase 2 fuzzes whole mining jobs with five profiles: random,
carry-edge starts (saturated low limbs), 2^256-boundary crossings,
impossible difficulty with exact hash-count assertion, and CPU-known
solutions to catch false negatives. Seeded and reproducible; also
asserts the submitted work bytes match the nonce.
Verified clean: 10M bulk hashes + ~1,500 seals across 6 seeds.
- miner-service re-verifies every candidate on the CPU (one hash) before
sending JobResult; an engine-produced invalid seal is never submitted
and is logged loudly instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Exploring a fully native Metal port of the mining kernel showed it runs
~16% faster than the wgpu path at equal batch size. Decomposing that gap:
~9% is naga's injected runtime bounds checks and forced loop bounding,
~1.5% is mulhi()-based multiplication (inexpressible in WGSL), <1% is
the constant address space; the rest is binding/codegen residue.
The dominant term is recoverable inside wgpu on every backend:
the engine now builds its pipelines with create_shader_module_trusted +
ShaderRuntimeChecks::unchecked(). This is sound for our shaders: sources
are compiled into the binary, every buffer access is a constant-bounded
loop index into fixed-size bindings the engine allocates itself, all
loops have static bounds, and both variants remain covered by the
dual-shader component suite and the CPU parity example.
Apple M4: 11.39 -> 12.34 MH/s at the default 1M batch (+8.3%), 11.74 ->
12.73 MH/s at 8M (+8.4%). Cumulative vs main: 4.4x.
New examples/trusted_hashrate.rs measures checked vs trusted on the same
shader and CPU-verifies results.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Evaluated all ideas from #80 against the native-u64 kernel. Its three
main wins (lazy field arithmetic, precomputed sponge state, conditional
second squeeze) were already present in stronger form; two micro-opts
carried over and measured positive at large batches (+1.3% at 16M,
noise-level at the 1M default):
- Hoist midstate/target/start-nonce storage reads out of the nonce loop.
- Produce byte-swapped hash words on demand during the target compare,
so the reject path skips building the swapped hash entirely.
Forcing nonces_per_thread=2 was also tested and regressed; one thread
per nonce stays.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three stacked optimizations, each verified bit-exact against the CPU
reference (qp-poseidon-core) by the dual-shader component suite and a
new GPU/CPU parity example:
- mining_u64.wgsl: native 64-bit Goldilocks arithmetic with plonky2-style
lazy reduction, used automatically when the adapter supports
SHADER_INT64 (all Apple Silicon, NVIDIA, modern AMD). mining.wgsl
remains the 32-bit fallback. (+50%)
- Midstate precompute: the first two of five sponge permutations absorb
the header and high nonce half, both constant per batch; they are now
computed once on the CPU (pow_core::mining_midstate) and resumed on
the GPU. Batches are clamped so nonces never carry past 2^256. (+77%)
- Lazy second squeeze: the first squeeze yields the most significant 256
bits of the hash, which decide hash-vs-target unless exactly equal to
the target's high half, so the common reject path skips the final
permutation. (+56%)
Apple M4 throughput: 2.81 -> 11.4 MH/s at the default 1M batch
(criterion large_range_1m/gpu: 355.8ms -> 86.3ms, -75.7%).
Also:
- Fix criterion GPU bench crash: thread-local worker resources are now
tagged with an engine id so multiple GpuEngines per process never mix
devices.
- Fix stale end-to-end test harness (6-binding layout) and run the full
component suite against both shader variants.
- New examples: hashrate, gpu_cpu_parity (25 jobs incl. 2^256-boundary
crossing), gpu_features.
- Add Apple M5 family GPU tiers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* handle lost device correctly
* Update lib.rs
* allow-integrated flag otherwise exclude
* DeviceLost is a status
* Integrated GPU skipped when discrete init fails
* nits
On Windows wgpu enumerates each physical GPU once per backend (Vulkan +
DX12) plus a CPU-emulated fallback ("Microsoft Basic Render Driver").
Building a mining context for every entry causes VRAM contention and
OOMs the process during benchmark/serve startup (#61).
Instead of deduplicating by (vendor, device) PCI IDs - which would
collapse rigs with multiple identical cards into a single context - drop
CPU-emulated adapters and keep all adapters from the highest-ranked
backend present (Vulkan/Metal, then DX12). Within a single backend each
physical GPU appears exactly once, so identical cards are preserved by
construction and no physical-ID matching is needed.
Selected adapters are ordered discrete-first so `--gpu-devices 1` picks
the discrete card on hybrid laptops. Skipped adapters are logged at info
level; if nothing usable remains, init fails with an explicit error.
Selection logic is a pure index-based function unit-tested against the
exact enumeration reported in #61, identical multi-GPU rigs, DX12-only
machines, and software-only environments.
Co-authored-by: illuzen <illuzen@users.noreply.github.com>
* support AMD GPUs on windows
* add support for more GPUs
* remove shader dump from logs
* handle gpu initialization failure better
* fmt
* re-order matching to avoid mobile / desktop false positives
* harden gpu matching
* add tests to engine-gpu module
* handle dead device and failed mapping correctly
* handle timeout properly with tokio
* nits
* fix benchmarks
* fmt
* improve logging
* don't let the queue spill, just drain it
* move cancellation out of the shader, since it doesn't work
* simplify batch size mechanism, improve logs
* fmt
* address review items
* single hash, new poseidon
* fix build
* new poseidon api
* fmt
* fix race condition with stale jobs being reported
* fmt
* fix deps
* format toml
remove stale comments
---------
Co-authored-by: Nikolaus Heger <nheger@gmail.com>
* implement quic
* simpler names
* simplify worker logic
* simplify gpu configuration logic
* miner initiates and picks random start nonce
* remove mining service
* clean up logs and remove unused engines
* clean up metrics
* metrics fix
* shared worker pool, cpu driven cancellation
just 10x the gpu thru-put, no more batch-size on gpu
* fmt
* non-local git
* taplo derp
* fix the benches
* handle stale results correctly in gpu
* more accurate gpu hashrate
* fmt
* remove unused
* handle duplicate solutions, cli controls gpu cancel frequency
* put plonky2 back
* demote noisy log
