03: engine comparability

This commit is contained in:
rob thijssen
2025-09-07 21:34:44 +08:00
parent 85a4818c09
commit ea40fdf6e5
8 changed files with 955 additions and 24 deletions
@@ -0,0 +1,113 @@
# Prompt Summary (03): Accurate Metrics for Engine Comparisons, Logging Defaults, CLI Flag Rename, and Dashboards
## Context
To compare the performance of different mining engines (e.g., `cpu-baseline` vs `cpu-fast`) in Grafana using Prometheus metrics, we need precise per-job and per-thread hash counts. Currently, when a worker thread finds a solution, the engine reports only the `Candidate` without the associated `hash_count`. This causes undercounting in totals/rates, especially for the winning thread.
Additionally, we want a better out-of-the-box operator experience and evaluation tooling:
- Default the logging level to `info` when `RUST_LOG` is not set.
- Rename the CLI flag `--num-cores` to `--cores` but keep the internal field `num_cores`, and preserve `--num-cores` as an alias for backward compatibility.
- Provide two Grafana dashboards:
- A “Miner Overview” dashboard for operators, focusing on the fastest implementation.
- An “Engines Comparison” dashboard for performance evaluations across implementations.
We want to extend the engine’s result reporting so that the final “Found” event carries the `hash_count` accumulated by that thread. The miner-service will consume this information to update metrics correctly (global, per-job, per-thread), enabling accurate comparison between engines.
We also want sane defaults for logging, a consistent CLI, and packaged dashboards to support evaluation and production usage.
## Objectives
- Extend the engine result type to carry per-thread `hash_count` when a solution is found.
- Update both CPU engines (`cpu-baseline` and `cpu-fast`) to return the `hash_count` on `Found`.
- Update the miner-service to:
- Include `hash_count` from the `Found` status in totals and rate calculations.
- Maintain engine-aware, per-job, and per-thread metrics parity.
- Preserve the node-facing HTTP API and service behavior.
- Keep engine selection and metrics toggle behavior unchanged.
- Default RUST_LOG to `info` if not set, so operators get useful logs without extra env configuration.
- Rename CLI flag to `--cores` (keep `--num-cores` as an alias) while keeping the code field as `num_cores`.
- Add two Grafana dashboards under `docs/grafana/`:
- `miner-dashboard.json` for operators running the fastest engine.
- `engines-comparison-dashboard.json` for evaluating implementations.
## Requirements
- Engine API:
- Change `EngineStatus::Found(Candidate)` to carry `{ candidate: Candidate, hash_count: u64 }`.
- Maintain the existing `Exhausted { hash_count }` and `Cancelled { hash_count }` patterns for consistency.
- Engines:
- `BaselineCpuEngine` and `FastCpuEngine` must return accurate `hash_count` on `Found`.
- Service:
- In `update_from_results`, when a `ThreadResult` is built from an engine’s `Found`, carry the returned `hash_count` and accumulate it in:
- Global totals: `miner_hashes_total`
- Per-job totals: `miner_job_hashes_total{engine,job_id}`
- Per-thread totals: `miner_thread_hashes_total{engine,job_id,thread_id}`
- Ensure per-thread delta-rate and EMA smoothing logic still runs using the `hash_count` returned by `Found`.
- Metrics:
- No new metrics required; existing labeled counters/gauges should be used.
- Accuracy improves because the winning thread’s final work is now captured.
- Logging:
- If `RUST_LOG` is unset, default to `info` at startup.
- CLI:
- Rename `--num-cores` to `--cores`, keeping `--num-cores` as an alias and `num_cores` as the field name.
- Dashboards:
- Add `docs/grafana/miner-dashboard.json` (operator view).
- Add `docs/grafana/engines-comparison-dashboard.json` (evaluation view).
## Constraints and Compatibility
- The public HTTP API (endpoints/payloads) must remain unchanged.
- The engine abstraction is internal to the repo, so the type change is acceptable and should be updated wherever consumed (service only).
- Maintain compatibility between `cpu-baseline` and `cpu-fast` engines for apples-to-apples comparison.
- Logging default must not override an explicitly set `RUST_LOG`.
- CLI change must preserve backward compatibility (`--num-cores` alias).
- Dashboards must be optional artifacts (no runtime impact).
## Acceptance Criteria
- `EngineStatus::Found` carries `{ candidate, hash_count }`.
- Both engines report the correct `hash_count` when returning `Found`.
- Miner-service uses this `hash_count` to update:
- `miner_hashes_total`
- `miner_job_hashes_total{engine,job_id}`
- `miner_thread_hashes_total{engine,job_id,thread_id}`
- Per-thread delta-rate → `miner_thread_hash_rate{engine,job_id,thread_id}` (EMA-smoothed).
- Build success in release mode.
- No changes to node-facing HTTP behavior.
- Grafana can now accurately compare `cpu-baseline` vs `cpu-fast` with improved fidelity for the winning thread.
- When `RUST_LOG` is unset, logs default to `info`.
- `--cores` is available (with `--num-cores` as an alias) and maps to `num_cores`.
- Two dashboards exist under `docs/grafana/` for operators and engine evaluation.
## Deliverables
- Engine API update:
- Enum change for `EngineStatus::Found` to include `hash_count`.
- Engine implementations:
- Baseline and fast CPU engines updated to return `hash_count` on `Found`.
- Service updates:
- `mine_range_with_engine` and `update_from_results` modified to forward and accumulate the `hash_count` from `Found`.
- Metrics update hooks remain correct and comprehensive (global, per-job, per-thread, per-thread EMA rate).
- CLI/logging updates:
- Default `RUST_LOG` to `info` when unset.
- Expose `--cores` flag (keep `--num-cores` alias) while retaining `num_cores` as the code field.
- Dashboards:
- `docs/grafana/miner-dashboard.json` (operator-focused).
- `docs/grafana/engines-comparison-dashboard.json` (engine evaluation).
- Documentation:
- Capture this prompt and a separate response summary as iteration “03” in `docs/implementation/`.
## Out of Scope
- Additional metrics beyond those already implemented.
- Changes to the HTTP API or external protocol.
- Performance optimizations (e.g., Montgomery arithmetic) not directly related to the `hash_count` reporting tweak.
- Further CLI renames or logging customization (beyond defaults).
- Grafana provisioning/export automation.
## Notes
- This tweak corrects the main remaining accuracy gap for the winning thread in metrics.
- With this change, Grafana dashboards can make fair and accurate comparisons between different engines using the `engine` label, across global/job/thread-level telemetry.
- Default logging reduces operational friction; CLI alias retains backward compatibility during the transition.
- Packaged dashboards (overview and engines comparison) provide out-of-the-box visibility for operators and evaluators.
@@ -0,0 +1,89 @@
# Response Summary (03): Engine `Found` hash_count Included for Accurate Metrics, Logging/CLI Updates, and Dashboards
This document summarizes the implementation that augments the mining engine’s result reporting so that the final “Found” event carries the per-thread `hash_count`. This enables accurate global, per-job, and per-thread metrics (including rates), improving Grafana comparisons between engines (e.g., `cpu-baseline` vs `cpu-fast`).
## Overview
- Extended the engine result type so `Found` includes `{ candidate, hash_count }`.
- Updated both CPU engines to return the final `hash_count` on `Found`.
- Modified the service to forward and accumulate this count:
- Global totals (`miner_hashes_total`)
- Per-job totals (`miner_job_hashes_total{engine,job_id}`)
- Per-thread totals (`miner_thread_hashes_total{engine,job_id,thread_id}`)
- Per-thread hash-rate gauge via delta-rate with EMA smoothing
- No changes to the node-facing HTTP API.
## Changes Implemented
### 1) Engine API
- File: `crates/engine-cpu/src/lib.rs`
- Updated `EngineStatus`:
- Before: `Found(Candidate)`
- Now: `Found { candidate: Candidate, hash_count: u64 }`
- Other variants (`Exhausted`, `Cancelled`, `Running`) retain their `hash_count` shape for consistency.
### 2) Engines
- Files: `crates/engine-cpu/src/lib.rs`
- `BaselineCpuEngine`:
- On discovering a solution, returns `EngineStatus::Found { candidate, hash_count }` with the accumulated count for the current range.
- `FastCpuEngine`:
- Same behavior as baseline but using the incremental path (init once with `init_worker_y0`, then `step_mul` per nonce).
### 3) Service
- File: `crates/miner-service/src/lib.rs`
- `mine_range_with_engine`:
- Updated to extract `hash_count` from `EngineStatus::Found` and include it in the `ThreadResult`.
- `update_from_results`:
- Accumulates `hash_count` into:
- `total_hash_count`
- Global/engine-aware per-job/per-thread counters
- Per-thread delta-rate calculation with EMA smoothing now uses the `hash_count` from `Found` as well.
- All metric updates are engine-aware and job-/thread-labeled where appropriate.
### 4) Metrics
- No new metrics introduced; existing labeled series are now fully accurate for winning threads:
- Global:
- `miner_jobs_total{status}`
- `miner_hashes_total`
- `miner_hash_rate`
- Engine-/job-aware:
- `miner_job_hashes_total{engine,job_id}`
- `miner_job_hash_rate{engine,job_id}`
- `miner_job_status{engine,job_id,status}` (IntGauge)
- `miner_jobs_by_engine_total{engine,status}`
- Engine-/job-/thread-aware:
- `miner_thread_hashes_total{engine,job_id,thread_id}`
- `miner_thread_hash_rate{engine,job_id,thread_id}` (EMA-smoothed)
## Compatibility
- The public HTTP API (endpoints and payloads) remains unchanged.
- Engine abstraction change is internal to the workspace; service updated accordingly.
- Works with both `cpu-baseline` and `cpu-fast`, enabling apples-to-apples comparisons using the `engine` label in Grafana.
## Build and Run
- Build (release):
- `cargo build -p miner-cli --release`
- Run (baseline, no metrics):
- `cargo run -p miner-cli -- --port 9833 --engine cpu-baseline`
- Run (fast engine, with metrics):
- `cargo run -p miner-cli -- --port 9833 --engine cpu-fast --metrics-port 9900`
- Scrape: `http://localhost:9900/metrics`
## Verification
- Compile-time: Verified in release mode after updating the service to consume the new `Found` shape.
- Runtime: The winning thread’s final work is now counted in totals and informs the per-thread rate EMA, reducing undercounting and improving rate fidelity.
## Notes and Next Steps
- With winning-thread `hash_count` included, Grafana dashboards can fairly compare engines across global/job/thread metrics.
- Next logical optimization steps:
- Add correctness tests that compare baseline vs fast engine outcomes on small ranges (golden tests).
- Add benchmark harness to quantify nonces/sec improvements.
- Implement Montgomery multiplication in `pow-core` and switch `cpu-fast` to it for further speedups.