03: engine comparability
This commit is contained in:
@@ -0,0 +1,113 @@
|
||||
# Prompt Summary (03): Accurate Metrics for Engine Comparisons, Logging Defaults, CLI Flag Rename, and Dashboards
|
||||
|
||||
## Context
|
||||
|
||||
To compare the performance of different mining engines (e.g., `cpu-baseline` vs `cpu-fast`) in Grafana using Prometheus metrics, we need precise per-job and per-thread hash counts. Currently, when a worker thread finds a solution, the engine reports only the `Candidate` without the associated `hash_count`. This causes undercounting in totals/rates, especially for the winning thread.
|
||||
|
||||
Additionally, we want a better out-of-the-box operator experience and evaluation tooling:
|
||||
- Default the logging level to `info` when `RUST_LOG` is not set.
|
||||
- Rename the CLI flag `--num-cores` to `--cores` but keep the internal field `num_cores`, and preserve `--num-cores` as an alias for backward compatibility.
|
||||
- Provide two Grafana dashboards:
|
||||
- A “Miner Overview” dashboard for operators, focusing on the fastest implementation.
|
||||
- An “Engines Comparison” dashboard for performance evaluations across implementations.
|
||||
|
||||
We want to extend the engine’s result reporting so that the final “Found” event carries the `hash_count` accumulated by that thread. The miner-service will consume this information to update metrics correctly (global, per-job, per-thread), enabling accurate comparison between engines.
|
||||
|
||||
We also want sane defaults for logging, a consistent CLI, and packaged dashboards to support evaluation and production usage.
|
||||
|
||||
## Objectives
|
||||
|
||||
- Extend the engine result type to carry per-thread `hash_count` when a solution is found.
|
||||
- Update both CPU engines (`cpu-baseline` and `cpu-fast`) to return the `hash_count` on `Found`.
|
||||
- Update the miner-service to:
|
||||
- Include `hash_count` from the `Found` status in totals and rate calculations.
|
||||
- Maintain engine-aware, per-job, and per-thread metrics parity.
|
||||
- Preserve the node-facing HTTP API and service behavior.
|
||||
- Keep engine selection and metrics toggle behavior unchanged.
|
||||
- Default RUST_LOG to `info` if not set, so operators get useful logs without extra env configuration.
|
||||
- Rename CLI flag to `--cores` (keep `--num-cores` as an alias) while keeping the code field as `num_cores`.
|
||||
- Add two Grafana dashboards under `docs/grafana/`:
|
||||
- `miner-dashboard.json` for operators running the fastest engine.
|
||||
- `engines-comparison-dashboard.json` for evaluating implementations.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Engine API:
|
||||
- Change `EngineStatus::Found(Candidate)` to carry `{ candidate: Candidate, hash_count: u64 }`.
|
||||
- Maintain the existing `Exhausted { hash_count }` and `Cancelled { hash_count }` patterns for consistency.
|
||||
- Engines:
|
||||
- `BaselineCpuEngine` and `FastCpuEngine` must return accurate `hash_count` on `Found`.
|
||||
- Service:
|
||||
- In `update_from_results`, when a `ThreadResult` is built from an engine’s `Found`, carry the returned `hash_count` and accumulate it in:
|
||||
- Global totals: `miner_hashes_total`
|
||||
- Per-job totals: `miner_job_hashes_total{engine,job_id}`
|
||||
- Per-thread totals: `miner_thread_hashes_total{engine,job_id,thread_id}`
|
||||
- Ensure per-thread delta-rate and EMA smoothing logic still runs using the `hash_count` returned by `Found`.
|
||||
- Metrics:
|
||||
- No new metrics required; existing labeled counters/gauges should be used.
|
||||
- Accuracy improves because the winning thread’s final work is now captured.
|
||||
- Logging:
|
||||
- If `RUST_LOG` is unset, default to `info` at startup.
|
||||
- CLI:
|
||||
- Rename `--num-cores` to `--cores`, keeping `--num-cores` as an alias and `num_cores` as the field name.
|
||||
- Dashboards:
|
||||
- Add `docs/grafana/miner-dashboard.json` (operator view).
|
||||
- Add `docs/grafana/engines-comparison-dashboard.json` (evaluation view).
|
||||
|
||||
## Constraints and Compatibility
|
||||
|
||||
- The public HTTP API (endpoints/payloads) must remain unchanged.
|
||||
- The engine abstraction is internal to the repo, so the type change is acceptable and should be updated wherever consumed (service only).
|
||||
- Maintain compatibility between `cpu-baseline` and `cpu-fast` engines for apples-to-apples comparison.
|
||||
- Logging default must not override an explicitly set `RUST_LOG`.
|
||||
- CLI change must preserve backward compatibility (`--num-cores` alias).
|
||||
- Dashboards must be optional artifacts (no runtime impact).
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- `EngineStatus::Found` carries `{ candidate, hash_count }`.
|
||||
- Both engines report the correct `hash_count` when returning `Found`.
|
||||
- Miner-service uses this `hash_count` to update:
|
||||
- `miner_hashes_total`
|
||||
- `miner_job_hashes_total{engine,job_id}`
|
||||
- `miner_thread_hashes_total{engine,job_id,thread_id}`
|
||||
- Per-thread delta-rate → `miner_thread_hash_rate{engine,job_id,thread_id}` (EMA-smoothed).
|
||||
- Build success in release mode.
|
||||
- No changes to node-facing HTTP behavior.
|
||||
- Grafana can now accurately compare `cpu-baseline` vs `cpu-fast` with improved fidelity for the winning thread.
|
||||
- When `RUST_LOG` is unset, logs default to `info`.
|
||||
- `--cores` is available (with `--num-cores` as an alias) and maps to `num_cores`.
|
||||
- Two dashboards exist under `docs/grafana/` for operators and engine evaluation.
|
||||
|
||||
## Deliverables
|
||||
|
||||
- Engine API update:
|
||||
- Enum change for `EngineStatus::Found` to include `hash_count`.
|
||||
- Engine implementations:
|
||||
- Baseline and fast CPU engines updated to return `hash_count` on `Found`.
|
||||
- Service updates:
|
||||
- `mine_range_with_engine` and `update_from_results` modified to forward and accumulate the `hash_count` from `Found`.
|
||||
- Metrics update hooks remain correct and comprehensive (global, per-job, per-thread, per-thread EMA rate).
|
||||
- CLI/logging updates:
|
||||
- Default `RUST_LOG` to `info` when unset.
|
||||
- Expose `--cores` flag (keep `--num-cores` alias) while retaining `num_cores` as the code field.
|
||||
- Dashboards:
|
||||
- `docs/grafana/miner-dashboard.json` (operator-focused).
|
||||
- `docs/grafana/engines-comparison-dashboard.json` (engine evaluation).
|
||||
- Documentation:
|
||||
- Capture this prompt and a separate response summary as iteration “03” in `docs/implementation/`.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Additional metrics beyond those already implemented.
|
||||
- Changes to the HTTP API or external protocol.
|
||||
- Performance optimizations (e.g., Montgomery arithmetic) not directly related to the `hash_count` reporting tweak.
|
||||
- Further CLI renames or logging customization (beyond defaults).
|
||||
- Grafana provisioning/export automation.
|
||||
|
||||
## Notes
|
||||
|
||||
- This tweak corrects the main remaining accuracy gap for the winning thread in metrics.
|
||||
- With this change, Grafana dashboards can make fair and accurate comparisons between different engines using the `engine` label, across global/job/thread-level telemetry.
|
||||
- Default logging reduces operational friction; CLI alias retains backward compatibility during the transition.
|
||||
- Packaged dashboards (overview and engines comparison) provide out-of-the-box visibility for operators and evaluators.
|
||||
@@ -0,0 +1,89 @@
|
||||
# Response Summary (03): Engine `Found` hash_count Included for Accurate Metrics, Logging/CLI Updates, and Dashboards
|
||||
|
||||
This document summarizes the implementation that augments the mining engine’s result reporting so that the final “Found” event carries the per-thread `hash_count`. This enables accurate global, per-job, and per-thread metrics (including rates), improving Grafana comparisons between engines (e.g., `cpu-baseline` vs `cpu-fast`).
|
||||
|
||||
## Overview
|
||||
|
||||
- Extended the engine result type so `Found` includes `{ candidate, hash_count }`.
|
||||
- Updated both CPU engines to return the final `hash_count` on `Found`.
|
||||
- Modified the service to forward and accumulate this count:
|
||||
- Global totals (`miner_hashes_total`)
|
||||
- Per-job totals (`miner_job_hashes_total{engine,job_id}`)
|
||||
- Per-thread totals (`miner_thread_hashes_total{engine,job_id,thread_id}`)
|
||||
- Per-thread hash-rate gauge via delta-rate with EMA smoothing
|
||||
- No changes to the node-facing HTTP API.
|
||||
|
||||
## Changes Implemented
|
||||
|
||||
### 1) Engine API
|
||||
|
||||
- File: `crates/engine-cpu/src/lib.rs`
|
||||
- Updated `EngineStatus`:
|
||||
- Before: `Found(Candidate)`
|
||||
- Now: `Found { candidate: Candidate, hash_count: u64 }`
|
||||
- Other variants (`Exhausted`, `Cancelled`, `Running`) retain their `hash_count` shape for consistency.
|
||||
|
||||
### 2) Engines
|
||||
|
||||
- Files: `crates/engine-cpu/src/lib.rs`
|
||||
- `BaselineCpuEngine`:
|
||||
- On discovering a solution, returns `EngineStatus::Found { candidate, hash_count }` with the accumulated count for the current range.
|
||||
- `FastCpuEngine`:
|
||||
- Same behavior as baseline but using the incremental path (init once with `init_worker_y0`, then `step_mul` per nonce).
|
||||
|
||||
### 3) Service
|
||||
|
||||
- File: `crates/miner-service/src/lib.rs`
|
||||
- `mine_range_with_engine`:
|
||||
- Updated to extract `hash_count` from `EngineStatus::Found` and include it in the `ThreadResult`.
|
||||
- `update_from_results`:
|
||||
- Accumulates `hash_count` into:
|
||||
- `total_hash_count`
|
||||
- Global/engine-aware per-job/per-thread counters
|
||||
- Per-thread delta-rate calculation with EMA smoothing now uses the `hash_count` from `Found` as well.
|
||||
- All metric updates are engine-aware and job-/thread-labeled where appropriate.
|
||||
|
||||
### 4) Metrics
|
||||
|
||||
- No new metrics introduced; existing labeled series are now fully accurate for winning threads:
|
||||
- Global:
|
||||
- `miner_jobs_total{status}`
|
||||
- `miner_hashes_total`
|
||||
- `miner_hash_rate`
|
||||
- Engine-/job-aware:
|
||||
- `miner_job_hashes_total{engine,job_id}`
|
||||
- `miner_job_hash_rate{engine,job_id}`
|
||||
- `miner_job_status{engine,job_id,status}` (IntGauge)
|
||||
- `miner_jobs_by_engine_total{engine,status}`
|
||||
- Engine-/job-/thread-aware:
|
||||
- `miner_thread_hashes_total{engine,job_id,thread_id}`
|
||||
- `miner_thread_hash_rate{engine,job_id,thread_id}` (EMA-smoothed)
|
||||
|
||||
## Compatibility
|
||||
|
||||
- The public HTTP API (endpoints and payloads) remains unchanged.
|
||||
- Engine abstraction change is internal to the workspace; service updated accordingly.
|
||||
- Works with both `cpu-baseline` and `cpu-fast`, enabling apples-to-apples comparisons using the `engine` label in Grafana.
|
||||
|
||||
## Build and Run
|
||||
|
||||
- Build (release):
|
||||
- `cargo build -p miner-cli --release`
|
||||
- Run (baseline, no metrics):
|
||||
- `cargo run -p miner-cli -- --port 9833 --engine cpu-baseline`
|
||||
- Run (fast engine, with metrics):
|
||||
- `cargo run -p miner-cli -- --port 9833 --engine cpu-fast --metrics-port 9900`
|
||||
- Scrape: `http://localhost:9900/metrics`
|
||||
|
||||
## Verification
|
||||
|
||||
- Compile-time: Verified in release mode after updating the service to consume the new `Found` shape.
|
||||
- Runtime: The winning thread’s final work is now counted in totals and informs the per-thread rate EMA, reducing undercounting and improving rate fidelity.
|
||||
|
||||
## Notes and Next Steps
|
||||
|
||||
- With winning-thread `hash_count` included, Grafana dashboards can fairly compare engines across global/job/thread metrics.
|
||||
- Next logical optimization steps:
|
||||
- Add correctness tests that compare baseline vs fast engine outcomes on small ranges (golden tests).
|
||||
- Add benchmark harness to quantify nonces/sec improvements.
|
||||
- Implement Montgomery multiplication in `pow-core` and switch `cpu-fast` to it for further speedups.
|
||||
Reference in New Issue
Block a user