Three sources of author-label churn split the per-author hashrate series
into fragments (lair/quantus#11 follow-up):
- Attribution state lived in memory, so every restart stripped every
author of its name for a few blocks. Votes, held names and the peer-id
to name map are now saved to --state-file (the service uses
/var/lib/quantus-arena/attribution.json) at most every 10 s and
restored at start.
- A name dropped back to the preimage whenever confidence dipped under
0.6. A held name now stays until its node has no vote left in the
20-block window or another node out-votes it outright.
- A duplicate name got its peer-id suffix only once a second node with
that name was attributed, re-labelling the first. Any name used by more
than one node on the feed now always carries the suffix.
Dashboard: the hashrate-by-author panel no longer spans nulls, so a label
change shows as a gap rather than a slope drawn between two points.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
New interval variable 'author window' (auto: about a quarter of the
range, never under 2 h; or pinned 1h..2d) replaces the fixed 6 h trailing
window in the hashrate-by-author panel, so a week view smooths and a
short view resolves.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
After a restart the window started empty, so authors who had not sealed a
block since had no leaderboard row source, no author label series and no
timing caps until they authored again; on the dashboard that showed as
blank author cells for the smaller miners. The first poll now fills the
window (3600 blocks, 15 s against bob) from chain history, window only:
the authored counter is untouched so increase() sees no phantom burst, and
no tip timing is observed for blocks not watched arriving. Capped at 180 s
of wall clock.
The leaderboard's top-20 also drops rows with zero blocks in the range;
they were stale series with nothing to show.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
Each author's estimated hashrate over time: share of blocks authored in
the trailing 6 h times the 6 h average network hashrate, legend by the
telemetry node name where attributed, else the abbreviated preimage.
Authors under 10 MH/s are left out. Makes pool growth visible as a trend
rather than a leaderboard snapshot.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
The exporter emits quantus_author_node_info only for authors in its rolling
block window, which is rebuilt from live blocks after a restart, while the
leaderboard covers the dashboard range. Authors outside the window showed a
blank author cell. Query G now takes the current series and, for preimages
without one, the last label set seen in the range (one per preimage).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
Telemetry node ids are per connection: a node that restarts or renames
comes back under a new id, so votes keyed by node id split across the old
and new identity, and with a lifetime attempt count the new identity had to
outvote the whole history before the name came back. Votes are now keyed by
peer id (stable across restarts and renames; node id only when telemetry
gave no peer id), kept for the last 20 blocks per author, and displayed
with the latest name telemetry reported for that peer, so a rename shows
within a block and a restart within a few. Lifetime attempts/attributed
counters are unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
The exporter ingests the feed's ImportedBlock (code 6) reports and keeps,
per block hash, the first reporter and its lead over the second. Each tip
block is joined with that after 8 s; a reward preimage gets the node's
telemetry name once at least 3 of its blocks were attempted and 60% or more
were first reported by the same node. Emitted as
quantus_author_node_info{preimage,display,node_name,peer_id,self} confidence
with attempts/attributed counters per preimage; display is the name, or
the abbreviated preimage (first 8, last 4) when no confident attribution.
Measured today: leads are 40-100 ms for authors on the feed (the network is
better connected than when 200-700 ms was measured), so the lead threshold
is 20 ms and identity consistency carries the attribution. Validated for
ten minutes against bob: ours -> baba-gorchitsa, 0x2ccf39 -> quanpool-planck,
GUI miners -> QuantusMinerGUI at confidence 1.0.
Dashboard: the leaderboard joins quantus_author_node_info and shows the
display label as "author" in place of the reward preimage; the "ours" column
is gone, so the panel is fit to share. ARENA_DEBUG=1 turns on debug logging
(per-block attribution decisions).
Closes#11.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
Author timestamps are set when the node builds the proposal, right after
the previous block arrives, so gaps between author timestamps measure
proposal timing and flagged our own honest node at 2.8x the network's
quick-follow share. Withheld blocks show in arrival timing instead:
quantus_block_arrival_gap_seconds uses the exporter's observation time
(poll interval now 1 s; telemetry first-seen replaces it under #11). The
leaderboard's quick-follow column and the Author timing panel now use it,
with a 2 s window to sit above the poll resolution.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
For a Poisson block process the 10th-percentile gap of an honest author is
about one second by construction, so a fixed threshold flagged everyone.
quick-follow x is the share of an author's blocks arriving within 1 s of
the previous block divided by the network's share: 1 is normal, 2 or more
is withheld-block behaviour; needs 10 samples. Bucket boundaries are
selected as le="1.0" (Prometheus 3 normalises them to float strings).
Threshold bases are transparent so authors without samples show blank
cells, not red or blue.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
est. MH/s as a bar on its own scale; gap p10 and drift median as
threshold-coloured cells (low gap and positive drift are the suspicious
directions); a burst ratio column, consecutive self-blocks over the count
expected from the author's share, coloured amber above 2 and red above 4
and shown only where the expected count exceeds one.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
After the multi-query join Grafana ranged the gauge cell across every numeric
field in the frame, so 31 blocks was drawn against a 675 MH/s maximum.
fieldMinMax on the blocks column restores the per-column scale. The gap and
drift quantiles are filtered with expr == expr so authors without samples in
the range show a blank cell instead of NaN.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
Estimated hashrate is the author's share of blocks over the range times the
network hashrate estimate: statistical, roughly ±10% at a few hundred
blocks, and labelled as such. The timing columns join the new per-author
histograms (lair/quantus#12) on the preimage; outer join filtered to the
top-20 rows from the blocks query so authors without timing data still
appear with blanks.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
Two signals that expose what other miners are doing, both from data the
exporter already fetches plus one chain_getBlock per tip block:
- quantus_block_gap_seconds{preimage}: histogram of the time between a
block and the previous one, by the author of the later block, from the
authors' own timestamps (the Timestamp::set inherent, pallet 1 call 0,
decoded from the first extrinsic). Bursts in the lowest buckets for one
author right after others' blocks are withheld blocks.
- quantus_block_timestamp_drift_seconds{preimage}: block timestamp minus
the time the exporter first saw the block. Negative is propagation and
poll lag; positive is a future-dated block, which the Homestead-style
retarget rewards slightly (the inherent allows 30 s).
- quantus_consecutive_self_blocks_total{preimage}: blocks authored directly
after the same author's previous block.
Same top-N cap as the leaderboard, with "other" merged and "all" for the
network. Validated live against bob for three minutes. Dashboard row
"Author timing" with p10 gap, median drift and consecutive-block rate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
The miner never touches VRAM, so memory at full clock is watts taken from the
SMs inside the same power cap. Measured with the CUDA engine, miner paused:
4090 at 250 W 410 -> 463 MH/s (SM 1950 -> 2190 MHz); 3060 at 130 W 81 -> 83.
405 MHz drops the card into a low-power state (155 MH/s) and is refused.
Applied by nvidia-power-limit.service from the same config file
(NVIDIA_MEMORY_CLOCK_MHZ), configured per host in GPU_MEMORY_CLOCKS; hosts
without an entry are reset to the driver default. beast is not listed:
inference is memory-bound.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
quantus_blocks_authored is a rolling window by block count (3600 blocks)
served as a gauge, so the leaderboard panel ignored Grafana's time picker.
Add quantus_blocks_authored_total, cumulative since exporter start and not
capped to top N (a counter that comes and goes loses increments; the label
set is bounded by distinct miners seen), and query the panel as
increase() over $__range. The windowed gauge stays for the fixed view.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
node-exporter's host label is the short name (benjy), the first rule derived
the FQDN from the miner's instance, and quantus:device_hashrate_per_watt
produced no series. Verified against live labels on the metrics host.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
Monitoring side of quantus/miner#9. The miner now exports build identity,
per-device and per-job series (quantus/miner#14); this makes them readable.
- asset/prometheus/quantus-rules.yml: recording rules joining the miner's
series with the GPU series on a derived `host` and device index —
hashrate by device and commit, hashrate per watt, batch time off the GPU,
stale work, seal latency p95, idle waiting for the node.
- script/infra-setup.sh (metrics role): prometheus.yml gains `rule_files:`
for /etc/prometheus/rules.d, guarded like scrape_config_files; the dir;
a sudoers grant for the rules file.
- deploy.yaml deploy-metrics: pushes the rules (reload is already gated on
content change); validate asserts the quantus-miner rule group loaded.
- asset/grafana/quantus.json: "Performance by build" row, eight panels, and
a deploy annotation from changes in miner_build_info. No credentials
involved; Prometheus is the annotation source.
- asset/gpu/nvidia-textfile.sh (gpu role, operator-shipped): five
nvidia_gpu_event_reason_active{reason=...} gauges from nvidia-smi's clock
event reasons, so a regression that coincides with thermal or power-brake
capping can be told from a code regression.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
lair/quantus#4 step 2 (miner half). quantus/miner#12 deploys the miner from
that repo's main on every push, with its own validate and rollback, and owns
the miner role of infra-setup (deploy/infra-setup.sh there). Two deployers
for one host would flip the binary on every run, so this one stops.
- deploy.yaml: drop the deploy-miner job, the miner download in fetch, the
MINER_VERSION pin and miner_version input; deploy-metrics needs only
deploy-node. SCRAPE_MINERS stays and must agree with quantus/miner's
matrix; `validate metrics` still asserts every target is up.
- infra-setup.sh: drop the miner role. gpu, node, metrics, edge unchanged.
- asset/: remove the miner unit, sysusers, firewalld service and env
template (moved verbatim to quantus/miner deploy/).
- readme, CLAUDE: point at the new owner.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
The window is a block count capped at 3600, refills from empty after an
exporter restart, and is not the dashboard time range. Adds blocks-held
and wall-clock-elapsed stats beside the table, amber until half full,
and drops the constant 'component' column.
sum() strips labels while quantus_network_hashrate carries job and
instance, so the division matched nothing. Use scalar(), and take the
numerator from the hashes counter rather than the spiky gauge.
topk(...) with range=true returns one point per scrape, so every
timestamp became a bar and the miners smeared together. A leaderboard is
a snapshot: instant query, rendered as a sorted table with gauge cells,
our own row marked, and a total in the footer.
miner_hash_rate spikes to impossible values after a restart (1771 MH/s
on a 4090 that peaks near 145) because it is computed per job over a
tiny elapsed time. Hashrate panels now use rate(miner_hashes_total).
Adds node and miner restart annotations so version changes are visible
on every graph, a node version stat, a hash-rate-per-watt panel, and
puts discovered peers on its own axis so connected is legible.
nvidia-smi -pl survives neither a reboot nor a driver unload, and both
failures are silent -- the card returns to its default limit and nobody
notices until a connector runs warm. Hence a boot-time unit, which also
enables persistence mode so the limit is not lost when the last GPU
client detaches.
Per-host limits configured as host=watts in infra-setup.sh, clamped to
each card's own min/max. Verified: a 300W request on a 5090 whose floor
is 400W is corrected with an explicit log line rather than failing or
being ignored.
Measured on a 4090 mining Poseidon2:
450W (stock) 185.5 MH/s 70C 0.413 MH/s/W
300W 183.2 MH/s 61C 0.611
250W 180.3 MH/s 58C 0.721
44% less power and 12C cooler for 2.8% less hashrate -- this workload is
register-resident integer math and does not need the top of the clock
curve. It also cuts current through the 12VHPWR connector, which is the
one component on these cards with a documented melting failure mode.
Ordered Before=quantus-miner.service so a GPU is never briefly at its
default limit under full mining load.
Adds a Now row of gauge panels at the top of the GPU dashboard,
mirroring node-exporter's CPU Busy: threshold markers, amber then red,
one gauge per GPU series rather than one aggregate.
GPU Busy (utilisation), Temperature with amber at 80C and red at 87C
where consumer NVIDIA cards start thermal throttling, and Power as a
fraction of the card's enforced limit rather than raw watts -- a 4090
at 450W and a 5090 at 600W are both fully loaded, and only the ratio
says so on one scale.
Labels are shortened to host + gpu index for the gauges; the timeseries
below keep the full name.
node-exporter already runs on every host, is already scraped (job
'node', targets from cichlid's inventory, labelled by host), and already
has the textfile collector enabled with a spool at
/var/lib/prometheus/node-exporter. So a textfile drop needs no new
exporter, no port allocation, no firewalld service and no scrape config
-- which is why it is one-time provisioning rather than a deploy job.
DCGM was the alternative and is the wrong tool: it targets datacenter
cards and its GeForce support is partial. nvidia-smi reports everything
wanted.
The script writes to a temp file and renames atomically, because the
collector reads whole files and a half-written one surfaces as
node_textfile_scrape_error. When nvidia-smi fails it emits an explicit
nvidia_gpu_scrape_ok 0 rather than leaving a stale file, so a broken
driver reads as broken instead of the graphs quietly flatlining.
Verified on the dual-GPU host: both cards reporting temperature, power,
utilisation, memory, clocks and fan through node-exporter with
node_textfile_scrape_error 0.
Fleet-general rather than quantus-specific -- it lives here because
there is no monitoring repo yet, and the dashboard says so.
opened MINUS closed per direction, plotted against the caps. Occupancy
is the measurement that matters and cumulative opens actively mislead:
inbound churns constantly (170 opened, 156 closed) while outbound is
deliberately maintained (36 opened, 27 closed), so a high inbound open
count says nothing about whether its cap is full. Reading the ratio of
cumulative opens is how issue #1 came to recommend the wrong knob.
Measured now: outbound 9 held against a cap of 8 -- saturated -- while
inbound sits at 14 of 32 and the node has discovered 300 peers. So
--out-peers binds and --in-peers does not, the reverse of what #1 said.
No exporter change needed; both series are already scraped from the
node. Adds the discovered-peers count and rounds-per-minute alongside,
since those are the other two the experiment compares.
The chain can only see miners that WIN blocks. A miner too small to take
a block in any given window is invisible in the authorship leaderboard
while still being connected and still burning electricity. Telemetry
sees every node, so the two series together answer a question neither
can alone: nodes flat while distinct authors falls means small miners
are being out-competed, not quitting.
Measured on the live feed: 32 nodes against 9-21 distinct authors, 229
aggregate CPU cores, 15 VMs, and five client versions in the wild --
half the network is still on 0.6.x/0.7.x/0.8.x, which is a fork risk at
a runtime upgrade and worth knowing before 2026-09-09.
python3-websockets is a SOFT dependency, packaged not pip: without it
the exporter serves every chain metric and reports
quantus_telemetry_up 0, so a missing package degrades rather than
breaks. infra-setup.sh installs it and treats failure as non-fatal.
The genesis hash is discovered from the node rather than configured --
one less value to get wrong when the chain changes for mainnet.
subxt refuses plain ws:// to any host but localhost (InsecureUrl), so
quantus-cli against a remote node needs either an ssh tunnel or TLS.
This adds the vhost and an edge role in infra-setup.sh to mint the cert,
install it, arm step@quantus.timer and reload nginx.
Operator-run rather than CI-deployed, matching lair/mail: the runner's
rights never extend to reading certificate keys or reloading nginx on a
shared edge proxy.
Two things found while writing it. The vhost must listen on
127.0.0.1:14443 with proxy_protocol, not 443, because a stream SNI
router owns 443 on the edge proxies. And the sites-available copy of an
existing vhost is NOT a safe template: 11 of 19 enabled vhosts on
hanzalova are regular files rather than symlinks and have drifted from
sites-available since the SNI migration -- agent0's enabled copy listens
on 14443 while its sites-available copy still says 443. This vhost is
installed as a symlink so it cannot drift the same way.
Verified: nginx -t parses the config on the real host, failing only on
the not-yet-minted certificate.
The node crash-looped 7 times and was down ~100s, not mining:
Error: --rpc-external option shouldn't be used if the node is running
as a validator. Use `--unsafe-rpc-external` or `--rpc-methods=unsafe`
if you understand the risks.
Substrate rejects --rpc-external on a validator outright and exits,
whatever --rpc-methods says. --unsafe-rpc-external binds identically
('Same as --rpc-external' per --help) and differs only in demanding an
explicit acknowledgement. The safety still comes from --rpc-methods
safe, which is unchanged.
Restored on the host immediately; this makes CI converge on the same
thing rather than reverting it.
Also: validate opened with a bare , which exits 3
the moment a unit is 'activating' and under set -e aborted the step
before printing anything. A crash-looping unit sits in
activating/auto-restart forever, so a validator refusing its own flags
surfaced as an opaque 'exit code 3'. It now waits for a terminal state
and, on failure, reports NRestarts and the actual error lines from the
journal.
