The power limit and 810 MHz memory lock were a host setting, applied at
every boot by nvidia-power-limit.service regardless of what the GPU was
doing. A host switched from mining back to LLM inference kept the lock
with the miner stopped, and inference is memory-bandwidth bound: a 27B
model on 2x RTX 5090 decoded at 3.8 tok/s on a build that had measured
21.9 tok/s. Nothing reported it — a -lmc lock is not a clocks event
reason, and utilisation still reads ~98%.
The profile now follows the workload:
- nvidia-power-limit.sh takes a mode. `miner` applies the configured
profile as before; `default` resets each card to its own default power
limit and memory clock.
- nvidia-power-limit.service runs `default` at boot. The miner applies
`miner` in ExecStartPre and `default` in ExecStopPost (quantus/miner).
- A drop-in for neuron.service makes it conflict with the miner and
reset to default before starting, so either one can take the host with
a single systemctl start and the clocks follow.
- infra-setup.sh installs the drop-in, always writes the profile config
(so removing a host clears it), and applies whichever profile matches
the running workload instead of restarting the boot unit under a live
miner.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA
Mainnet launches 2026-09-10 09:09 UTC with the chain spec published as a
file before any binary embeds it. The node deploy can now run from
/etc/quantus-node/chainspec.json (CHAINSPEC_URL or the chainspec_url
input), asserts the started node's block-0 hash against EXPECTED_GENESIS
(required with a spec: it is the only defence against a look-alike), and
prints the spec id, which names <base-path>/chains/<id>/ and is what the
miner deploy must be pointed at; validate checks the miner credentials
exist under it. BOOTNODES are dialled and held as reserved peers.
The unit raises --out-peers to 32 and --in-peers to 64: bob sits at the
default cap of 40 peers today, and launch is a propagation race
(quantus/chain#1). infra-setup grants the runner the chain spec path
(applied to bob 2026-09-09).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
Three sources of author-label churn split the per-author hashrate series
into fragments (lair/quantus#11 follow-up):
- Attribution state lived in memory, so every restart stripped every
author of its name for a few blocks. Votes, held names and the peer-id
to name map are now saved to --state-file (the service uses
/var/lib/quantus-arena/attribution.json) at most every 10 s and
restored at start.
- A name dropped back to the preimage whenever confidence dipped under
0.6. A held name now stays until its node has no vote left in the
20-block window or another node out-votes it outright.
- A duplicate name got its peer-id suffix only once a second node with
that name was attributed, re-labelling the first. Any name used by more
than one node on the feed now always carries the suffix.
Dashboard: the hashrate-by-author panel no longer spans nulls, so a label
change shows as a gap rather than a slope drawn between two points.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
Author timestamps are set when the node builds the proposal, right after
the previous block arrives, so gaps between author timestamps measure
proposal timing and flagged our own honest node at 2.8x the network's
quick-follow share. Withheld blocks show in arrival timing instead:
quantus_block_arrival_gap_seconds uses the exporter's observation time
(poll interval now 1 s; telemetry first-seen replaces it under #11). The
leaderboard's quick-follow column and the Author timing panel now use it,
with a 2 s window to sit above the poll resolution.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
lair/quantus#4 step 2 (miner half). quantus/miner#12 deploys the miner from
that repo's main on every push, with its own validate and rollback, and owns
the miner role of infra-setup (deploy/infra-setup.sh there). Two deployers
for one host would flip the binary on every run, so this one stops.
- deploy.yaml: drop the deploy-miner job, the miner download in fetch, the
MINER_VERSION pin and miner_version input; deploy-metrics needs only
deploy-node. SCRAPE_MINERS stays and must agree with quantus/miner's
matrix; `validate metrics` still asserts every target is up.
- infra-setup.sh: drop the miner role. gpu, node, metrics, edge unchanged.
- asset/: remove the miner unit, sysusers, firewalld service and env
template (moved verbatim to quantus/miner deploy/).
- readme, CLAUDE: point at the new owner.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBgs2nSi4H2mdh8kD8vMX5
nvidia-smi -pl survives neither a reboot nor a driver unload, and both
failures are silent -- the card returns to its default limit and nobody
notices until a connector runs warm. Hence a boot-time unit, which also
enables persistence mode so the limit is not lost when the last GPU
client detaches.
Per-host limits configured as host=watts in infra-setup.sh, clamped to
each card's own min/max. Verified: a 300W request on a 5090 whose floor
is 400W is corrected with an explicit log line rather than failing or
being ignored.
Measured on a 4090 mining Poseidon2:
450W (stock) 185.5 MH/s 70C 0.413 MH/s/W
300W 183.2 MH/s 61C 0.611
250W 180.3 MH/s 58C 0.721
44% less power and 12C cooler for 2.8% less hashrate -- this workload is
register-resident integer math and does not need the top of the clock
curve. It also cuts current through the 12VHPWR connector, which is the
one component on these cards with a documented melting failure mode.
Ordered Before=quantus-miner.service so a GPU is never briefly at its
default limit under full mining load.
node-exporter already runs on every host, is already scraped (job
'node', targets from cichlid's inventory, labelled by host), and already
has the textfile collector enabled with a spool at
/var/lib/prometheus/node-exporter. So a textfile drop needs no new
exporter, no port allocation, no firewalld service and no scrape config
-- which is why it is one-time provisioning rather than a deploy job.
DCGM was the alternative and is the wrong tool: it targets datacenter
cards and its GeForce support is partial. nvidia-smi reports everything
wanted.
The script writes to a temp file and renames atomically, because the
collector reads whole files and a half-written one surfaces as
node_textfile_scrape_error. When nvidia-smi fails it emits an explicit
nvidia_gpu_scrape_ok 0 rather than leaving a stale file, so a broken
driver reads as broken instead of the graphs quietly flatlining.
Verified on the dual-GPU host: both cards reporting temperature, power,
utilisation, memory, clocks and fan through node-exporter with
node_textfile_scrape_error 0.
Fleet-general rather than quantus-specific -- it lives here because
there is no monitoring repo yet, and the dashboard says so.
The chain can only see miners that WIN blocks. A miner too small to take
a block in any given window is invisible in the authorship leaderboard
while still being connected and still burning electricity. Telemetry
sees every node, so the two series together answer a question neither
can alone: nodes flat while distinct authors falls means small miners
are being out-competed, not quitting.
Measured on the live feed: 32 nodes against 9-21 distinct authors, 229
aggregate CPU cores, 15 VMs, and five client versions in the wild --
half the network is still on 0.6.x/0.7.x/0.8.x, which is a fork risk at
a runtime upgrade and worth knowing before 2026-09-09.
python3-websockets is a SOFT dependency, packaged not pip: without it
the exporter serves every chain metric and reports
quantus_telemetry_up 0, so a missing package degrades rather than
breaks. infra-setup.sh installs it and treats failure as non-fatal.
The genesis hash is discovered from the node rather than configured --
one less value to get wrong when the chain changes for mainnet.
Substrate binds JSON-RPC to loopback and makes opening it deliberately
awkward, which is correct for a validator. rpc_expose in the node matrix
takes the complete flag string; empty keeps it on loopback.
--rpc-methods safe is the load-bearing flag. It permits every state read
and author_submitExtrinsic (signed, so no key exposure) while blocking
author_rotateKeys, system_addReservedPeer and the rest. --rpc-methods
auto happens to downgrade once listening externally, but relying on that
makes the safety depend on a flag nobody wrote down.
The firewalld rich rule is scoped to the node's own /16, derived from
its own resolved address at deploy time -- so no subnet literal enters
this repo and the rule follows the host between sites.
Loopback is whitelisted from rate limiting because the arena exporter
runs beside the node and makes ~4 calls every 2s, which is 120/min --
enough to throttle itself against a naive limit.
validate now asserts the bind matches intent in both directions: nothing
listening while rpc_expose is set is a failure, and loopback-only is
reported as configuration rather than silence.
quantus-node has no balance subcommand, and the faucet_getAccountInfo
RPC that MINING.md documents is absent from this build -- rpc_methods
returns only account_nextIndex and system_accountNextIndex. So the
exporter builds the System.Account storage key itself: twox128(System)
++ twox128(Account) ++ blake2_128_concat(account).
XXH64 is inlined because python3-xxhash is not installed on the node
hosts and this exporter stays dependency-free. It asserts the two
universally-known prefixes at import, so a broken implementation fails
loudly instead of silently querying the wrong key.
The reward address is Poseidon2(inner_hash). Rather than reimplement
Poseidon2 over Goldilocks, the deploy lifts it from the node's own
startup log; empty is tolerated and the exporter simply omits balance
metrics, which is what a genuinely first deploy will see.
Verified live: 158.153404340522 PLK, 12 decimals, zero RPC errors.
Also: str.replace fails silently on a missing anchor, which is how the
first attempt shipped an exporter calling a method that was never
inserted. Edits to this file now assert their anchors.
Closes the observability gap in #1 and builds the chain-scraper half of
#2. Every block header carries its miner's reward preimage in a
PreRuntime digest (engine id pow_), so authorship for EVERY miner on the
network is derivable from headers alone -- no indexer, no cooperation,
no event decoding. Difficulty comes from QPoWApi_get_difficulty via
state_call, verified working against the live node.
That makes expected share computable for the first time: our hashrate
over difficulty/interval, plotted against the fraction of blocks we
actually author. The gap between those two lines is orphan and
stale-work loss, which is the only honest test of whether tuning
anything helped.
Pure stdlib Python, deliberately: no pip, no venv, no build step, and it
deploys like every other asset here. Runs as its own service account
against the node's loopback RPC -- it has no business being able to
touch the validator's state or reward config.
Two things learned testing it live. The window is a BLOCK COUNT, not a
duration: a time window is meaningless while the node catches up, since
it ingests history at import speed -- the first version reported a 9ms
block interval and 102 GH/s. And per-preimage series are capped at top-N
plus our own, because an unbounded label set would be a cardinality bomb
if the network grows.
Prior art in Quantus-Network/qsafe.af derived miners from balance events
instead; the digest is lighter and unambiguous, and drops the GraphQL
indexer dependency entirely.
One node fans the same job out to every connected miner
(node/src/miner_server.rs holds a HashMap of MinerHandle and
broadcast_job's to all of them) and each miner picks its own random
starting nonce, so a second miner needs no coordination and no second
node. quadbrat joins as the test of that: RTX 3060, idle, gitea_ci
already present.
Separately, --chain was hardcoded in the unit while the workflow already
carried CHAIN in env for the credential path -- so switching chains
would have meant editing a static asset. With mainnet expected
2026-09-09 that is the wrong shape: the chain now comes from config, and
the switch is one line plus the version pins.
readme gains a mainnet-readiness section listing what the deploy cannot
decide: a mainnet reward wallet, the unpublished chain spec name, disk
for a second chains/ directory, and the GPU contention question.
Adds a deploy-metrics job shipping a Prometheus scrape drop-in and a
provisioned Grafana dashboard to the fleet metrics host, plus the
exposure work the exporters needed to be reachable at all.
The node's exporter was loopback-only, so --prometheus-external. The
miner's already bound 0.0.0.0 unconditionally (crates/metrics builds the
listener from [0,0,0,0], no bind option) -- I had recorded it as loopback
in the port registry, which was wrong and is corrected. Both exporters
are unauthenticated, so each gets a firewalld service opened by a rich
rule scoped to the scrape host; that rule, not the bind, is the access
boundary.
Dashboard panels are built on metric names read off the running node and
the miner's source, not guessed. It leads with the states that look
healthy but are not: peers at zero, hash rate at zero while the unit is
active, GPU devices at zero, jobs never arriving.
Two shared-infra changes are in infra-setup.sh rather than the workflow,
since other projects depend on both files: prometheus.yml gains
scrape_config_files so projects stop sharing one file, and
grafana.container gains a dashboards mount it did not have.
Opening 30333 on the host does not make a node dialable. Behind NAT
libp2p advertises the address it can see -- a 10.x mesh address -- so an
edge port forward delivers packets that no peer ever knows to send.
--public-addr is what closes that gap.
Plumbed as a per-host matrix field rather than a constant, since a
mesh-only node should not advertise anything. The unit takes it as a
bare $VAR so systemd word-splits it: a complete flag becomes two
arguments, empty becomes zero, and one unit serves both cases.
Validate now reports which state a node is in. An undialable validator
mines happily and reports active while contributing nothing back to the
network, which is exactly the class of silent failure the rest of the
validate step exists to catch.
Three bugs from run 2.
The real one: RestrictAddressFamilies omitted AF_NETLINK. libp2p calls
getifaddrs(), which opens a NETLINK_ROUTE socket, so the node started,
reported active, logged
failed to fetch network interfaces error=GetIfAddrsError("getifaddrs", -1)
litep2p started with no listen addresses, cannot accept inbound connections
and never bound --port. A silently undialable peer. Verified on the live
host: adding AF_NETLINK binds 30333 on both stacks, removing it unbinds.
The validate listener check caught this correctly — it was not a false
positive.
Two in validate itself: ss was invoked with no -t/-u (the protocol was
captured into $2 and only ever echoed), which errors with RTNETLINK
answers: Invalid argument; and 'awk ... exit' / 'grep -q' on a pipe
SIGPIPEs the writer, which pipefail reports as 141 — that was the exit
code that killed the sync check. Both now capture and parse via
here-string. Also fixed the same latent pattern in the getent lookup.
This change updates the `--name` parameter in the quantus-node systemd
service
to use "baba-gorchitsa" instead of the hostname. This ensures a
consistent
node identifier across deployments.
Site names, host assignments and hardware inventories were scattered
through readme.md, CLAUDE.md and a unit-file comment. They belong in the
deploy matrices — the single source of infra truth — and nowhere else.
readme.md now describes roles and measured hardware classes rather than
which box is where; the GPU-contention section is written as a check to
perform rather than a named host. CLAUDE.md gains a standing rule so this
does not creep back.
The node embeds the reward preimage verbatim in the PreRuntime digest of
every block it authors (client/consensus/qpow/src/lib.rs:726), and the
mining-rewards pallet reads that digest back to derive the payout
address on-chain. It is public from the first mined block.
So the Gitea secret, the 0640 handling and the ps-visibility caveat were
all theatre over a value published on a blockchain. It moves to the
workflow env: block with the rest of the infra truth, and
infra-setup.sh loses --mint-reward-key entirely along with its pass and
Gitea-API dependencies.
This also answers why the node takes a preimage and not an address: the
runtime does the derivation, which is what proves the payout target is a
canonical wormhole address rather than an arbitrary account. An address
would give it nothing to verify.
Node on bob, external GPU miner on benjy, split because
--miner-listen-port disables the node's built-in CPU mining.
The PoW is Poseidon2 over Goldilocks: register-resident, no memory
hardness, dominated by 64-bit modular multiplication. Measured on the
fleet, GPU beats CPU by ~200x (RTX 5090 368 MH/s @ 600W, RTX 4090
183 MH/s @ 450W, Ryzen 9 9950X3D 4.15 MH/s), so CPU mining is disabled
everywhere including on the GPU hosts.
The workflow deploys or validates. Validate mode asserts the hash
counter is advancing and the GPU is drawing power, because a miner that
enumerated no adapter and a node authoring nothing both look healthy to
systemctl is-active.
infra-setup.sh provisions the scoped gitea_ci accounts, generates the
node's libp2p key (the node refuses to start without one and it must
never be rotated by a deploy), and derives the reward preimage from a
seed phrase on stdin. It deliberately will not mint a fresh wallet: that
phrase is escalation-grade and fails agent-credentials.md §1.
