nvidia-smi -pl survives neither a reboot nor a driver unload, and both
failures are silent -- the card returns to its default limit and nobody
notices until a connector runs warm. Hence a boot-time unit, which also
enables persistence mode so the limit is not lost when the last GPU
client detaches.
Per-host limits configured as host=watts in infra-setup.sh, clamped to
each card's own min/max. Verified: a 300W request on a 5090 whose floor
is 400W is corrected with an explicit log line rather than failing or
being ignored.
Measured on a 4090 mining Poseidon2:
450W (stock) 185.5 MH/s 70C 0.413 MH/s/W
300W 183.2 MH/s 61C 0.611
250W 180.3 MH/s 58C 0.721
44% less power and 12C cooler for 2.8% less hashrate -- this workload is
register-resident integer math and does not need the top of the clock
curve. It also cuts current through the 12VHPWR connector, which is the
one component on these cards with a documented melting failure mode.
Ordered Before=quantus-miner.service so a GPU is never briefly at its
default limit under full mining load.
subxt refuses plain ws:// to any host but localhost (InsecureUrl), so
quantus-cli against a remote node needs either an ssh tunnel or TLS.
This adds the vhost and an edge role in infra-setup.sh to mint the cert,
install it, arm step@quantus.timer and reload nginx.
Operator-run rather than CI-deployed, matching lair/mail: the runner's
rights never extend to reading certificate keys or reloading nginx on a
shared edge proxy.
Two things found while writing it. The vhost must listen on
127.0.0.1:14443 with proxy_protocol, not 443, because a stream SNI
router owns 443 on the edge proxies. And the sites-available copy of an
existing vhost is NOT a safe template: 11 of 19 enabled vhosts on
hanzalova are regular files rather than symlinks and have drifted from
sites-available since the SNI migration -- agent0's enabled copy listens
on 14443 while its sites-available copy still says 443. This vhost is
installed as a symlink so it cannot drift the same way.
Verified: nginx -t parses the config on the real host, failing only on
the not-yet-minted certificate.
The node crash-looped 7 times and was down ~100s, not mining:
Error: --rpc-external option shouldn't be used if the node is running
as a validator. Use `--unsafe-rpc-external` or `--rpc-methods=unsafe`
if you understand the risks.
Substrate rejects --rpc-external on a validator outright and exits,
whatever --rpc-methods says. --unsafe-rpc-external binds identically
('Same as --rpc-external' per --help) and differs only in demanding an
explicit acknowledgement. The safety still comes from --rpc-methods
safe, which is unchanged.
Restored on the host immediately; this makes CI converge on the same
thing rather than reverting it.
Also: validate opened with a bare , which exits 3
the moment a unit is 'activating' and under set -e aborted the step
before printing anything. A crash-looping unit sits in
activating/auto-restart forever, so a validator refusing its own flags
surfaced as an opaque 'exit code 3'. It now waits for a terminal state
and, on failure, reports NRestarts and the actual error lines from the
journal.
Run 11 failed with
sudo: a password is required
rsync error: error in rsync protocol data stream (code 12)
because quantus-node-rpc.xml was added to the deploy and to
infra-setup.sh's grants, but infra-setup had not been re-run against the
host. The failure landed forty lines into deploy-node, after several
files had already been written, and said nothing about the actual cause.
Each deploy job now compares the paths infra-setup.sh grants for its
role against sudo -n -l on the target, before touching anything. The two
cannot drift because the expected list is extracted from
infra-setup.sh itself rather than duplicated, so an asset added in
future is covered automatically.
Verified against the live host: it correctly reported
/etc/firewalld/services/quantus-node-rpc.xml as missing before
infra-setup was re-run, and reports all 26 paths permitted after.
Substrate binds JSON-RPC to loopback and makes opening it deliberately
awkward, which is correct for a validator. rpc_expose in the node matrix
takes the complete flag string; empty keeps it on loopback.
--rpc-methods safe is the load-bearing flag. It permits every state read
and author_submitExtrinsic (signed, so no key exposure) while blocking
author_rotateKeys, system_addReservedPeer and the rest. --rpc-methods
auto happens to downgrade once listening externally, but relying on that
makes the safety depend on a flag nobody wrote down.
The firewalld rich rule is scoped to the node's own /16, derived from
its own resolved address at deploy time -- so no subnet literal enters
this repo and the rule follows the host between sites.
Loopback is whitelisted from rate limiting because the arena exporter
runs beside the node and makes ~4 calls every 2s, which is 120/min --
enough to throttle itself against a naive limit.
validate now asserts the bind matches intent in both directions: nothing
listening while rpc_expose is set is a failure, and loopback-only is
reported as configuration rather than silence.
One node fans the same job out to every connected miner
(node/src/miner_server.rs holds a HashMap of MinerHandle and
broadcast_job's to all of them) and each miner picks its own random
starting nonce, so a second miner needs no coordination and no second
node. quadbrat joins as the test of that: RTX 3060, idle, gitea_ci
already present.
Separately, --chain was hardcoded in the unit while the workflow already
carried CHAIN in env for the credential path -- so switching chains
would have meant editing a static asset. With mainnet expected
2026-09-09 that is the wrong shape: the chain now comes from config, and
the switch is one line plus the version pins.
readme gains a mainnet-readiness section listing what the deploy cannot
decide: a mainnet reward wallet, the unpublished chain spec name, disk
for a second chains/ directory, and the GPU contention question.
Adds a Mining efficiency row: mining rounds started per minute, hashes
per mining round, and active jobs over time.
These are proxies, not the answer. Neither binary exports what the
question actually needs -- quantus-node has no difficulty metric and no
authored-block counter (substrate_proposer_block_constructed counts
proposals constructed, not rounds won), and quantus-miner has no
solutions-found counter. So authored-blocks-vs-expected-share cannot be
plotted, and a panel implying otherwise would be worse than none.
A text panel on the row states that plainly rather than leaving someone
to infer a conclusion the data does not support. lair/quantus#1 tracks
closing the gap -- a chain-query exporter counting blocks credited to
the reward address is the cheapest route -- and the experiments that
depend on it.
Adds a deploy-metrics job shipping a Prometheus scrape drop-in and a
provisioned Grafana dashboard to the fleet metrics host, plus the
exposure work the exporters needed to be reachable at all.
The node's exporter was loopback-only, so --prometheus-external. The
miner's already bound 0.0.0.0 unconditionally (crates/metrics builds the
listener from [0,0,0,0], no bind option) -- I had recorded it as loopback
in the port registry, which was wrong and is corrected. Both exporters
are unauthenticated, so each gets a firewalld service opened by a rich
rule scoped to the scrape host; that rule, not the bind, is the access
boundary.
Dashboard panels are built on metric names read off the running node and
the miner's source, not guessed. It leads with the states that look
healthy but are not: peers at zero, hash rate at zero while the unit is
active, GPU devices at zero, jobs never arriving.
Two shared-infra changes are in infra-setup.sh rather than the workflow,
since other projects depend on both files: prometheus.yml gains
scrape_config_files so projects stop sharing one file, and
grafana.container gains a dashboards mount it did not have.
The deploy restarted the node and miner unconditionally, and rsync's
default size+mtime check reported a change on every run anyway because
fetch re-downloads the binaries each time (verified: '<f..T......' on an
identical file, empty under -c).
So every run bounced a node that is 6% through a million-block sync,
dropping every peer and reopening RocksDB for no change at all.
push() now compares by checksum and itemizes; restart is gated on a real
content difference in the binary, unit or config. firewalld definitions
go through a separate helper since --reload picks them up and they are
never a reason to bounce the daemon. An unchanged but stopped unit is
still started.
This matters beyond tidiness: the workflow's validate mode exists to
check an untouched deployment, which it cannot honestly do while
restarting it first.
Opening 30333 on the host does not make a node dialable. Behind NAT
libp2p advertises the address it can see -- a 10.x mesh address -- so an
edge port forward delivers packets that no peer ever knows to send.
--public-addr is what closes that gap.
Plumbed as a per-host matrix field rather than a constant, since a
mesh-only node should not advertise anything. The unit takes it as a
bare $VAR so systemd word-splits it: a complete flag becomes two
arguments, empty becomes zero, and one unit serves both cases.
Validate now reports which state a node is in. An undialable validator
mines happily and reports active while contributing nothing back to the
network, which is exactly the class of silent failure the rest of the
validate step exists to catch.
Site names, host assignments and hardware inventories were scattered
through readme.md, CLAUDE.md and a unit-file comment. They belong in the
deploy matrices — the single source of infra truth — and nowhere else.
readme.md now describes roles and measured hardware classes rather than
which box is where; the GPU-contention section is written as a check to
perform rather than a named host. CLAUDE.md gains a standing rule so this
does not creep back.
The reward preimage is static and published in every block header, so two
nodes sharing one are publicly and permanently identifiable as the same
operator. That silently defeats the point of running nodes at separate
sites, so the inner_hash moves out of a single env: value and into the
deploy-node matrix, one per host, alongside the miners each node serves.
doc/wormhole-rewards.md records the derivation chain
(secret -> Poseidon2(salt|secret) = inner_hash -> Poseidon2(inner_hash) =
address) with source citations, and the analysis: publishing the
inner_hash is not a theft risk because spending needs a plonky2 proof of
knowledge of the secret, but it does make mining income permanently
attributable. The privacy in this design is at the exit, not at receipt,
which is not what MINING.md's wording suggests.
It also records the operational flaw: the inner_hash cannot be derived
from any public material and the Android wallet does not expose it, so a
seed phrase must reach some machine to produce a value that is public
anyway. Mitigation is a dedicated mining wallet, not carefulness.
The node embeds the reward preimage verbatim in the PreRuntime digest of
every block it authors (client/consensus/qpow/src/lib.rs:726), and the
mining-rewards pallet reads that digest back to derive the payout
address on-chain. It is public from the first mined block.
So the Gitea secret, the 0640 handling and the ps-visibility caveat were
all theatre over a value published on a blockchain. It moves to the
workflow env: block with the rest of the infra truth, and
infra-setup.sh loses --mint-reward-key entirely along with its pass and
Gitea-API dependencies.
This also answers why the node takes a preimage and not an address: the
runtime does the derivation, which is what proves the payout target is a
canonical wormhole address rather than an arbitrary account. An address
would give it nothing to verify.
Neither the node nor the miner needs a mnemonic. The node needs only the
32-byte preimage, from which it derives the reward address; the mining
protocol carries no reward address at all, so the miner cannot know it.
Deriving an inner_hash from a wallet is a one-time offline step and has
no business happening on the deploy workstation. infra-setup.sh now
accepts only the inner_hash, validates it is 0x + 64 hex, and refuses a
mnemonic outright. quantus-node itself refuses --words/--seed as argv
for the same reason: argv is world-readable and recorded in shell
history and audit logs.
This drops the seed phrase, the temporary binary download and the local
key derivation from the provisioning path entirely.
Node on bob, external GPU miner on benjy, split because
--miner-listen-port disables the node's built-in CPU mining.
The PoW is Poseidon2 over Goldilocks: register-resident, no memory
hardness, dominated by 64-bit modular multiplication. Measured on the
fleet, GPU beats CPU by ~200x (RTX 5090 368 MH/s @ 600W, RTX 4090
183 MH/s @ 450W, Ryzen 9 9950X3D 4.15 MH/s), so CPU mining is disabled
everywhere including on the GPU hosts.
The workflow deploys or validates. Validate mode asserts the hash
counter is advancing and the GPU is drawing power, because a miner that
enumerated no adapter and a node authoring nothing both look healthy to
systemctl is-active.
infra-setup.sh provisions the scoped gitea_ci accounts, generates the
node's libp2p key (the node refuses to start without one and it must
never be rotated by a deploy), and derives the reward preimage from a
seed phrase on stdin. It deliberately will not mint a fresh wallet: that
phrase is escalation-grade and fails agent-credentials.md §1.
Rob Thijssen
