The dashboard JSON was deployed but its provider config never was, so
Grafana had no reason to read the directory -- the dashboard would have
been a file on disk and an empty folder in the UI.
Tracked separately from the dashboard because Grafana reads providers
only at container startup while dashboards are re-read on the
provisioning interval: a provider change restarts grafana, a dashboard
change does not.
validate now queries Grafana's search API for the dashboard uid rather
than testing that a file exists, since a provisioned dashboard that
fails to parse leaves the JSON looking perfectly fine on disk.
Adds a deploy-metrics job shipping a Prometheus scrape drop-in and a
provisioned Grafana dashboard to the fleet metrics host, plus the
exposure work the exporters needed to be reachable at all.
The node's exporter was loopback-only, so --prometheus-external. The
miner's already bound 0.0.0.0 unconditionally (crates/metrics builds the
listener from [0,0,0,0], no bind option) -- I had recorded it as loopback
in the port registry, which was wrong and is corrected. Both exporters
are unauthenticated, so each gets a firewalld service opened by a rich
rule scoped to the scrape host; that rule, not the bind, is the access
boundary.
Dashboard panels are built on metric names read off the running node and
the miner's source, not guessed. It leads with the states that look
healthy but are not: peers at zero, hash rate at zero while the unit is
active, GPU devices at zero, jobs never arriving.
Two shared-infra changes are in infra-setup.sh rather than the workflow,
since other projects depend on both files: prometheus.yml gains
scrape_config_files so projects stop sharing one file, and
grafana.container gains a dashboards mount it did not have.
The deploy restarted the node and miner unconditionally, and rsync's
default size+mtime check reported a change on every run anyway because
fetch re-downloads the binaries each time (verified: '<f..T......' on an
identical file, empty under -c).
So every run bounced a node that is 6% through a million-block sync,
dropping every peer and reopening RocksDB for no change at all.
push() now compares by checksum and itemizes; restart is gated on a real
content difference in the binary, unit or config. firewalld definitions
go through a separate helper since --reload picks them up and they are
never a reason to bounce the daemon. An unchanged but stopped unit is
still started.
This matters beyond tidiness: the workflow's validate mode exists to
check an untouched deployment, which it cannot honestly do while
restarting it first.
quantus-miner --node-addr parses as a Rust SocketAddr -- an IP and a
port, with no DNS resolution -- so passing the node's hostname failed at
argument parsing:
error: invalid value 'bob.hanzalova.internal:9833' for
'--node-addr <NODE_ADDR>': invalid socket address syntax
Resolve on the miner host, which is the one dialling and therefore has
the authoritative view, and guard that the result is a mesh address.
Keeps the literal out of the repo, same as the firewall rich rule.
Verified on the host: the hostname is rejected at parse, an IP gets
through to the next check.
Run 3's validate ran ~3s after systemctl restart, while the journal was
still at 'Using litep2p network backend'. The node had not yet opened
tcp/30333, so a healthy node starting normally was reported as not
listening -- and the failure skipped deploy-miner entirely.
Bounded poll (120s) for the P2P socket before the listener assertions,
rather than a guessed sleep. Verified out of band that the node does
bind and is dialable from outside the site.
systemctl restart returns on exec, not on ready; the node's startup is
dominated by opening RocksDB, so this gap is wide and reproducible.
/dns4/ rather than the literal WAN address so an IP change needs no
redeploy, matching the per-site indirection convention in public-dns.md.
Paired with a 'bob' host alias on the site router (10.6.0.193, created
via opn-cli to match the existing hanzalova/oci/quadbrat convention).
The inbound 30333/tcp forward that makes both meaningful is still
outstanding and has to be a console change: opn-cli exposes firewall
alias and rule but no NAT subcommand.
Opening 30333 on the host does not make a node dialable. Behind NAT
libp2p advertises the address it can see -- a 10.x mesh address -- so an
edge port forward delivers packets that no peer ever knows to send.
--public-addr is what closes that gap.
Plumbed as a per-host matrix field rather than a constant, since a
mesh-only node should not advertise anything. The unit takes it as a
bare $VAR so systemd word-splits it: a complete flag becomes two
arguments, empty becomes zero, and one unit serves both cases.
Validate now reports which state a node is in. An undialable validator
mines happily and reports active while contributing nothing back to the
network, which is exactly the class of silent failure the rest of the
validate step exists to catch.
Three bugs from run 2.
The real one: RestrictAddressFamilies omitted AF_NETLINK. libp2p calls
getifaddrs(), which opens a NETLINK_ROUTE socket, so the node started,
reported active, logged
failed to fetch network interfaces error=GetIfAddrsError("getifaddrs", -1)
litep2p started with no listen addresses, cannot accept inbound connections
and never bound --port. A silently undialable peer. Verified on the live
host: adding AF_NETLINK binds 30333 on both stacks, removing it unbinds.
The validate listener check caught this correctly — it was not a false
positive.
Two in validate itself: ss was invoked with no -t/-u (the protocol was
captured into $2 and only ever echoed), which errors with RTNETLINK
answers: Invalid argument; and 'awk ... exit' / 'grep -q' on a pipe
SIGPIPEs the writer, which pipefail reports as 141 — that was the exit
code that killed the sync check. Both now capture and parse via
here-string. Also fixed the same latent pattern in the getent lookup.
run() is `ssh ... "$@"`, and ssh concatenates its argument vector with
spaces for the remote shell to re-split, so local quoting is lost. The
rich rule arrived as a dozen bare words:
firewall-cmd: error: unrecognized arguments: family=ipv4 source
address=10.6.0.146/32 service name=quantus-node-miner accept
Pass the three rich-rule invocations as one pre-quoted string each.
Verified against the real host as gitea_ci: the unquoted form reproduces
the CI error, the quoted form returns 'no' — so it parses and the scoped
sudoers rule matches it.
Every other command in the deploy survived only because no other
argument contains a space; CLAUDE.md now records the trap.
This change updates the `--name` parameter in the quantus-node systemd
service
to use "baba-gorchitsa" instead of the hostname. This ensures a
consistent
node identifier across deployments.
Update the inner_hash value for the node host bob.hanzalova.internal to
a new value derived from its own wallet. This ensures the node uses the
correct preimage for reward calculations and maintains proper
cryptographic integrity.
Site names, host assignments and hardware inventories were scattered
through readme.md, CLAUDE.md and a unit-file comment. They belong in the
deploy matrices — the single source of infra truth — and nowhere else.
readme.md now describes roles and measured hardware classes rather than
which box is where; the GPU-contention section is written as a check to
perform rather than a named host. CLAUDE.md gains a standing rule so this
does not creep back.
The reward preimage is static and published in every block header, so two
nodes sharing one are publicly and permanently identifiable as the same
operator. That silently defeats the point of running nodes at separate
sites, so the inner_hash moves out of a single env: value and into the
deploy-node matrix, one per host, alongside the miners each node serves.
doc/wormhole-rewards.md records the derivation chain
(secret -> Poseidon2(salt|secret) = inner_hash -> Poseidon2(inner_hash) =
address) with source citations, and the analysis: publishing the
inner_hash is not a theft risk because spending needs a plonky2 proof of
knowledge of the secret, but it does make mining income permanently
attributable. The privacy in this design is at the exit, not at receipt,
which is not what MINING.md's wording suggests.
It also records the operational flaw: the inner_hash cannot be derived
from any public material and the Android wallet does not expose it, so a
seed phrase must reach some machine to produce a value that is public
anyway. Mitigation is a dedicated mining wallet, not carefulness.
The node embeds the reward preimage verbatim in the PreRuntime digest of
every block it authors (client/consensus/qpow/src/lib.rs:726), and the
mining-rewards pallet reads that digest back to derive the payout
address on-chain. It is public from the first mined block.
So the Gitea secret, the 0640 handling and the ps-visibility caveat were
all theatre over a value published on a blockchain. It moves to the
workflow env: block with the rest of the infra truth, and
infra-setup.sh loses --mint-reward-key entirely along with its pass and
Gitea-API dependencies.
This also answers why the node takes a preimage and not an address: the
runtime does the derivation, which is what proves the payout target is a
canonical wormhole address rather than an arbitrary account. An address
would give it nothing to verify.
Neither the node nor the miner needs a mnemonic. The node needs only the
32-byte preimage, from which it derives the reward address; the mining
protocol carries no reward address at all, so the miner cannot know it.
Deriving an inner_hash from a wallet is a one-time offline step and has
no business happening on the deploy workstation. infra-setup.sh now
accepts only the inner_hash, validates it is 0x + 64 hex, and refuses a
mnemonic outright. quantus-node itself refuses --words/--seed as argv
for the same reason: argv is world-readable and recorded in shell
history and audit logs.
This drops the seed phrase, the temporary binary download and the local
key derivation from the provisioning path entirely.
Node on bob, external GPU miner on benjy, split because
--miner-listen-port disables the node's built-in CPU mining.
The PoW is Poseidon2 over Goldilocks: register-resident, no memory
hardness, dominated by 64-bit modular multiplication. Measured on the
fleet, GPU beats CPU by ~200x (RTX 5090 368 MH/s @ 600W, RTX 4090
183 MH/s @ 450W, Ryzen 9 9950X3D 4.15 MH/s), so CPU mining is disabled
everywhere including on the GPU hosts.
The workflow deploys or validates. Validate mode asserts the hash
counter is advancing and the GPU is drawing power, because a miner that
enumerated no adapter and a node authoring nothing both look healthy to
systemctl is-active.
infra-setup.sh provisions the scoped gitea_ci accounts, generates the
node's libp2p key (the node refuses to start without one and it must
never be rotated by a deploy), and derives the reward preimage from a
seed phrase on stdin. It deliberately will not mint a fresh wallet: that
phrase is escalation-grade and fails agent-credentials.md §1.
Rob Thijssen
