quantus/readme.md

# quantus

Deployment for a [Quantus Network](https://quantus.com) Planck node and its
external GPU miner.

Two components:

| Component | Role |
| --- | --- |
| `quantus-node` | consensus, P2P, reward address. Disk + network bound. |
| `quantus-miner` | Poseidon2-over-Goldilocks search. Pure GPU compute. |

**Scope:** which hosts run what lives in the `deploy-node` / `deploy-miner`
matrices in `.gitea/workflows/deploy.yaml` and nowhere else — that is the
convention (`deployment-gitea-actions.md`) and it keeps the fleet's shape in one
place instead of scattered through prose. Don't restate host assignments,
hardware inventories, sites or addressing in this file.

**One node serves many miners.** `node/src/miner_server.rs` keeps a `HashMap` of
connected miners and `broadcast_job`s the same work to all of them; each miner
picks its own random starting nonce, so there is no coordination or range
allocation. A node per miner would only be needed for a separate reward address.

They are split deliberately. Setting `--miner-listen-port` on the node
**disables its built-in CPU mining**, so the node authors nothing unless an
external miner is connected — the split is not optional once that flag is set.
The miner connects *out* to the node over QUIC/9833 and listens on nothing but a
loopback metrics port.

Conventions: `~/git/architecture` — `generic.md` for the on-host shape,
`deployment-gitea-actions.md` for the deploy mechanism. Deviations are listed
below.

## Why GPU, and why only these hosts

The proof-of-work is Poseidon2 over the Goldilocks field (`p = 2⁶⁴ − 2³² + 1`),
width 12, 4 external + 22 internal rounds — an arithmetic hash, not a bit-mixing
one. Cost is dominated by 64-bit modular multiplication, the state lives entirely
in registers, and there is no memory hardness at all. Measured on this fleet:

| Hardware | Hashrate | Power | Efficiency |
| --- | --- | --- | --- |
| RTX 5090 | 368 MH/s | 600 W | 0.60 MH/s/W |
| RTX 5090 @ 400 W floor | 281 MH/s | 400 W | 0.70 MH/s/W |
| RTX 4090 | 183 MH/s | 450 W | 0.41 MH/s/W |
| current-gen desktop CPU, 32t | ~4 MH/s | — | — |
| older server CPU, 12t | ~0.5 MH/s | — | — |

Measured on representative hardware, not estimated. A single 5090 at its minimum
power limit out-hashes a couple of dozen server CPUs combined. CPU mining is
therefore disabled everywhere (`--cpu-workers 0`), including on GPU hosts, where
it would add ~1% hashrate for a couple hundred watts. Older server CPUs are
node-capable and mining-worthless.

GPU/CPU parity is verified upstream (`engine-gpu`'s `gpu_cpu_parity` example,
25/25 jobs) — the GPU is doing real work, not advancing a lying counter.

## Ports

Registered in `~/git/architecture/port-allocations.md` §5.

| Port | Proto | Component | Exposure |
| --- | --- | --- | --- |
| `30333` | tcp | node P2P (libp2p) | **public** — named firewalld service, plus an inbound forward at the site edge |
| `9833` | udp | node ← miner control channel | mesh, firewalld **rich rule** scoped to the miner host only |
| `9615` | tcp | node Prometheus | loopback |
| `9900` | tcp | miner Prometheus | loopback |

**Deviation from `port-allocations.md` §3:** these are upstream protocol
defaults, not numbers derived from the service name. `30333` is the chain's
well-known P2P port and changing it buys nothing; `9833` is the miner protocol's
default and both ends must agree. They are recorded in the registry, which is
what the doc says actually prevents collisions. Neither lands in the crowded
alt-HTTP cluster.

The miner link gets a rich rule rather than a plain `--add-service` because the
fleet uses a single default zone (`generic.md` §9): adding the service outright
would publish `9833/udp` on every address the host carries, and upstream is
explicit that this port must never face the internet.

## GPU power limits

Applied at boot by `nvidia-power-limit.service`, configured per host in
`infra-setup.sh`'s `GPU_POWER_LIMITS` as `host=watts`, and **clamped to each
card's own floor and ceiling** — a request the hardware refuses is corrected with
a log line rather than silently ignored.

This is not a compromise. Measured on a 4090 mining Poseidon2:

| limit | hashrate | temp | efficiency |
| --- | --- | --- | --- |
| 450 W (stock) | 185.5 MH/s | 70 °C | 0.413 MH/s/W |
| 300 W | 183.2 MH/s | 61 °C | 0.611 MH/s/W |
| 250 W | 180.3 MH/s | 58 °C | **0.721 MH/s/W** |

**44% less power and 12 °C cooler for 2.8% less hashrate.** The workload is
register-resident integer math and does not need the top of the clock curve, so
the last few hundred MHz were very nearly pure heat. It also cuts current through
the 12VHPWR connector, the one component on 4090/5090 cards with a documented
melting failure mode.

A boot-time unit is required because `nvidia-smi -pl` survives neither a reboot
nor a driver unload. The unit also enables persistence mode, which stops the
limit being lost when the last GPU client detaches — a silent reversion that
would otherwise show up only as a warm connector. The dashboard's **GPU Power**
gauge reads draw against the *enforced* limit, so a reverted cap appears as the
gauge dropping from ~100% to ~60% rather than as nothing at all.

## Monitoring

Scraped by the fleet Prometheus and visualised in Grafana (folder **Quantus**).
Both exporters are **unauthenticated**, and the miner's binds `0.0.0.0`
unconditionally with no loopback option, so what bounds access in each case is a
firewalld rich rule scoped to the scrape host — not the bind address.

| Asset | Deployed to |
| --- | --- |
| `asset/prometheus/quantus.yml.tmpl` | `/etc/prometheus/scrape_configs.d/quantus.yml` |
| `asset/grafana/quantus.json` | `/etc/grafana/provisioning/dashboards/quantus/` |

Scrape targets are rendered from `SCRAPE_NODES` / `SCRAPE_MINERS` in the workflow
`env:`, which must agree with the deploy matrices. `validate metrics` asserts
Prometheus holds every `quantus-*` target and that all are `up`, so drift fails
the run rather than quietly producing an empty graph.

Two pieces of **shared** infrastructure needed one-time changes, which is why
they live in `infra-setup.sh --metrics-hosts` rather than the workflow:
`prometheus.yml` gained `scrape_config_files:` so each project can own a drop-in
instead of every project editing one file, and `grafana.container` gained a
dashboards bind mount — it previously mounted only the datasource. Both are
additive, guarded, and back up what they touch.

**What the dashboard cannot tell you:** neither binary exports a difficulty
metric or an authored-block counter, and the miner exports no solutions-found
counter — so "do we win the share of blocks our hashrate predicts?", the only
honest measure of orphan and stale-work loss, is not plottable today. The
"Mining efficiency" row shows the measurable proxies and says so explicitly.
Closing that gap, and the experiments that depend on it, are tracked in
[issue #1](https://git.lair.cafe/lair/quantus/issues/1).

The dashboard is provisioned with `allowUiUpdates: false`: edit the JSON here and
redeploy, so a browser tweak cannot silently diverge from source.

## Mainnet readiness

Planck is a **testnet**. Mainnet is expected **2026-09-09**, and this repo exists
so that switching is a config change rather than a scramble:

| To switch | Change |
| --- | --- |
| Chain | `CHAIN:` in the workflow `env:` — the unit reads it from config, and the miner credential path `<base-path>/chains/<chain>/` follows from the same value |
| Versions | `NODE_VERSION` / `MINER_VERSION` pins |
| Reward address | `inner_hash` per node in the matrix |

Still to decide before launch day, none of which the deploy can settle for you:

- **A mainnet reward wallet.** The current `inner_hash` was derived for testnet.
  A dedicated mining wallet is the recommendation either way
  (`doc/wormhole-rewards.md` §6).
- **The mainnet chain spec name**, which is not yet published.
- **Whether a fresh base-path is wanted.** Switching `CHAIN` creates a new
  `chains/<chain>/` directory and leaves the old one on disk; the testnet chain
  reached ~1M blocks, so check free space rather than assume.
- **GPU contention** on any host also serving inference.

Sync at launch is trivial — a new chain starts at genesis — so the long testnet
sync is not a preview of mainnet day.

## `wss://quantus.internal`

`subxt` — and therefore `quantus-cli` — refuses plain `ws://` to any host but
localhost (`InsecureUrl`), so without TLS every client needs an ssh tunnel:

```sh
ssh -f -N -L 19944:127.0.0.1:9944 <node>
quantus balance --node-url ws://127.0.0.1:19944 --address <addr>
```

The tunnel-free answer is a vhost on the site's edge proxy, per
`architecture/reverse-proxies.md` and `internal-tls.md`:

```sh
./script/infra-setup.sh --pubkey ~/.ssh/id_gitea_ci.pub   # includes the edge role
```

It mints an internal-CA cert for `quantus.internal`, installs
`asset/nginx/quantus.internal.conf`, arms `step@quantus.timer` for renewal, and
reloads nginx. **Operator-run, never CI-deployed** — the runner has no rights to
read certificate keys or reload nginx on a shared edge proxy.

Split-horizon DNS is a separate manual step (the record lives in the site
router's Unbound, not here):

```sh
opn-cli --config ~/.opn-cli/<site>.yml unbound host create \
  --hostname quantus --domain internal --server <edge mesh ip>
```

Two traps worth knowing:

- **`listen 127.0.0.1:14443 ssl proxy_protocol;`, not `listen 443`.** A stream
  SNI router owns TCP 443 on the edge proxies and hands non-passthrough names to
  the local https tier. Copy the `listen` line from a currently **enabled**
  vhost — on hanzalova, 11 of 19 enabled vhosts are regular files that have
  drifted from their `sites-available` counterparts since that migration, so the
  `sites-available` copy can be stale and wrong.
- The RPC behind this is **unauthenticated**. `--rpc-methods safe` blocks the
  dangerous calls, but anyone reaching the name reads all state and can submit
  signed extrinsics. `.internal` is mesh-only and must never get a public record.

## Reaching the JSON-RPC from other machines

Loopback by default, which is the right posture for a validator. Opening it
needs three things, and the flags are the part people forget:

```
--unsafe-rpc-external                           bind beyond loopback (see below)
--rpc-methods safe                              NOT auto, NOT unsafe
--rpc-cors all                                  only if browsers will call it
--rpc-rate-limit 300
--rpc-rate-limit-whitelisted-ips 127.0.0.1/32   or the local exporter throttles itself
```

Set them as a complete string in the node matrix's `rpc_expose`; empty leaves it
on loopback. The deploy then opens `9944` with a firewalld rich rule scoped to
**the node's own /16, derived from its own address at deploy time** — so no
subnet literal lives in this repo and the rule follows the host between sites.

**`--rpc-external` does not work on a validator.** Substrate refuses it and
exits — whatever `--rpc-methods` says — demanding `--unsafe-rpc-external` or
`--rpc-methods=unsafe` as an explicit acknowledgement. The two binding flags are
identical ("Same as `--rpc-external`" per `--help`); only the name differs. So
the correct pairing is the alarming-looking `--unsafe-rpc-external` **with**
`--rpc-methods safe`, which binds externally while still refusing every unsafe
method.

`safe` is the load-bearing one. It still permits every state read and
`author_submitExtrinsic` (signed, so no key exposure), while blocking
`author_rotateKeys`, `system_addReservedPeer` and the rest of the unsafe set.
`--rpc-methods auto` happens to downgrade to `safe` once listening externally,
but relying on that means the safety depends on a flag nobody wrote down. Never
`unsafe` on a validator.

The rate-limit whitelist matters more than it looks: the arena exporter runs on
the node host and makes about four RPC calls every two seconds, which is 120 a
minute — enough to throttle itself against a naive limit.

`9944` is HTTP **and** WebSocket on the same port, so `ws://<node>:9944` and
`http://<node>:9944` both work. It must never be reachable from the WAN.

## Being dialable

Opening `30333` on the host is only one of three things a node needs to accept
inbound peers:

1. **firewalld on the host** — the deploy does this.
2. **An inbound TCP forward at the site edge** to the node host. Not managed
   here; `opn-cli` exposes `firewall alias` and `firewall rule` but **no NAT
   subcommand**, so a port forward is a console change.
3. **`public_addr` in the deploy matrix.** Behind NAT, libp2p advertises the
   address it can see — a mesh address no external peer can dial. A forward gets
   packets in but never tells anyone to send them. Prefer a `/dns4/` site
   indirection name over a literal address so a WAN IP change needs no redeploy.

Leave `public_addr` empty for a deliberately mesh-only node; the unit takes it as
a bare `$VAR` that systemd word-splits, so empty yields no flag. Validate reports
which of the two states a node is in, because an undialable validator still mines
and still looks healthy — it just takes from the network without adding to it.

## Secrets, and one thing that only looks like one

| Name | What | Where |
| --- | --- | --- |
| `RSYNC_SSH_KEY` | the shared fleet runner key | Gitea secret — already exists, do **not** rotate |
| `inner_hash` | 32-byte wormhole preimage, **one per node host** | workflow `deploy-node` matrix — **public, not a secret** |

**The inner_hash is not secret.** The node embeds it verbatim in the
`PreRuntime` digest of every block it authors
(`client/consensus/qpow/src/lib.rs:726`), and the `mining-rewards` pallet reads
that digest back and derives the payout address on-chain. It is published from
your first mined block onward, so it is configuration, and it lives in the
`deploy-node` matrix with the other infra truth.

**One inner_hash per host — never share one.** The value is static (derived per
wallet, not per block), so two nodes carrying the same inner_hash are publicly
and permanently identifiable as the same operator, which collapses the
independence that running nodes at separate sites is supposed to provide. The
matrix carries one per entry for that reason.

Publishing it is not a theft risk — spending needs a plonky2 proof of knowledge
of the underlying `secret`, and recovering that from the inner_hash means
inverting Poseidon2. But it does make mining income permanently attributable:
the privacy in this design is at the **exit**, not at receipt. The full analysis,
with source citations, is in **`doc/wormhole-rewards.md`**.

That on-chain derivation is also why the node wants a **preimage rather than an
address**: the runtime does the deriving, which is what proves the payout target
is a canonical wormhole address rather than an arbitrary account. Passing an
address would give the runtime nothing to verify. `--rewards-inner-hash` that
isn't a canonical Poseidon digest is rejected at startup.

Knowing the preimage does not let anyone spend the rewards — spending authority
is the wallet key, which never goes near this fleet. Deriving an inner_hash for
a wallet you already control is a one-time offline step, done where the seed
already lives:

```sh
quantus-node key quantus --scheme wormhole --words < mnemonic.txt
```

The binary refuses `--words`/`--seed` as command-line arguments, since argv is
world-readable and lands in shell history and audit logs. Nothing in this repo
needs, wants, or accepts a seed phrase.

The miner's credentials — `miner-auth-token` and `miner-tls-cert-sha256` — are
genuine secrets, and are the only ones here. The node generates them on first
start and regenerates them if its base-path is wiped, so the deploy copies them
from node to miner on every run. That makes the pairing self-healing instead of
a silent auth failure six months later.

## Usage

One-time, from a workstation with sudo ssh to the targets:

```sh
./script/infra-setup.sh --pubkey ~/.ssh/id_gitea_ci.pub
```

Re-run it whenever the deploy gains a new file to ship — each job preflights the
target's sudoers against the grants in `infra-setup.sh` and fails up front,
naming the missing paths, rather than dying partway through an rsync.

Then set each node's `inner_hash` in the `deploy-node` matrix. Derive one per
host, offline, from a **dedicated mining wallet** — see
`doc/wormhole-rewards.md` §6 for why that matters more than being careful.

Then push to `main`, or run the **deploy** workflow from the Actions UI. It has
two modes:

- **deploy** (default, and what a push does) — apply, then validate.
- **validate** — check only, change nothing. Asserts each unit is active, the
  installed binary matches the pinned version, the node's P2P and miner ports
  are listening, peers are connected, the GPU is enumerated and drawing power,
  and `miner_hashes_total` is advancing.

**A deploy that changes nothing restarts nothing.** Files are compared by
checksum, not rsync's default size+mtime — `fetch` re-downloads the binaries
every run, so their mtimes are always new and the default heuristic would report
a change on every deploy. Only a genuine content difference in the binary, unit
or config triggers a restart; firewalld definitions are reloaded separately and
never bounce the daemon. A unit that is unchanged but *stopped* is started.

That matters more than tidiness: bouncing a syncing node costs it every peer and
a RocksDB reopen, and this workflow is also the thing you run to check an
untouched deployment is still healthy.

`systemctl is-active` alone is not evidence this stack works: a miner that found
no GPU adapter, and a node that is authoring nothing because no miner is
attached, both look perfectly healthy to systemd. Hence the counter check.

Upgrades are deliberate — bump `NODE_VERSION` / `MINER_VERSION` in the workflow.
There is no auto-update on a validator on purpose.

## Known conflict: GPU contention with helexa ⚠️

A miner host may already be running helexa `neuron` against the same GPU
(`architecture/generic.md` §11, `inference.md`). They contend directly: the miner
drives the GPU to ~95% utilisation and near its power limit continuously, which
makes inference latency on that host by turns awful and unpredictable.

Nothing here resolves that, and the deploy does not try to arbitrate. Options,
in rough order of preference:

1. Put mining on a host that is not serving inference.
2. Make the miner yield — `--gpu-throttle-ms`, or an `ExecCondition` that
   refuses to start while a model is loaded.
3. Accept the contention and treat inference on that host as best-effort.

Check what a candidate miner host is already serving before enabling
`quantus-miner.service` on it, not after.

## Deviations from house convention

- **`runs-on: infra`, not `fedora-43`.** `gitea-runners.md` §4 says a deploy
  needs only ssh + rsync and so fits `fedora-43`. That is about tooling, not
  routing — these targets are mesh-only `.internal` names, and `lair/mail`'s two
  working deploys both use `infra` for that reason. Revisit if the fedora runners
  gain mesh reachability.
- **Upstream protocol ports** rather than derived ones — see Ports above.
- **`MemoryDenyWriteExecute=false`** on the node (Substrate JITs its wasm runtime
  through wasmtime) and **`PrivateDevices=false`** on the miner (needs
  `/dev/nvidia*`). Both are annotated in the unit files; neither is tidy-up-able.
- **`Type=simple`, not `notify`** — `quantus-node` does not `sd_notify(READY=1)`.
- **No Rust workspace.** This repo deploys upstream release binaries; there is
  nothing to build, so `generic.md` §1–§2 do not apply.

## Upstream

- node — <https://github.com/Quantus-Network/chain> (`MINING.md` is the real doc)
- miner — <https://github.com/Quantus-Network/quantus-miner>
- guide — <https://docs.quantus.com/guides/mining>

Note that Planck is a **testnet**: "tokens have no monetary value". Mainnet
difficulty and reward schedule are unpublished, so no profitability claim here is
possible yet. Also worth tracking: this PoW has no memory hardness and a small
fixed state, which makes it an unusually clean FPGA/ASIC target — relevant to any
decision to buy hardware for it.