lair/quantus
2
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity

Measure authoring efficiency once the node is synced #1

New Issue
Open
opened 2026-08-31 16:05:24 +00:00 by grenade · 3 comments
grenade commented 2026-08-31 16:05:24 +00:00
Owner
Copy Link

Once substrate_sub_libp2p_is_major_syncing reads 0, run the experiments below to find out whether connectivity tuning does anything measurable. Nothing here is worth attempting while syncing — the node does not author until it is caught up.

The question

In PoW, peer count does not change our share of hashrate, so it cannot increase how often we find a block. What it can change is waste, in two ways:

  1. Mining a dead tip. We hash against the best tip we know. Learning about a new block 400 ms late means 400 ms spent on a dead parent — roughly 7% of a 6 s block interval, guaranteed worthless.
  2. Losing our own finds to a race. A block we mine must propagate before a competing block at the same height.

Both are waste reduction bounded by current propagation delay. Expect a few percent, not a multiplier. The purpose of these experiments is to find out whether it is even that much.

Blocker: the thing worth measuring is not currently observable

quantus-node exports no difficulty metric and no authored-block counter. substrate_proposer_block_constructed counts proposals constructed, not rounds won. quantus-miner exports no solutions-found counter — the full set is miner_hashes_total, miner_hash_rate, miner_cpu_hash_rate, miner_gpu_hash_rate, miner_active_jobs, miner_workers, miner_cpu_workers, miner_gpu_devices, miner_effective_cpus, and nothing else.

So "do we win the share of blocks our hashrate predicts?" — the only honest measure of orphan and stale-work loss — cannot be plotted today.

Options to close the gap, cheapest first:

  • Chain query. Periodically count blocks whose author is our wormhole address (qzp2AxZw…), via RPC against the local node. Needs a small exporter or a timer writing to a textfile collector. No upstream change required.
  • Log-derived. If the node logs a distinguishable line on authoring, count it. Needs verification that such a line exists.
  • Upstream. Ask for a difficulty gauge and an authored-blocks counter. Both are cheap for them and useful to every miner. Worth raising regardless — see the questions already listed in doc/wormhole-rewards.md §8.

Do the chain-query exporter first. Without it the experiments below can only measure proxies, and proxies will not settle the question.

Baseline to capture before changing anything

Let it run at least 24h synced, then record:

  • miner_hash_rate (measured 183 MH/s on the 4090 at ~450 W)
  • blocks authored per hour, and network blocks per hour
  • substrate_sub_libp2p_peers_count, plus the in/out split from substrate_sub_libp2p_connections_opened_total{direction=}
  • mining rounds/min (rate(substrate_proposer_block_constructed_count[5m]) * 60) — should sit near 10/min at a 6 s target
  • hashes per mining round

Experiments

1. Peer count

Current defaults are --out-peers 8, --in-peers 32. Inbound dominates by an order of magnitude — measured 63 inbound connection opens against 5 outbound, with 23 established inbound. So --in-peers is the binding constraint and --out-peers is nearly irrelevant here; raising the latter will do close to nothing.

Try --in-peers 64. Hold for 24h. Compare authored-share and rounds/min against baseline.

Cost side worth watching: this chain uses Dilithium for p2p identity (Using litep2p network backend (with Dilithium)), so post-quantum handshakes make each peer more expensive than on a typical Substrate chain. Watch node CPU and bandwidth, not just the peer gauge.

One genuine advantage here: mining is external, so the node's CPU is not competing with hashing. This node can afford more peers than the usual solo-miner advice assumes.

2. GPU power limit

Measured on a 5090: 600 W → 368 MH/s (0.60 MH/s/W), 450 W → 305 (0.68), 400 W floor → 281 (0.70). 18% better hash per watt at the floor for 78% of peak. The 4090 measured 183 MH/s at ~450 W (0.41 MH/s/W); its own curve is unmeasured.

If the goal is paying an electricity bill rather than maximising hashrate, the floor is likely the right operating point. Measure the 4090's curve, then decide.

3. Miner placement

The GPU host also serves helexa inference from the same card. Mining saturates it. Quantify what inference actually loses before deciding whether to move mining, throttle it, or accept the contention — this is the open decision recorded in readme.md.

Definition of done

A number, not an impression: authored blocks as a fraction of the share our hashrate predicts, before and after each change, with enough samples to be more than noise. At ~10 blocks/min network-wide and a small hashrate share, that may need days rather than hours — decide the sample size up front rather than stopping when the graph looks favourable.

Context

  • Dashboard: Grafana → Quantus folder → Quantus — node & miner, "Mining efficiency" row.
  • The dashboard's text panel carries the same caveat about what is not observable.
  • Assets live in asset/grafana/ and asset/prometheus/; edit there and redeploy, not in the Grafana UI (allowUiUpdates: false).
Once `substrate_sub_libp2p_is_major_syncing` reads 0, run the experiments below to find out whether connectivity tuning does anything measurable. Nothing here is worth attempting while syncing — the node does not author until it is caught up. ## The question In PoW, peer count does **not** change our share of hashrate, so it cannot increase how often we *find* a block. What it can change is waste, in two ways: 1. **Mining a dead tip.** We hash against the best tip we know. Learning about a new block 400 ms late means 400 ms spent on a dead parent — roughly 7% of a 6 s block interval, guaranteed worthless. 2. **Losing our own finds to a race.** A block we mine must propagate before a competing block at the same height. Both are waste reduction bounded by current propagation delay. Expect a few percent, not a multiplier. The purpose of these experiments is to find out whether it is even that much. ## Blocker: the thing worth measuring is not currently observable **`quantus-node` exports no difficulty metric and no authored-block counter.** `substrate_proposer_block_constructed` counts proposals *constructed*, not rounds *won*. **`quantus-miner` exports no solutions-found counter** — the full set is `miner_hashes_total`, `miner_hash_rate`, `miner_cpu_hash_rate`, `miner_gpu_hash_rate`, `miner_active_jobs`, `miner_workers`, `miner_cpu_workers`, `miner_gpu_devices`, `miner_effective_cpus`, and nothing else. So *"do we win the share of blocks our hashrate predicts?"* — the only honest measure of orphan and stale-work loss — cannot be plotted today. Options to close the gap, cheapest first: - **Chain query.** Periodically count blocks whose author is our wormhole address (`qzp2AxZw…`), via RPC against the local node. Needs a small exporter or a timer writing to a textfile collector. No upstream change required. - **Log-derived.** If the node logs a distinguishable line on authoring, count it. Needs verification that such a line exists. - **Upstream.** Ask for a difficulty gauge and an authored-blocks counter. Both are cheap for them and useful to every miner. Worth raising regardless — see the questions already listed in `doc/wormhole-rewards.md` §8. **Do the chain-query exporter first.** Without it the experiments below can only measure proxies, and proxies will not settle the question. ## Baseline to capture before changing anything Let it run at least 24h synced, then record: - `miner_hash_rate` (measured 183 MH/s on the 4090 at ~450 W) - blocks authored per hour, and network blocks per hour - `substrate_sub_libp2p_peers_count`, plus the in/out split from `substrate_sub_libp2p_connections_opened_total{direction=}` - mining rounds/min (`rate(substrate_proposer_block_constructed_count[5m]) * 60`) — should sit near 10/min at a 6 s target - hashes per mining round ## Experiments ### 1. Peer count Current defaults are `--out-peers 8`, `--in-peers 32`. **Inbound dominates by an order of magnitude** — measured 63 inbound connection opens against 5 outbound, with 23 established inbound. So `--in-peers` is the binding constraint and `--out-peers` is nearly irrelevant here; raising the latter will do close to nothing. Try `--in-peers 64`. Hold for 24h. Compare authored-share and rounds/min against baseline. Cost side worth watching: this chain uses **Dilithium** for p2p identity (`Using litep2p network backend (with Dilithium)`), so post-quantum handshakes make each peer more expensive than on a typical Substrate chain. Watch node CPU and bandwidth, not just the peer gauge. One genuine advantage here: mining is external, so the node's CPU is not competing with hashing. This node can afford more peers than the usual solo-miner advice assumes. ### 2. GPU power limit Measured on a 5090: 600 W → 368 MH/s (0.60 MH/s/W), 450 W → 305 (0.68), 400 W floor → 281 (0.70). **18% better hash per watt at the floor for 78% of peak.** The 4090 measured 183 MH/s at ~450 W (0.41 MH/s/W); its own curve is unmeasured. If the goal is paying an electricity bill rather than maximising hashrate, the floor is likely the right operating point. Measure the 4090's curve, then decide. ### 3. Miner placement The GPU host also serves helexa inference from the same card. Mining saturates it. Quantify what inference actually loses before deciding whether to move mining, throttle it, or accept the contention — this is the open decision recorded in `readme.md`. ## Definition of done A number, not an impression: authored blocks as a fraction of the share our hashrate predicts, before and after each change, with enough samples to be more than noise. At ~10 blocks/min network-wide and a small hashrate share, that may need days rather than hours — decide the sample size up front rather than stopping when the graph looks favourable. ## Context - Dashboard: Grafana → **Quantus** folder → *Quantus — node & miner*, "Mining efficiency" row. - The dashboard's text panel carries the same caveat about what is not observable. - Assets live in `asset/grafana/` and `asset/prometheus/`; edit there and redeploy, not in the Grafana UI (`allowUiUpdates: false`).
grenade commented 2026-09-01 03:04:28 +00:00
Author
Owner
Copy Link

Correction: experiment 1 had the wrong knob

The original recommendation — raise --in-peers to 64, leave --out-peers alone — was backwards. Do not run it as written.

What I measured wrong

I used cumulative connection opens: 63 inbound against 5 outbound, concluded inbound dominated, and inferred --in-peers was the binding cap.

Cumulative opens measure churn, not occupancy. Inbound connections arrive and leave constantly; outbound ones are deliberately maintained and stable. So a high inbound open count says nothing about whether the inbound cap is full.

The right measurement

opened − closed per direction, against the cap. Now that the node is synced and dialable:

direction   opened   closed   currently held   cap
in             170      156               14    32
out             36       27                9     8   <-- SATURATED

Also: substrate_sub_libp2p_peerset_num_discovered = 300.

  • Outbound is saturated. The node knows about 300 peers and is permitted to dial 8. That cap is actively binding.
  • Inbound is less than half full (14 of 32). Raising --in-peers would change nothing — inbound is demand-driven, bounded by how many peers choose to dial us, not by our ceiling.

Corrected experiment 1

Change --out-peers from 8 to 24. Leave --in-peers at its default.

Rationale: with 300 discovered peers and 8 dial slots, more outbound links mean learning about new tips from more independent sources, which is the mechanism that shrinks the stale-tip window. Keep it modest — this chain uses Dilithium for p2p identity, so each peer costs more than on a typical Substrate chain. 8 → 24 is defensible; 8 → 100 is not.

The gain remains bounded: this reduces wasted work, it cannot increase hashrate share. Expect a few percent, and expect it to be hard to distinguish from noise.

Run order

Do not change anything yet. Take a clean 24h baseline first — a stable node with no config changes, since every change restarts it and drops every peer. The exporter now measures what is needed:

what metric
observed share quantus_share_observed
expected share sum(miner_hash_rate) / quantus_network_hashrate
stale-tip proxy rate(substrate_proposer_block_constructed_count[5m]) * 60
peer occupancy opened − closed per direction, as above
network context quantus_difficulty, quantus_block_interval_seconds

Then: change --out-peers, hold 24h, compare the same five.

Sample size still governs the answer. At ~10s blocks and ~16% share, 24h is roughly 8600 network blocks and ~1400 of ours — enough that a few-percent effect is near the noise floor. Decide before looking whether a difference that small would change anything you do; if not, the honest outcome is "no measurable effect", and that is a result worth recording rather than a failure.

## Correction: experiment 1 had the wrong knob The original recommendation — raise `--in-peers` to 64, leave `--out-peers` alone — was **backwards**. Do not run it as written. ### What I measured wrong I used *cumulative connection opens*: 63 inbound against 5 outbound, concluded inbound dominated, and inferred `--in-peers` was the binding cap. Cumulative opens measure **churn, not occupancy**. Inbound connections arrive and leave constantly; outbound ones are deliberately maintained and stable. So a high inbound open count says nothing about whether the inbound cap is full. ### The right measurement `opened − closed` per direction, against the cap. Now that the node is synced and dialable: ``` direction opened closed currently held cap in 170 156 14 32 out 36 27 9 8 <-- SATURATED ``` Also: `substrate_sub_libp2p_peerset_num_discovered` = **300**. - **Outbound is saturated.** The node knows about 300 peers and is permitted to dial 8. That cap is actively binding. - **Inbound is less than half full** (14 of 32). Raising `--in-peers` would change nothing — inbound is demand-driven, bounded by how many peers choose to dial us, not by our ceiling. ### Corrected experiment 1 Change **`--out-peers` from 8 to 24**. Leave `--in-peers` at its default. Rationale: with 300 discovered peers and 8 dial slots, more outbound links mean learning about new tips from more independent sources, which is the mechanism that shrinks the stale-tip window. Keep it modest — this chain uses Dilithium for p2p identity, so each peer costs more than on a typical Substrate chain. 8 → 24 is defensible; 8 → 100 is not. The gain remains bounded: this reduces *wasted* work, it cannot increase hashrate share. Expect a few percent, and expect it to be hard to distinguish from noise. ### Run order **Do not change anything yet.** Take a clean 24h baseline first — a stable node with no config changes, since every change restarts it and drops every peer. The exporter now measures what is needed: | what | metric | | --- | --- | | observed share | `quantus_share_observed` | | expected share | `sum(miner_hash_rate) / quantus_network_hashrate` | | stale-tip proxy | `rate(substrate_proposer_block_constructed_count[5m]) * 60` | | peer occupancy | `opened − closed` per direction, as above | | network context | `quantus_difficulty`, `quantus_block_interval_seconds` | Then: change `--out-peers`, hold 24h, compare the same five. Sample size still governs the answer. At ~10s blocks and ~16% share, 24h is roughly 8600 network blocks and ~1400 of ours — enough that a few-percent effect is near the noise floor. Decide before looking whether a difference that small would change anything you do; if not, the honest outcome is "no measurable effect", and that is a result worth recording rather than a failure.
grenade commented 2026-09-01 03:46:58 +00:00
Author
Owner
Copy Link

Baseline window opens: 2026-09-01T03:46Z

Earliest the --out-peers change should be applied: 2026-09-02T03:46Z.

GPU power limits were reduced immediately before this, so every earlier hashrate figure in this issue is stale. The baseline must run against the new limits or the expected-share denominator is wrong.

Configuration under test (baseline arm)

host GPU power limit hashrate temp
benjy RTX 4090 250 W (was 450) 174.8 MH/s 57 °C
quadbrat RTX 3060 130 W (was 170) 42.0 MH/s 76 °C
beast 2× RTX 5090 400 W (clamped from 300; floor is 400) not mining idle

Ours: 216.8 MH/s. --out-peers and --in-peers at their defaults of 8 and 32.

Chain state at window open

difficulty        14,875,185,846
network hashrate  1,367.7 MH/s
block interval    10.876 s   (target 6 s — the chain is running slow)
observed share    15.79 %
expected share    15.85 %    (216.8 / 1367.7)
reward balance    228.105 PLK
peers             25
outbound held     4 of 8

Observed and expected within 0.06pp of each other at window open. That is a coincidence of timing rather than a result — the window is what produces a result.

Note outbound held is 4 of 8, not saturated at this instant, where it was 9 of 8 earlier. Occupancy fluctuates as peers rotate; the case for raising the cap rests on it sitting at the cap over hours, which is what this window will show. If outbound averages well under 8 across 24h, the whole experiment is moot and that is a valid outcome to record.

What must not happen during the window

Do not push. Ten commits are outstanding, several touching the node unit or its config, so a deploy restarts the node and drops every peer — which resets peer occupancy and voids the arm.

If those commits should land sooner, the honest sequence is: push, wait ~30 minutes for peers to re-establish, then reopen the window with a new timestamp. Losing a few hours of baseline is cheaper than comparing across a restart.

Nothing else in flight touches the node: GPU power limits are applied by nvidia-smi without restarting the miner, and the arena exporter reads over RPC.

At 2026-09-02T03:46Z

  1. Record the same five figures averaged over the window, not spot values.
  2. Set --out-peers 24 in the node matrix, deploy, wait ~30 min.
  3. Run the second arm for 24h and compare.

Sample size caveat stands: ~8,000 network blocks and ~1,250 of ours per arm, so a few-percent effect sits near the noise floor. The decision of whether a 1–2% difference would change anything should be made now, before the numbers exist.

## Baseline window opens: 2026-09-01T03:46Z Earliest the `--out-peers` change should be applied: **2026-09-02T03:46Z**. GPU power limits were reduced immediately before this, so every earlier hashrate figure in this issue is stale. The baseline must run against the new limits or the expected-share denominator is wrong. ### Configuration under test (baseline arm) | host | GPU | power limit | hashrate | temp | | --- | --- | --- | --- | --- | | benjy | RTX 4090 | 250 W (was 450) | 174.8 MH/s | 57 °C | | quadbrat | RTX 3060 | 130 W (was 170) | 42.0 MH/s | 76 °C | | beast | 2× RTX 5090 | 400 W (clamped from 300; floor is 400) | not mining | idle | Ours: **216.8 MH/s**. `--out-peers` and `--in-peers` at their defaults of 8 and 32. ### Chain state at window open ``` difficulty 14,875,185,846 network hashrate 1,367.7 MH/s block interval 10.876 s (target 6 s — the chain is running slow) observed share 15.79 % expected share 15.85 % (216.8 / 1367.7) reward balance 228.105 PLK peers 25 outbound held 4 of 8 ``` Observed and expected within 0.06pp of each other at window open. That is a coincidence of timing rather than a result — the window is what produces a result. Note **outbound held is 4 of 8**, not saturated at this instant, where it was 9 of 8 earlier. Occupancy fluctuates as peers rotate; the case for raising the cap rests on it *sitting* at the cap over hours, which is what this window will show. If outbound averages well under 8 across 24h, the whole experiment is moot and that is a valid outcome to record. ### What must not happen during the window **Do not push.** Ten commits are outstanding, several touching the node unit or its config, so a deploy restarts the node and drops every peer — which resets peer occupancy and voids the arm. If those commits should land sooner, the honest sequence is: push, wait ~30 minutes for peers to re-establish, then reopen the window with a new timestamp. Losing a few hours of baseline is cheaper than comparing across a restart. Nothing else in flight touches the node: GPU power limits are applied by `nvidia-smi` without restarting the miner, and the arena exporter reads over RPC. ### At 2026-09-02T03:46Z 1. Record the same five figures averaged over the window, not spot values. 2. Set `--out-peers 24` in the node matrix, deploy, wait ~30 min. 3. Run the second arm for 24h and compare. Sample size caveat stands: ~8,000 network blocks and ~1,250 of ours per arm, so a few-percent effect sits near the noise floor. The decision of whether a 1–2% difference would change anything should be made now, before the numbers exist.
grenade commented 2026-09-01 03:52:42 +00:00
Author
Owner
Copy Link

Correction to the previous comment: the push warning was overstated

I said "ten commits are outstanding, several touching the node unit or its config". That was wrong — I was quoting a stale count from my own commit log rather than checking origin/main.

Actual state at the time of writing: one unpushed commit, e178bcb (boot-time GPU power limits).

It touches only asset/gpu/*, asset/systemd/nvidia-*, script/infra-setup.sh and readme.md. Nothing under the node's unit, binary or config, and its effects are already applied to the hosts by hand.

So pushing it during the baseline window is safe. The deploy triggers (asset/** matches the paths: filter) but is content-addressed — push() compares by checksum, nothing the node depends on has changed, RESTART stays 0, and systemctl restart quantus-node is not called. Peer occupancy is undisturbed.

The general rule still holds and is worth keeping: a push that changes the node's unit, binary or rendered config restarts it and voids the arm. Check what a pending commit actually touches before assuming either way — the deploy log's changed: lines say exactly what moved.

## Correction to the previous comment: the push warning was overstated I said "ten commits are outstanding, several touching the node unit or its config". That was wrong — I was quoting a stale count from my own commit log rather than checking `origin/main`. Actual state at the time of writing: **one** unpushed commit, `e178bcb` (boot-time GPU power limits). It touches only `asset/gpu/*`, `asset/systemd/nvidia-*`, `script/infra-setup.sh` and `readme.md`. **Nothing under the node's unit, binary or config**, and its effects are already applied to the hosts by hand. So pushing it during the baseline window is safe. The deploy triggers (`asset/**` matches the `paths:` filter) but is content-addressed — `push()` compares by checksum, nothing the node depends on has changed, `RESTART` stays 0, and `systemctl restart quantus-node` is not called. Peer occupancy is undisturbed. The general rule still holds and is worth keeping: **a push that changes the node's unit, binary or rendered config restarts it and voids the arm.** Check what a pending commit actually touches before assuming either way — the deploy log's `changed:` lines say exactly what moved.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-bot grenade
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lair/quantus#1
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?