observer/CLAUDE.md

CLAUDE.md

Guidance for Claude Code working in this repository.

Read readme.md first — it carries the mechanism (how authorship is decoded from block headers), the reasoning behind the window semantics, and the list of deliberate deviations from house convention. This file is the things that will bite you.

Conventions: ~/git/architecture — generic.md for the workspace shape, deployment-gitea-actions.md for the deploy, port-allocations.md for the port, reverse-proxies.md and external-tls.md/internal-tls.md for the vhosts.

Work is filed before it is done

Most fixes and features start as a Gitea issue on quantus/blackbeard.observer, and the commit that implements one closes it:

Closes #12

The point is an auditable history — why a thing was built, what was believed at the time, and what turned out to be wrong on the way. That last part is the valuable half, so when an investigation contradicts the issue, comment on the issue rather than quietly building the right thing. Issue #1 asserted that execute_at came from PendingTransfers; it does not, and the correction is a comment there, dated, rather than a surprise in a diff.

An issue is not a ceremony for a typo. It is for anything where somebody later would reasonably ask "why is it like this".

The things that cost an afternoon

substrate-telemetry sends its JSON in binary WebSocket frames, not text. A client that handles only Message::Text connects, subscribes, reports itself healthy, and receives absolutely nothing — no error, no decode failure, an empty feed and telemetry_nodes: null. Worse, a throwaway probe written in Python works, because json.loads accepts bytes, so the two disagree for no visible reason. Both frame types are accepted in telemetry.rs and rpc.rs. Do not "simplify" either match arm.

Difficulty is a little-endian U512. state_call returns a SCALE-encoded U512, byte-reversed relative to how a hash reads. Decoding it big-endian yields a plausible-looking number that is wrong by ~10^150, and every hashrate derived from it is wrong without ever looking broken. digest::u512_le has the test.

A measured block interval is only valid at the tip. A syncing node imports history at disk speed; dividing difficulty by that gives a hashrate wrong by orders of magnitude that still renders as a number. RollingWindow::push takes an explicit at_tip flag and backfilled blocks contribute no timing sample. The Interval enum exists so a caller cannot forget to say which kind it has.

Changing what events are kept does nothing to history already read. event_scan records where the backfill has been, not under which policy, and never revisits a range it has claimed. When the allowlist became a skip-list, every block read before that deploy stayed without Wormhole::* (mainnet below 18510) while indexed_from said 1 — found only because a wallet's deposits started at transfer 17,142. Any change to SKIPPED_EVENTS that keeps more needs a migration that collapses the cursor (0008_rescan_events.sql), so the backfill reads history again. Inserts upsert, so a re-read is safe.

A head that closes a gap is not a tip observation. measured_interval divides elapsed time by height difference, which is the chain's rate only if every height between two samples was watched arriving. A gap fill is proof they were not: the heights advanced while this process was elsewhere, so a sample pair straddling one measures how fast the observer caught up. Planck read 0.175 s/block against a true 29.8 — a factor of 170 — and it was flagged measured, so the headline block time, the network hashrate (877 GH/s on a testnet nobody mines) and the window label were all wrong and none of them looked it. ingest now calls forget_tip_samples() whenever it fills a gap and records the closing head with at_tip: false. The interval falls back to nominal until twenty fresh samples exist, which is MIN_TIP_SAMPLES doing its job: nominal and labelled nominal beats measured and wrong. The same applies to an ordinary burst import, where the next head is several heights on with no elapsed time.

Fixing the code did not fix the rows already written. at_tip in the table is whatever the process that wrote it believed, and every process before the receipt-stamping fix believed a drained channel was a tip observation. Those rows are still there, and restore_ticker_and_timing replayed them on every start — Heisenberg reported 0.054 s/block while producing one block every five minutes, entirely from restored samples, with the running code already correct. Restored samples are now checked against the one clock the observer cannot fake: at the tip a block is seen roughly when it is authored, so the observed span should resemble the authored span. Propagation and skew are seconds; a catch-up is a factor. A batch that fails goes unrestored and the interval starts nominal.

observed_at is stamped where the head arrives, not where it is handled. Every head costs two RPC round trips inside record, so on a remote endpoint the 64-deep head channel backs up and drains in a burst — and a batch stamped with Utc::now() at processing time carries near-identical observation times while its heights march on. measured_interval reads that as a chain twenty times faster than it is: Heisenberg reported 0.052 s/block against a real six, and Planck 4.9 against thirty, after the gap-fill guard below had already removed the coarser version of the same error. subscribe_new_heads sends a SeenHead carrying the moment the frame was read off the socket, and that is what becomes observed_at. Do not replace it with a fresh now().

A miner's history chart spans its window, not a hard-coded duration. bucketing used to read 600 blocks -> 1 hour, true only at a 6 s target — so on Planck, whose rate fell ninefold, a rank computed over a five-hour window sat above a chart drawing one hour of it, and nothing said so. It takes the window's measured span now and sizes buckets to hit MINER_SERIES_POINTS, falling back to the block count at the chain's target rate only before the window has filled. Bounded at both ends: a floor so a burst cannot collapse the chart to minutes, a ceiling so the longest window on a slow chain cannot ask the database for a year.

A window's duration is measured, never blocks x interval. The block count is the window; the duration is a consequence of the chain's rate, and that rate changes. RollingWindow::span_seconds is the authored-time span of exactly the blocks tallied and is already the denominator of every per-miner hashrate — so the label is built from it and carries no "~". Multiplying the count by the current interval describes the rate now rather than the period covered, and on Planck those differed by a factor of five even before the interval was itself wrong. Windows are also named for their block count, in the selector and the URL both, for the same reason: /planck/six_hours was a claim the site could not stand behind. The old names still parse and are never emitted.

A miner's hashrate is its summed work over time, never its share of the network. Difficulty is expected hashes per block, so the difficulty of the blocks a miner won, divided by the time they spanned, is its hashrate directly — and each block carries the difficulty in force when it was won.

share of blocks × network hashrate is only correct while the network is the same size across the whole window. On mainnet's launch day difficulty rose 346-fold inside one 14,400-block window, so a miner that won early — when a block cost a three-hundredth of what it costs now — had that share multiplied by today's network and was credited with ten times the hashrate it had (25.4 GH/s reported against 2.5 GH/s actual). Both the leaderboard and miner_series were wrong this way. The headline network figure still divides current difficulty by the measured interval, because that one is asking "how hard is this to win right now" rather than "what did this miner do".

The window's duration comes from authored_at, not observed_at: a stretch this process caught up on carries observation times seconds apart for blocks minutes apart, and dividing real work by that invents hashrate.

difficulty really is expected hashes, and this has been checked three ways. Before "correcting" any hashrate figure, know that the formula is the chain's own: qpow-math::is_valid_nonce accepts when hash < U512::MAX / difficulty, pallet_qpow::verify_nonce_internal passes it exactly what get_difficulty() returns, and the hash is uniform — 200,000 samples of qp_poseidon_core::hash_squeeze_twice gave a mean of 0.4995 of 2^512 against a uniform 0.5, and 12.48% below 2^509 against 12.5%. So acceptance is 1/difficulty and expected trials is difficulty, exactly.

A per-miner hashrate is a Poisson estimate and needs its error bar. The estimator is unbiased; that is not the same as accurate. k blocks carries a relative standard error of 1/√k — ±100% at one block, ±71% at two, and only ±20% at twenty-five. The leaderboard printed three significant figures off a single block for a miner last seen four hours earlier. miner_hashrate returns the count's error alongside the figure and the UI renders it; do not drop it.

Our own miners disagree with the chain by ~3.5× and it is not the observer. Measured 2026-09-09: beast 1.116, benjy 0.461, quadbrat 0.082 GH/s — 1.66 GH/s total, and 1.39 GH/s instantaneous over a 90 s sample. Over 29 blocks (±18%) in 6.9 h the chain credited the same preimage 1.32e14 expected hashes, i.e. 5.3 GH/s. Sampling does not explain a 14-sigma gap. miner_cpu_hash_rate is 0 on all three hosts — every hash is on GPU — and while the miner's CPU path calls qpow_math::get_nonce_hash directly, the CUDA kernel is a separate implementation with its own counter. That is where to look, not here.

Windows are block counts, never durations. See readme.md. Anything that turns Window into a time range is wrong.

The difficulty of block h lives in the state of block h-1. pallet_qpow::on_finalize retargets at the end of every block, so QPoWApi_get_difficulty at the latest state returns what the next block must meet, not what the tip met. Recording that against the block just seen is off by one, which on a gradual retarget is a number that still looks entirely right. RpcClient::difficulty_for_child_of asks at header.parent_hash, which is the same call the miner made when it built the block. It needs state the node may have pruned; an error there is None, never a fallback to the current value — substituting it is how a gap fill stamps today's difficulty across a stretch of old heights and draws a flat plateau in the history chart.

Attribution confidence is measured over votes cast, not over the window. votes holds one slot per block checked and a None for every block whose first reporter the feed could not separate from its second. Those are abstentions. Dividing agreement by the window length counts them as votes against, and on a chain where clean leads are the minority that makes MIN_CONFIDENCE unreachable: mainnet resolves a lead on about a quarter of blocks, which capped every author at 0.25 against a 0.6 bar and named nobody, however unanimous the cast votes were. MIN_VOTES is the companion guard, so one lucky vote in a silent window is not 100% agreement.

Two things that look like this bug and are not: the feed sends a 190-node roster before any live block traffic — ImportedBlock does not start until about 40 s after subscribe:, so any probe shorter than that sees zero and concludes the feed is dead. And the roster's AddedNode entries carry a placeholder block of height 0 with a zero hash; counting those as import reports inflates the apparent number of nodes claiming to be first.

observed_at is only a measurement when at_tip is true. The flag is now a column (0002). A gap fill writes a whole batch of observed_at within the same second, so any query treating it as timing data must filter on it. The chain series sidesteps this entirely by using authored_at, the chain's own clock.

chain_subscribeNewHeads skips blocks. It reports the best head, so when several import at once the intermediates never arrive. It is a liveness signal, not a ledger — ingest fills gaps against chain_getBlockHash. Removing that would quietly under-count exactly the miners who won blocks during a burst.

Blocks are keyed on (chain, height), not hash. This chain reorgs; the upsert is what makes a replacement overwrite rather than accumulate. A schema keyed on the hash would inflate the losing fork's author forever.

A reorg copies the old row into block_displacement first (0003), in the same statement as the upsert. That is not optional bookkeeping: the node cannot be asked for the block later. blocks-pruning defaults to archive-canonical, so non-canonical bodies go once finality passes them — about a hundred blocks behind the tip on Planck — and difficulty is a state read against a 256-block default. Whatever record_blocks captures is all there will ever be. The two halves are one statement because every data-modifying CTE sees the same snapshot, so the copy reads the pre-update row; splitting them would need an explicit transaction to be equivalent.

A block decoded against the wrong runtime must fail, not succeed. The whole metadata-oracle approach rests on Runtime::decode_events refusing a partial read. Events are a Vec<EventRecord> and SCALE is self-describing only against a registry, so a nearly-right registry decodes a nearly-right answer and leaves bytes over. That leftover is the only signal an upgrade boundary was crossed — it is what makes trying the tip's runtime speculatively safe, and what triggers runtime_at to ask which runtime actually produced the block. Relaxing that check to tolerate trailing bytes would turn a loud failure into silently wrong rewards.

An extrinsic's first byte is not its version. The top two bits are a type tag — 0b10 signed, 0b00 bare, 0b01 general — and only the low six are the version. Mainnet emits 0x84 (signed, v4) and 0x05 (bare, v5) in the same block while the metadata declares extrinsic version 4. A decoder that reads the byte as a version and checks it against the metadata rejects every timestamp inherent on the chain.

Never store a signature. Dilithium is 5.3 KiB per transaction — two orders of magnitude larger than the call it authorises. decode_extrinsic decodes it because that is the only way to know where the call begins, then keeps the scheme's name and the byte count and discards the bytes. At a block a second, storing them would be gigabytes a month that nothing renders.

A length prefix out of a blob must never size an allocation. scale_value decodes a sequence by doing Vec::with_capacity(remaining()) before it decodes the first item, where remaining() is the compact length read straight out of the bytes. A Value<u32> is 80 bytes, so a blob that disagrees with the registry can ask for any allocation at all — and a failed allocation aborts the process rather than returning an error. Deployed, that was memory allocation of 82014765760 bytes failed, 316 times in ten hours, from a claimed length of 1,025,184,572. The step that would have caught the mismatch is the one that never runs, so .ok() at the call site catches nothing: there is no Err, only SIGABRT. Every decode therefore goes through Runtime::decode_checked, which walks the bytes first with scale_decode's IgnoreVisitor — that crate contains no with_capacity anywhere, so an impossible length runs out of input on the first item and comes back as an error. This is what makes "a block decoded against the wrong runtime must fail" true in the case where it previously did neither. Do not route a new decode around it.

A catch-up that writes once at the end makes no progress at all. fill_gap used to accumulate the whole gap and call record_blocks after the loop, so anything that ended the process first discarded every block collected. Each block is four RPC round trips, one a historical state read, and this service is restarted several times a day by blackbeard-api-cert.path alone — so a gap wider than the interval between routine restarts was permanently unfillable while logging filling a gap in the head stream on every start, always with the same from, reading exactly like progress. Both testnets sat a day behind that way. It flushes every GAP_FLUSH_BLOCKS now; anything that walks history in this codebase has to be resumable, because the process is not long-lived.

Backfill progress is a cursor, not max(height) from chain_event. A block with no indexed event and a block never looked at give the same answer, and the first is the common case — INDEXED_EVENTS is deliberately narrow. Inferring progress from the rows makes the backfill re-read the same empty stretch forever. event_scan holds one contiguous interval, and each batch is walked in the direction that keeps it contiguous if it stops early: upward when extending high, downward when extending low. Walking a downward batch upward leaves the unread remainder between what was just read and low, which the cursor cannot express and nothing would ever notice.

Historical state_getRuntimeVersion costs seconds, not milliseconds. It makes the node load and instantiate the runtime WASM out of that block's state: measured at ~4 s against Heisenberg's public endpoint versus ~0.25 s at the tip. discover_runtimes is its own task for exactly this reason — a bisection is ~100 of those calls, and inside the four-second poll loop it stalls difficulty, the summary broadcast and the telemetry attach for minutes on every start. The 10 s RPC_TIMEOUT is what makes those calls succeed at all; shortening it would make runtime history quietly stop working while everything else stayed fine.

first_seen_height = 0 means "not known", not genesis. A runtime cached on the very first poll is recorded before the header subscription has delivered a height. lower_first_seen uses least, so a zero left in place would win forever and claim every runtime began at block 0 — it special-cases zero for that reason. Do not simplify it back to a plain least.

An account is told from a hash by registry path, never by shape. After normalise an AccountId32 and an H256 are both 0x plus sixty-four hex characters. decode_typed returns the account set the decoder collected while it still knew, and that is the only sound source — filtering the rendered JSON by shape claims System::ParentHash and ZkTree::Root name accounts.

A storage entry names an account only if its value is one. System::Events is full of accounts and names none of them: they are a payload it holds for one block. The first cut of the roles index treated "contains an account" as a role and labelled half the chain's active addresses Events. The test is value being a string that is itself in the account set.

A JSON-RPC error is the node answering; a transport failure is not. RpcClient::call fails over to the next endpoint on the second and never on the first. Moving on a real error would hide a caller's own mistake behind a second node making the same complaint — and would walk the whole endpoint list to do it. DataError::Rpc means "answered, with an error" and stops; Malformed, Http and timeouts mean "did not answer" and move on.

A pruned block is a success, not an unhealthy endpoint. state_getStorage for state a node has dropped returns {"result": null}, which reaches the caller as Null. Treating that as a failure would fail over through every endpoint asking a question none of them can answer, and turn a legitimate absence into an outage.

Which is why depth is a property of the endpoint, not of the response. Null is also the right answer for most keys read here — an account with no balance, an item never set, the treasury's System::Account entry. So classify_depth probes each endpoint once at startup, reading System::Number at block one: an endpoint that answers holds everything after it. Reads that name a block hash go through call_deep, which tries the sticky endpoint first — costing nothing in the common case — and asks an archive for a second opinion only when a non-archive endpoint returned null, or refused with an RPC error, which is what state_call does when it cannot execute against dropped state. A null from an archive is the answer and stops there. An endpoint that could not be probed stays Unknown and routes as pruned: depth is demonstrated, never assumed, or an unreachable host silently becomes the site's archive of record. All seven configured endpoints probe as archives today, so none of this changes any current behaviour — it is what keeps history from quietly going blank the day one of them stops being one.

Endpoints are stuck to, never round-robined. A storage read at an old block hash needs a node that still holds that block's state, and nodes prune on their own schedules — alternating between them returns a mixture of answers and absences that reads as sparse data rather than as a configuration problem. The cursor is shared across clones so a failover one task discovers is not rediscovered by every other task on that chain.

state_getKeysPaged rejects a count over 1,000, it does not truncate. Asking for 1,600 is an RPC error, and a caller that unwrap_or_default()s it has turned "could not ask" into "the answer is nothing" — which for a map like PendingTransfers reads exactly like a chain with nothing in flight. MAX_KEYS_PER_PAGE clamps it in rpc.rs, because the ceiling is the node's rather than a caller's preference. More generally: unwrap_or_default() on anything that talks to a node is how a wrong answer comes to look like a right one.

A wrong storage hasher reads as absent, not as an error. A key is twox128(pallet prefix) ++ twox128(item) ++ hashed keys, and each key's hash is declared per entry — System::Account is Blake2_128Concat, other maps are Twox64Concat. Get it wrong and state_getStorage returns null, which is indistinguishable from an empty entry. Runtime::storage_key therefore takes every part from the metadata, including PalletStorageMetadata.prefix, which is usually the pallet name and is not required to be. Its test pins two keys against ones read off the live chain by hand; keep it that way, because nothing else would catch a mistake here.

A counter that has never moved is absent, and absent means its default. The runtime writes a key only when something changes it, so TechReferenda::ReferendumCount and ReversibleTransfers::NextTransactionId both read as nothing on mainnet. Reporting them unknown hides the most interesting fact about them — that governance and reversible transfers are shipped and have never been used — so plain_value falls back to StorageTarget::default. That fallback is only correct because the target says whether the entry is Default or Optional; applying it blindly would turn "no such account" into "zero balance", which is the next paragraph.

Absent is not zero, and only the modifier knows which. An optional entry holding nothing means nothing. A default entry holding nothing means the runtime's default, which is a value the chain never wrote. StorageTarget carries the default so a caller can tell them apart, and read_balance deliberately does not apply it: AccountInfo defaults to a zeroed struct, and Substrate reaps empty accounts, so falling back would turn "this account does not exist" into "this account holds nothing". The treasury address on mainnet is exactly that case.

Facts established by measurement

Taken from the live Planck chain, 2026-09-04. Don't re-derive or contradict without re-measuring.

Planck is no longer the tracked chain. Mainnet launched 2026-09-09 and the node on bob was switched to it; the [[chains]] entry followed the node, so quantus is now what that endpoint serves and Planck is telemetry-listed only. The decoding facts below still hold — same runtime family, same digest shape — but every timing figure was measured on Planck against a 6 s target and says nothing about mainnet, which targets 12 s.

• Header digest shape is exactly 0x06 + 706f775f + compact 0x80 (32) + 32-byte preimage. Genesis is the only header without one; 12/12 recent headers decoded.
• Telemetry propagation: first reporter stamped 0, others 50–620 ms. The ATTRIBUTION_LEAD_MS = 20 threshold only has to exclude a tie — identity over many blocks does the discriminating.
• Planck's observed block interval is ~13–15 s against a 6 s target, with consecutive gaps of 1.6 s, 4.4 s, 13.5 s, 26 s, 27 s. This is why MIN_TIP_SAMPLES is 20: at five samples the headline hashrate swings by a factor of three between refreshes.
• system_properties: PLK, 12 decimals, ss58 prefix 189. Genesis 0x4901bf5c…e65e72.
• Mainnet, read from bob 2026-09-09: system_chain "Quantus", token QTC, 12 decimals, ss58 189, genesis 0xfb5487c0…626fba. Its target block time is 12 s, not Planck's 6 — TARGET_BLOCK_TIME_MS = 12_000 at commit b017e642, which is the 0.11.1-b017e6420aa the node reports. Check the constant at the commit the node actually runs, not at the tip of the chain repo: this figure is the denominator for every hashrate shown before twenty tip samples exist, so getting it wrong publishes a headline that is wrong by exactly that factor and labelled "nominal" rather than broken.
• The telemetry feed URL is wss://feed-telemetry.quantus.cat/feed — found in /tmp/env-config.js on the telemetry site, not the /feed path on the main host, which 404s.

Node metadata must not be reachable only through the mining inference. The feed hands over every node on subscribe — 288 on mainnet — while the miner route reaches one only when the first-reporter voting has settled, which needs blocks and a resolvable lead. Most nodes never mine, so that door admits almost nobody: a node visible on the official telemetry site was invisible here. The /:chain/node index and /:chain/node/:peer route have no entry condition, and the miner panel now links to them rather than being the only way in.

"Is this the standings?" is one function, not four. isStandings in lib/routes.ts. Adding the node section had to update the section nav, the window selector, the canonicalising redirect and the board's own render condition — and the first pass caught two of them. The redirect judged /quantus/node/<peer> to be the standings and replaced it with /quantus/3600-blocks, and the render condition left thirty unrelated mining rows under the node. Both were reported rather than caught: the route parsed, the page existed, the build was green, and only opening it showed either. A new section still has to be added to the predicate; it now has one place to be added to.

Telemetry hardware is the node's word, reached by inference. A miner is joined to a node through the same first-reporter voting that produces its display name — Attribution::peer_id is that join, and it is None for anything but a Telemetry attribution because there is no node behind a preimage or a carried name. So everything on the node panel carries two layers of doubt: a node could state its own CPU wrongly, and it might not be this miner's node at all. The panel puts the confidence and vote count in its header for that reason. The CPU is the sharpest case — beside a hashrate it reads as the thing hashing, and on this network miner_cpu_hash_rate is zero on every host measured, so the work is on GPUs telemetry never mentions.

Location is a country and no finer, and the schema enforces it. The feed sends [latitude, longitude, city]; blackbeard_core::geo derives a country offline and the coordinates are then dropped. node_telemetry has no city, latitude or longitude column — a column that does not exist cannot be rendered by mistake. IP geolocation is unreliable at city resolution, so publishing one would assert something nobody knows.

The telemetry wire format is positional and version-specific. NodeDetails is eleven fields in a bare array, and which field is which has changed between telemetry releases. The tests are built from frames captured off the live feed rather than from a spec, because bytes the feed actually sent are the only honest fixture; if a release renumbers a field, those are what says so. A null in a re-announcement never overwrites what is already known — the server geolocates asynchronously, so every reconnect re-sends every node with a null location, and a naive overwrite blanks every country on the site for as long as the lookups take to catch up. That guard exists in both the feed state and the on conflict clause.

A miner's reward address is derivable from its preimage, offline. pallets/mining-rewards pays qp_wormhole::derive_wormhole_address(preimage), which is qp_poseidon_core::rehash_to_bytes — Poseidon2 over Goldilocks. blackbeard-core::wormhole runs the same function and SS58-encodes it with prefix 189. It takes the preimage and nothing else: no genesis, no chain parameters, so one preimage is one address on every Quantus chain. That is also what makes carrying a name across chains sound. The test pins it against a real pair (baba-gorchitsa), so a qp-poseidon-core upgrade that changed the derivation fails the build rather than silently relabelling every miner.

An unnamed miner therefore renders as its address, not its preimage — both identify it exactly, and only one is a string its owner has ever seen. Do not infer "is this a name or an address?" from the string: several real node names begin with q, so RecentBlock and LeaderboardRow carry the AttributionSource and the UI reads that.

The frontend's types are generated

web/src/api/generated/ is written by ts-rs from blackbeard-entities when cargo test -p blackbeard-entities runs. Never edit those files. CI fails the build if the committed output is stale.

u64 maps to bigint by default, which is wrong — serde_json puts a u64 on the wire as a JSON number, so the type and the runtime value disagree. Every u64 DTO field carries #[ts(type = "number")]. Add it to any new one.

A restart is visible, so warm start has to put everything back

blackbeard-api-cert.path watches the host certificate and issues a full systemctl restart when it rotates — several times a day, on top of every deploy. Two pieces of state live only in memory and used to come back empty:

• The block ticker, which is built from live blocks and has no other source. It returned holding one row and refilled over the following minute.
• The tip samples behind the measured interval, because the window replay pushed everything with at_tip: false. Right for blocks a previous process caught up on; wrong for the ones it genuinely watched arrive, which 0002 has recorded ever since. Without them the headline hashrate falls back to the target — at mainnet's interval that is an eight-fold drop, correctly labelled nominal and reported as a bug every time.

restore_ticker_and_timing puts both back, so a restart is no longer user-visible. It runs after the attributions and carried names, because the ticker rows need their display names. Restored tip samples are capped at ten minutes old: the interval stays arithmetically valid across a gap, but it stops being current, and a sample from an hour ago averaged with one from now describes the hour rather than the tip.

Worth knowing: the store already recycles pooled connections hourly precisely so a rotated certificate is a routine reconnect. The path unit's restart is therefore belt-and-braces over a case the pool already handles.

Separation of concerns is load-bearing here

• blackbeard-core has no I/O, no clock, no sockets. That is what lets the decoding and the statistics — the two parts that are easy to get subtly wrong and hard to notice — be exercised by unit tests instead of only against a live chain. Keep it that way; if something there needs the time, pass it in.
• blackbeard-data owns retries, reconnects and schema.
• blackbeard-api wires them together and owns nothing else.

Locking

ChainRuntime::inner is a std::sync::RwLock, never held across an .await. Everything under it is CPU work on in-memory collections. Do not switch it to tokio's — that variant makes it easy to accidentally hold a lock across an await and stall a worker, for no benefit here.

Database work

Queries use sqlx::query! (compile-time checked) with the offline cache in .sqlx/ committed, so CI builds with SQLX_OFFLINE=true and no database.

After changing any query, regenerate the cache or CI fails on a stale one:

podman run -d --rm --name bb-pg -e POSTGRES_PASSWORD=dev \
  -e POSTGRES_DB=blackbeard -p 55432:5432 docker.io/library/postgres:18-alpine
export DATABASE_URL='postgres://postgres:dev@127.0.0.1:55432/blackbeard'
cargo sqlx prepare --workspace -- --all-targets

Migrations are sequentially versioned and immutable once committed. Correct a mistake with a new file, never by editing one that has landed — the runner's checksum diverges and it refuses to start.

The distribution table reports flows and does not judge them

/:chain/network carries a table of who won the blocks and what became of the rewards. It exists because a concern was raised that a pool might not be paying out, and it is built so that it can never be the thing making that accusation.

Direction is a fact in the payload. Wormhole::NativeTransferred and Balances::Transfer both carry from, to and amount, so inflow and outflow are counted from what the chain recorded rather than inferred from an account appearing near an event.

A nonce cannot answer this, and reaching for one is the obvious mistake. Wormhole transfers dispatch unsigned — verify_private_batch and verify_public_batch have no signer — so an address here can move a fortune with nonce: 0. Reading a zero nonce as "has never sent anything" is wrong on this chain.

Neither can outflow, because the chain hides who sends. A Wormhole::NativeTransferred records to and amount but its from is the sentinel 0x0101…0101 for every transfer produced by verify_public_batch or verify_private_batch — 9,894 transfers to 654 recipients on mainnet with no origin recorded. That is the chain's privacy working as designed, not a decoding gap, and it means sent on is blind to exactly the mechanism a wormhole-held balance would use to pay out. A zero there says "nothing left by a route the chain writes down", never "nothing left". The table shipped saying the stronger thing for a day; it does not any more.

Split the events by the extrinsic that produced them before believing any of this — block initialisation with a hidden sender is a mining reward, while verify_*_batch with a hidden sender is a real payment:

(no extrinsic — initialisation)  hidden   23,876 events    72 recipients
Utility::batch_all               visible  15,303        2,145
Wormhole::verify_public_batch    hidden    8,136          538
Balances::transfer_allow_death   visible   6,941          651
Wormhole::verify_private_batch   hidden    1,758          116

A mining address is keyless, so its balance is a receipt total and can only rise. This was once written here as "retained is the one measure privacy cannot hide", on the reasoning that a payment debits the account whichever route it takes. That reasoning is wrong on this chain and the column is gone.

pallets/mining-rewards pays derive_wormhole_address(preimage), a Poseidon2 hash no Dilithium keypair signs for, so nothing can transfer out. The only way to move a reward is a wormhole exit, and Wormhole::credit_and_record (pallets/wormhole/src/lib.rs:1348) credits the destination with Unbalanced::increase_balance and never debits the note's source. So the balance never falls, for anybody. Measured across all of mainnet — QUANPOOL at twelve samples from block 1,000 to 42,400, rising at every one, nonce 0 throughout.

Which means retained read ~100% for every miner because it had to, and a number identical for everyone reads as an accusation pointed at whoever is top of the table. A pool paying out in full and one paying nothing produce the same row. The balance column is received, never holds or balance: those words assert control that does not exist.

What the numbers cannot settle, and what the page therefore says out loud: an operator may hold several addresses; may settle off-chain; or may be a solo miner keeping what it earned, which is the ordinary case. Measured on mainnet, several large accounts send nothing at all, and the accounts distributing to hundreds of recipients have never mined a block — so payout wallets here are funded from somewhere other than mining addresses. There is also no such thing as a "pool" on this chain: the word appears only inside a name an operator reports over telemetry, which is why MinerFlow carries the AttributionSource and the table calls the name column its weakest. A site that flagged entities as delinquent on the strength of a name they chose themselves is one bad match away from libelling someone.

What earns the table its place is the shape of who wins blocks, and nothing more — the payout question belongs on the wormhole page, which is the only place any of it is observable. What comparison it does support has to be like for like. Measured over the full index, every one of the top 25 miners shows zero visible outflow and holds almost exactly what it mined — including a second address calling itself quantus-mainnet-pool. Zero outflow is universal among mining addresses here, so it distinguishes nobody. The accounts that do distribute to hundreds of recipients have never mined a block; they are funded from the genesis endowment, so setting a miner beside them is not a comparison, it is a category error. That mistake was made in this repository before the numbers came in.

Two columns are not measured the same way. received is chain state, true now; mined, sent on and recipients are over the indexed range. On a chain read from genesis they agree; on one still being indexed backwards a miner appears to hold far more than it ever mined. The note under the table says so, because the discrepancy looks exactly like the thing the table is for.

The wormhole page is where the chain's privacy is accounted for

A transfer inserts a commitment as a note in the zk tree; spending one proves a nullifier, which the runtime records in Wormhole::UsedNullifiers so it cannot be spent twice. So the pool is countable even though its movements are not:

ZkTree::LeafCount           82,644   notes created
Wormhole::UsedNullifiers    47,789   notes spent (map enumerated)
outstanding                 34,855   unspent — still inside

Wormhole::TransferCount and ZkTree::UnprocessedLeaves read as absent, so they come back as their declared defaults rather than as unknown — see the storage paragraph above.

Notes per day come from the transfers, not from ZkTree::LeafInserted. That event is in SKIPPED_EVENTS — it fires every block and names no account, so it is not indexed and should not be. Every NativeTransferred carries a leaf_index anyway, which is the same fact by a cheaper road.

Routes are separated by the extrinsic that produced them, because a hidden sender means two different things. In block initialisation it is the chain paying a miner, and the "sender" is the runtime. Inside verify_public_batch or verify_private_batch it is a person spending a note, and the origin genuinely is not recorded. Collapsing the two would make mining rewards look like anonymous payments.

Exit batches look exactly like payroll and mostly are not. Grouping NativeTransferred by producing extrinsic gives one sender, many recipients, repeating — and the obvious reading of it is wrong twice over.

Batch size is aggregation, not generosity. The pallet documents a public batch as one segment per inner private batch, and a segment as one client. A 120-output batch is an aggregator bundling unrelated people, so size measures aggregator throughput. It is deliberately not reported anywhere on the site.

The most recurrent recipient is the aggregator. The address credited in 774 batches is in 91.5% of every public batch anyone submitted and 0% of the private ones — that is the rebate settle_exit_bundle mints to the proof's aggregator_address. QUANPOOL's mining address appears in 501 batches for 7.54 QTC total, 0.015 each: the miner fee share credited to whoever included the batch. Classified, three addresses take 76% of everything the exits pay:

infrastructure     3 addresses   1,647 credits   4,018.9 QTC   2.44 each
recipient        515 addresses   3,167 credits   1,233.9 QTC   0.39 each
miner fee share   21 addresses     243 credits       5.2 QTC   0.02 each

exit_cohorts does that split, and INFRASTRUCTURE_SHARE is the threshold — credited in more than a fifth of all batches. The line is nowhere near anything: the real values are 91.5% on one side and single-digit percentages on the other.

What survives is genuine and is the reason the table ships: among real recipients, 35 addresses paid twenty or more times take two thirds of the remaining value, at twice everyone else's average credit. That is a recurring-payment shape and no more than that — the chain records no sender, so nothing names a payer, and a regular payee is as plausibly an exchange deposit or one person on a schedule as a pool meeting its obligations. Say that on the page every time the table is shown.

This page exists because the distribution table shipped an outflow column that read zero for every miner and looked like evidence. The site now has somewhere that says how much of the chain is unobservable and by which route, so that zero can be read correctly.

The network page mixes two kinds of certainty, and says which

/:chain/network puts chain state and this observer's index on one page, and they are not equally trustworthy. Supply, the map sizes and the capability counters are state: read now, true now. The daily activity is our index, and is only as complete as indexed_from..indexed_to, which the Activity panel prints beside itself and labels "the whole chain" or "a partial index". Never let a daily figure appear without that range: a chart whose axis claims a month over data covering a week is the failure this repository has already shipped twice.

Three specifics worth keeping:

• Signed extrinsics are separated from the total. Most traffic on this chain is the timestamp inherent, so one "transactions per day" line would be mostly clockwork — a number that reads as adoption and is not.

• Four empty lock maps are not "nothing is withheld". Balances::Locks, Holds, Freezes and Reserves are all empty — counted, not assumed — and this file once drew the conclusion that circulating therefore equals total issuance. Wrong by a factor of 436. This runtime's vesting is pot-based, not lock-based: pallets/vesting parks undistributed funds in the account its PalletId owns and pays out from there, which is exactly why no lock records them. modlqvesting held 5,669,940 QTC — 99.54% of every unit of balance at block 42,400, against issuance of 5,682,969. The page reports the pot and issuance-minus-pot; it still declines to say "circulating", because funds leave the pot on a schedule this observer does not read. The pot address comes from Runtime::constant_bytes("Vesting", "PalletId") through blackbeard_core::pallet::pallet_account, never from a hard-coded qvesting — the address is a consequence of a runtime constant, and a runtime that changed it would leave the site reporting an empty pot while looking perfectly healthy. Both steps are pinned by tests against the real mainnet metadata fixture.

• Never sum account balances and call it supply. They exceed TotalIssuance and the gap grows — +13,427 QTC by block 42,400, exactly zero at genesis. Bisected to block 4,800, whose only non-inherent extrinsic is a Wormhole::verify_public_batch: an exit credits its destination without debiting the source and without moving issuance, so privately-moved value is counted twice by any such total. TotalIssuance counts each unit once and is the honest figure. Single blocks reconcile exactly, so this shows up only over history — a spot check of one block will say everything is fine.

• Concentration excludes the vesting pot, and says so everywhere it appears. The pot holds 99.5% of every unit, so a top-1 share or a Gini computed with it in describes an undistributed reserve: it would sit near 1.000 and move only when the pot pays out. compute_concentration drops it by identity — its own storage key, derived from Vesting::PalletId — never by being the largest, because "the biggest account" is a description that would silently start meaning something else the day the pot drains. Measured outside it: top 1 39.6%, top 10 72.2%, top 100 92.2%, Gini 0.965.

  The denominator is the counted balance, not issuance, for the reason two paragraphs up: balances summed across the chain exceed issuance, so shares of issuance would total more than everything.

  And the tail is read with care. That 39.6% top holder is a mining address — keyless, its balance a cumulative receipt that cannot fall however much its operator pays out. Concentration here is partly a statement about who mined most rather than who holds most, and the panel says so in its own words rather than leaving a reader to infer it.

  Nothing renders unless the key walk reached the end of the map (complete): every share divides by a total, and a partial sum makes all of them wrong while looking entirely reasonable.

• Days bucket on the chain's clock, authored_at and chain_extrinsic.at, never on when this observer saw anything. A stretch caught up on carries observation times minutes apart for blocks spanning days.

The vesting route is the only forward-looking page, and it is exact

/:chain/vesting sums pallet_vesting's own accrual across every schedule: nothing before a grant's cliff, the whole grant from its end, linear between. That is arithmetic on chain state, not a projection — which is the only reason a site this careful about measurement will draw a curve into 2030 at all.

blackbeard_core::vesting restates the runtime's three branches and is the half that can be tested, because until 2027 nothing on chain will contradict it: the end branch returning total exactly rather than letting the linear case run out is what stops the final figure being total less a rounding crumb, and mul_div_floor splits the product because total × elapsed on a large grant passes what a u128 holds and would wrap into something small and believable.

Decoded by field name, never by offset. The figures that justified building this were read positionally out of 89 bytes and summed to the pot exactly — good evidence the layout was guessed right, and precisely the shortcut this file already forbids elsewhere. A runtime that reordered the struct would keep decoding, keep summing to something, and be wrong about who gets paid what. a_vesting_schedule_decodes_by_name_against_real_metadata pins the names against a captured mainnet entry.

matches_pot is reported, not assumed. The network page calls that account a vesting pot; what earns the word is that the grants sum to its balance to within ExistentialDeposit. If a grant is ever ended or the pot topped up, the page says the two have parted company rather than going on describing one as the other. Measured 2026-09-14: 48 grants, 5,669,940.000 QTC scheduled against 5,669,940.001 held, nothing claimed, cliff 2027-09-09, fully vested 2030-09-08.

Field is one component, not four. It had been written out three times and the copies had parted company — two rendered note as a visible caption, the third passed it to title as a hover. Both are wanted, so the shared one takes them as separate props rather than picking a winner and silently changing a page. Same lesson as isStandings, found the same way: by going to add a fourth.

The donation QR is generated, never shipped as an image

AboutPanel builds the code in the browser from DONATION_ADDRESS, the same constant it prints underneath. A supplied PNG beside a typed address is two independent claims about where money goes, and nothing on the page could tell a reader if they ever stopped agreeing — the failure is unrecoverable. Generated, there is one address and the code is a rendering of it.

Verified rather than assumed: the matrix qrcode-generator produces was decoded from first principles — function-pattern map, mask 3 unapplied, codewords de-interleaved across version 4-M's two blocks — and reads back qzkQq9ybiPm7gQR9E1ViQbW9RhqA2Rex9ZQkScU5zdJqjrre4 exactly. Worth knowing if this is ever re-checked: an independent encoder (segno) produces a different matrix for the same string at every mask, because mode segmentation differs. That is not a disagreement about the payload and comparing matrices proves nothing; only a decode does.

the_donation_address_on_the_site_is_a_valid_quantus_address reads the literal out of the frontend source and puts it through the SS58 decoder, so a typo fails CI. It is not duplicated into the test, because a copy is one more place for the two to disagree. Confirmed to fail on a single changed character.

The QR draws its own white background and black modules rather than using the palette. A code in the accent hue on warm-black does not scan, whatever the contrast validator says about text — scanners want dark on light, and this is the one mark on the site whose job is to be read by a machine.

Charts

Any chart added here must be read against the dataviz skill first. The constraint already in force: bronze #bd8829 and crimson #d8453a are adjacent hues and fail CVD separation as a categorical pair. Every chart on the site is therefore single-series — magnitude, one hue — with identity carried by labels and row treatment. A second series in the accent colour is the one change that would break the palette, and it would not look broken.

This is why the stat-tile sparklines do not follow the usual "current period in the accent" convention: the newest point is emphasised with --data-bright, a lighter step of the same hue. They also do not start at zero — a sparkline is read for shape — which is why the hover readout and the aria-label both carry real numbers, and why the readout borrows the tile's note line instead of floating a tooltip: five overlays in a row of 180px tiles is not a hover layer, it is a pile.

The live stage is driven by blocks, never by a timer. ServerMessage::Wormhole carries what one block did to the pool — notes committed, notes an exit released, and both amounts — broadcast from index_events after ServerMessage::Block rather than folded into it, because the counts need two more RPC round trips and the whole site's ticker must not wait for one panel. A block that touched no notes sends nothing: an empty pulse every twelve seconds would be noise, and a still stage means a still chain, which is information.

The stage draws the stock as well as the flow, and the stock is read rather than inferred. Flow alone left it nearly still, which is an honest picture of a chain committing ~1.9 notes a block and settling an exit every fifth one — but it left the page's headline figure, the tens of thousands of notes committed and unspent, as a number in a box. The mouth now holds a sample of that population, turning slowly, with arrivals passing in front of it.

Two things about it are load-bearing:

• Pulses must not accumulate into the stock. A pulse's notes_out is an exit crediting an account, and credit_and_record inserts a fresh leaf for that credit — so an exit adds a note rather than retiring one. What retires a note is a nullifier, and those are not counted per block. Adding and subtracting pulses would drift from the truth while looking authoritative, so the figure is re-read from /wormhole every STOCK_REFRESH_MS instead.
• It turns in CSS, not in the rAF loop. The flight marks stop when the chain is quiet; the population is always there, so a JS loop would run forever on a page left open all day. Each shell is a ring of marks on a circle inside a group squashed vertically, so one composited rotate traces an ellipse in the mouth's own plane. prefers-reduced-motion stops the turning in a media query and leaves the marks drawn — how much is inside is the information; only the motion goes.

One mark stands for many and the caption always says how many, printed whether or not a block has arrived yet: forty-five turning marks with nothing explaining them is worse than no marks at all.

Only the live pass may broadcast, and Pass is what makes a caller say so. index_events has two callers — record, for a block that just arrived, and backfill, walking history. The wormhole pulse was broadcast from inside it, so the backfill re-announced every block it re-read; and because the backfill walks towards the tip, the blocks it re-announced were the ones a viewer had just watched. The console listed the same block three times. This is the live/backfill confusion this file already warns about twice, arriving by a third road.

It was worse than a repeat, because the duplicates collided on their React keys (height-index looked unique and was not). React given duplicate keys renders duplicates and stops honouring the list's own length cap — twelve lines became eighteen. The key is a monotonic counter now, so a reorg re-recording a height cannot do the same thing. Nothing in the build caught either half: it shipped, and only watching the live page for a minute showed it.

A mark is not self-explanatory, so every mark gets a line. A dot travelling inward could be anybody paying anybody; without text beside it the stage is an ornament. ServerMessage::Wormhole therefore carries a sample of the individual notes — kind, recipient, amount — and the console below the stage lists them newest-first, the way the block ticker reads, which also means no auto-scroll to fight and the line a reader is looking for never moves.

kind is the third split this codebase makes on the producing extrinsic, and the only one that separates all three cases: reward has no extrinsic at all (block initialisation paying a miner), transfer has an ordinary one, and exit has a verify_*_batch. The recorded sender distinguishes none of them — it is the same sentinel for a reward and an exit both.

Two caps, and they interact:

• MAX_PULSE_NOTES bounds the wire. A block has carried hundreds of notes and sending every one to draw a handful of lines is kilobytes a block for text nobody reads. The counts beside them stay complete.
• CONSOLE_LINES_PER_BLOCK bounds one block's share of the console. Without it a single busy block fills all twelve lines and twelve lines from one block look exactly like twelve blocks of one — the reader loses all sense of rate.

The count of what is not shown hangs on the block's first line, not its last. The list is newest-first, so a count on the last line is the first thing CONSOLE_LINES trims, and a block of 260 would show four notes and never say so. That bug shipped to a screenshot before it was caught.

The console is fed before the reduced guard, so a viewer who asked for less motion still gets the whole reading — verified by forcing the hook: twelve lines, zero moving marks, population still drawn, caption still live.

Three rules keep WormholeStage honest, and each cost a rewrite:

• Marks are capped; the caption is not. A block committing 900 notes draws PER_PULSE_CAP of them and says nine hundred in words. A picture nobody can count is decoration, so the number lives in the text where it is exact.
• It renders outside React. Pulses go through Observer.onWormhole, a separate subscriber set from the state listeners, because a pulse is an event and not a fact about the page — putting it in ObserverState would re-render the leaderboard, the ticker and the whole tree once per block for one animation. The marks are pooled SVG nodes mutated from one rAF loop that starts on a pulse and ends itself when nothing is in flight.
• prefers-reduced-motion stops the motion, not the information. No marks spawn, the caption still updates with real numbers and the aria-label still describes the last block. Verified by forcing the hook rather than assuming.

Direction is the same encoding as the chart — inward against outward — for the same measured reason, and the mouth's ry is capped so the outer ring fits the stage and reused as the squash on every note's path: a mark has to travel in the plane of the thing it is falling into or it reads as passing in front of it.

On /:chain/wormhole the stage replaces the headline stat row. Hashrate, difficulty, block time, miners and height are on every other page and say nothing about the pool.

A diverging chart with one hue puts direction in the geometry. In and out are opposite directions, which is the textbook diverging case — two hues either side of a neutral midpoint. This site cannot have them, so FlowChart carries direction by which side of the zero rule a bar sits on and paints both sides the same bronze. That is a stronger encoding than hue anyway: it survives greyscale, print, forced-colors and every form of colour vision, and it leaves no categorical pair to validate.

The two-shade alternative was measured, not assumed, and it fails: #bd8829 against #e0a63a scores a normal-vision ΔE of 9.9, under the 15 floor, which is a hard fail no legend excuses. Do not give .flow-in and .flow-out different fills.

Three things make the geometry actually readable, and the chart was wrong without each of them — all three were caught by rendering it and looking, not by review:

• A surface gap at zero. Same hue plus touching bars draws one continuous mark straight through the rule, so the encoding vanishes exactly where it has to work. ZERO_GAP holds the house's 2px apart and the rule sits in it.
• A width cap. Five days across a full-width panel drew 170px slabs — the "thick saturated blocks" anti-pattern, wide enough to bury the rule. MAX_BAR_WIDTH keeps marks thin and spaces the columns out instead.
• Paths, not rects. rx rounds all four corners, so the end meeting the baseline rounded too and every bar read as a detached lozenge. The mark spec wants rounded data-ends; only a path does that.

web/src/index.css documents the validated values. Re-run the validator after touching them — both modes, each against its own surface:

node <skill>/scripts/validate_palette.js "#bd8829" --mode dark  --surface "#14110d"
node <skill>/scripts/validate_palette.js "#a8741c" --mode light --surface "#f5f1e8"

A light theme is not the dark theme inverted. #bd8829 reads at 6.6:1 on the warm-black and 2.77:1 on paper — under the 3:1 floor a mark has to clear — so light mode carries its own step of the same hue, and its --data-bright is darker than --data because on paper emphasis is weight rather than glare. The washes are tokens for the same reason: a glow at 0.08 alpha on warm-black is a smear at 0.08 on paper. There are no colour literals left outside the two token blocks; a new one is a colour the other theme cannot override.

auto is a preference, not a palette. The stylesheet only ever sees data-theme="light" or data-theme="dark", because a small inline script in index.html resolves the stored preference against prefers-color-scheme before first paint — anything that runs after the bundle loads runs after one paint, which on a light preference is a full-screen flash of warm-black. That resolution is deliberately duplicated between index.html and lib/theme.ts; the alternative is either the flash or a second copy of every token inside a media query. localStorage throws rather than returning null where site data is blocked, so every access is guarded and falls back to the browser's own answer.

Measured after the light theme landed: dark carries 46 text elements under 4.5:1 and light carries 3, all of which beat their dark counterparts. That gap is --text-muted: #7a6c59 at 3.7 against --surface-1, which is the existing deliberate value this file has always documented — not something the light theme introduced, and not something to "fix" without deciding to change the dark design.

Deployment gotchas learned the hard way

A sudoers grant matches the whole argument vector. The restorecon grant names three paths; running it with one path is a different command and asks for a password. Run the command exactly as infra-setup.sh grants it, which is what the workflow does.

The config check runs as blackbeard, not root — runas (blackbeard) in sudoers. A root-run check passes on a config the service account cannot read, which is the failure it exists to catch.

CorsLayer::allow_origin replaces, it does not append. Folding over a list of origins leaves only the last one allowed. It does not break the site — the frontend is same-origin and never consults CORS — so it is invisible until something else tries to call the API. Use AllowOrigin::list.

The API binds the host's mesh address, never loopback and never 0.0.0.0. The fleet has one firewalld default zone, so a wildcard bind plus the named service would publish it on every address the host carries; and the edge proxy is at another site, so loopback-only would not serve the site at all. Anything probing 127.0.0.1:25864 gets connection-refused against a perfectly healthy daemon — which is exactly how the first green build still failed its health check. Use $(hostname -f), which resolves to the mesh address.

The edge proxy is at a different site from the API. oolon (kosherinata) fronts the name; the API is on bob (hanzalova) because that is where the Planck node lives. A health probe on the API host cannot see that hop, so there is a separate check that curls the API from oolon. A firewalld service scoped to bob's own /16 would leave a live site with a dead /v1 and nothing would fail.

The edge proxy cannot reach its own public name. From inside the mesh blackbeard.observer resolves to the site's WAN address and dead-ends on the OPNsense LAN interface (reverse-proxies.md §2). Verify the vhost with --resolve blackbeard.observer:443:127.0.0.1, which still exercises the :443 stream router, SNI, the vhost, the cross-site hop and the API.

/v1/healthz's commit only ever reports the API's. It is stamped into the binary, so a frontend-only push — which by design does not rebuild or redeploy the API — leaves it reading the previous commit forever. A watcher polling it for the sha you just pushed waits for something that will never arrive, and reports "still deploying" over a site that finished deploying minutes ago. That is exactly what happened while the wormhole console shipped.

To verify a frontend deploy, compare what is served against what you built:

asset=$(curl -s https://blackbeard.observer/ | grep -o '/assets/index-[A-Za-z0-9_-]*\.js' | head -1)
basename "$asset"                                  # served
ls web/dist/assets/index-*.js | xargs -n1 basename # built

Vite hashes the bundle by content, so equal names mean the deployed frontend is byte-identical to the local build. Unequal means it has not landed yet. Grepping the fetched bundle for a string the change introduced is the same check by another road, and works when the local build is not to hand.

A frontend-only push skips the Rust build and the API deploy. The what changed step in deploy.yaml diffs against github.event.before and sets one output; the gate, the musl build, the ts-rs drift check and the whole deploy-api job hang off it. Two entries in that path list are not obvious and must not be trimmed: asset/, because deploy-api ships the systemd units, the firewalld service and the rendered config from it; and web/src/api/generated/, because the drift gate only runs when cargo test has regenerated those files, so a hand-edit of them must not be able to arrive labelled "frontend only" — which is precisely the change that gate exists to catch. Anything unrecognised counts as Rust: a false positive costs a slow deploy, a false negative leaves a binary on bob that does not match the commit and nothing reports it.

Runner labels. fedora-* images have no cargo — Rust builds go on rust, which is based on runner-fedora-44 and so carries node + pnpm too. Never corepack enable; pnpm is already on PATH (gitea-runners.md §4).

WebSocket upgrade tests need --http1.1. The vhost serves HTTP/2, and curl negotiates h2, where the Connection: Upgrade handshake is not how WebSockets work — you get a 400 from axum that looks like a proxy misconfiguration.

Run the gate CI runs, not one that looks like it

Before pushing, from web/:

pnpm format:check && pnpm lint && pnpm build

pnpm build is tsc -b && vite build. npx tsc --noEmit is not a substitute — it resolves a different project graph — and vite build type-checks nothing at all, so the pair of them passes code that CI rejects. That is exactly how Type '"" | Window | null' is not assignable reached a build.

The Rust half, likewise verbatim:

cargo fmt --all -- --check
SQLX_OFFLINE=true cargo clippy --all-targets --all-features -- -D warnings
SQLX_OFFLINE=true cargo test --all
git status --porcelain web/src/api/generated/   # must be empty: the drift gate

Verifying a change

systemctl is-active is not evidence this works. A daemon with a node it cannot decode, or a telemetry feed it silently ignores, is perfectly "active" while serving an empty leaderboard. Check the numbers:

cargo run -p blackbeard-cli -- probe --rpc-url http://bob.hanzalova.internal:9944
cargo run -p blackbeard-cli -- standings --chain quantus
curl -s localhost:25864/v1/chains/quantus/summary | python3 -m json.tool

The block route degrades rather than failing, so it is worth checking both ends of that: a recent height should carry difficulty, extrinsics and seconds_since_parent, while one a few days old keeps the body and loses difficulty to state pruning unless this observer recorded it at the time.

curl -s localhost:25864/v1/chains/quantus/blocks/1000 | python3 -m json.tool

A working deployment shows a non-null telemetry_nodes, distinct_miners above one, and named rows in the standings.