observer/CLAUDE.md

CLAUDE.md

Guidance for Claude Code working in this repository.

Read readme.md first — it carries the mechanism (how authorship is decoded from block headers), the reasoning behind the window semantics, and the list of deliberate deviations from house convention. This file is the things that will bite you.

Conventions: ~/git/architecture — generic.md for the workspace shape, deployment-gitea-actions.md for the deploy, port-allocations.md for the port, reverse-proxies.md and external-tls.md/internal-tls.md for the vhosts.

Work is filed before it is done

Most fixes and features start as a Gitea issue on quantus/blackbeard.observer, and the commit that implements one closes it:

Closes #12

The point is an auditable history — why a thing was built, what was believed at the time, and what turned out to be wrong on the way. That last part is the valuable half, so when an investigation contradicts the issue, comment on the issue rather than quietly building the right thing. Issue #1 asserted that execute_at came from PendingTransfers; it does not, and the correction is a comment there, dated, rather than a surprise in a diff.

An issue is not a ceremony for a typo. It is for anything where somebody later would reasonably ask "why is it like this".

The things that cost an afternoon

substrate-telemetry sends its JSON in binary WebSocket frames, not text. A client that handles only Message::Text connects, subscribes, reports itself healthy, and receives absolutely nothing — no error, no decode failure, an empty feed and telemetry_nodes: null. Worse, a throwaway probe written in Python works, because json.loads accepts bytes, so the two disagree for no visible reason. Both frame types are accepted in telemetry.rs and rpc.rs. Do not "simplify" either match arm.

Difficulty is a little-endian U512. state_call returns a SCALE-encoded U512, byte-reversed relative to how a hash reads. Decoding it big-endian yields a plausible-looking number that is wrong by ~10^150, and every hashrate derived from it is wrong without ever looking broken. digest::u512_le has the test.

A measured block interval is only valid at the tip. A syncing node imports history at disk speed; dividing difficulty by that gives a hashrate wrong by orders of magnitude that still renders as a number. RollingWindow::push takes an explicit at_tip flag and backfilled blocks contribute no timing sample. The Interval enum exists so a caller cannot forget to say which kind it has.

A miner's hashrate is its summed work over time, never its share of the network. Difficulty is expected hashes per block, so the difficulty of the blocks a miner won, divided by the time they spanned, is its hashrate directly — and each block carries the difficulty in force when it was won.

share of blocks × network hashrate is only correct while the network is the same size across the whole window. On mainnet's launch day difficulty rose 346-fold inside one 14,400-block window, so a miner that won early — when a block cost a three-hundredth of what it costs now — had that share multiplied by today's network and was credited with ten times the hashrate it had (25.4 GH/s reported against 2.5 GH/s actual). Both the leaderboard and miner_series were wrong this way. The headline network figure still divides current difficulty by the measured interval, because that one is asking "how hard is this to win right now" rather than "what did this miner do".

The window's duration comes from authored_at, not observed_at: a stretch this process caught up on carries observation times seconds apart for blocks minutes apart, and dividing real work by that invents hashrate.

difficulty really is expected hashes, and this has been checked three ways. Before "correcting" any hashrate figure, know that the formula is the chain's own: qpow-math::is_valid_nonce accepts when hash < U512::MAX / difficulty, pallet_qpow::verify_nonce_internal passes it exactly what get_difficulty() returns, and the hash is uniform — 200,000 samples of qp_poseidon_core::hash_squeeze_twice gave a mean of 0.4995 of 2^512 against a uniform 0.5, and 12.48% below 2^509 against 12.5%. So acceptance is 1/difficulty and expected trials is difficulty, exactly.

A per-miner hashrate is a Poisson estimate and needs its error bar. The estimator is unbiased; that is not the same as accurate. k blocks carries a relative standard error of 1/√k — ±100% at one block, ±71% at two, and only ±20% at twenty-five. The leaderboard printed three significant figures off a single block for a miner last seen four hours earlier. miner_hashrate returns the count's error alongside the figure and the UI renders it; do not drop it.

Our own miners disagree with the chain by ~3.5× and it is not the observer. Measured 2026-09-09: beast 1.116, benjy 0.461, quadbrat 0.082 GH/s — 1.66 GH/s total, and 1.39 GH/s instantaneous over a 90 s sample. Over 29 blocks (±18%) in 6.9 h the chain credited the same preimage 1.32e14 expected hashes, i.e. 5.3 GH/s. Sampling does not explain a 14-sigma gap. miner_cpu_hash_rate is 0 on all three hosts — every hash is on GPU — and while the miner's CPU path calls qpow_math::get_nonce_hash directly, the CUDA kernel is a separate implementation with its own counter. That is where to look, not here.

Windows are block counts, never durations. See readme.md. Anything that turns Window into a time range is wrong.

The difficulty of block h lives in the state of block h-1. pallet_qpow::on_finalize retargets at the end of every block, so QPoWApi_get_difficulty at the latest state returns what the next block must meet, not what the tip met. Recording that against the block just seen is off by one, which on a gradual retarget is a number that still looks entirely right. RpcClient::difficulty_for_child_of asks at header.parent_hash, which is the same call the miner made when it built the block. It needs state the node may have pruned; an error there is None, never a fallback to the current value — substituting it is how a gap fill stamps today's difficulty across a stretch of old heights and draws a flat plateau in the history chart.

Attribution confidence is measured over votes cast, not over the window. votes holds one slot per block checked and a None for every block whose first reporter the feed could not separate from its second. Those are abstentions. Dividing agreement by the window length counts them as votes against, and on a chain where clean leads are the minority that makes MIN_CONFIDENCE unreachable: mainnet resolves a lead on about a quarter of blocks, which capped every author at 0.25 against a 0.6 bar and named nobody, however unanimous the cast votes were. MIN_VOTES is the companion guard, so one lucky vote in a silent window is not 100% agreement.

Two things that look like this bug and are not: the feed sends a 190-node roster before any live block traffic — ImportedBlock does not start until about 40 s after subscribe:, so any probe shorter than that sees zero and concludes the feed is dead. And the roster's AddedNode entries carry a placeholder block of height 0 with a zero hash; counting those as import reports inflates the apparent number of nodes claiming to be first.

observed_at is only a measurement when at_tip is true. The flag is now a column (0002). A gap fill writes a whole batch of observed_at within the same second, so any query treating it as timing data must filter on it. The chain series sidesteps this entirely by using authored_at, the chain's own clock.

chain_subscribeNewHeads skips blocks. It reports the best head, so when several import at once the intermediates never arrive. It is a liveness signal, not a ledger — ingest fills gaps against chain_getBlockHash. Removing that would quietly under-count exactly the miners who won blocks during a burst.

Blocks are keyed on (chain, height), not hash. This chain reorgs; the upsert is what makes a replacement overwrite rather than accumulate. A schema keyed on the hash would inflate the losing fork's author forever.

A reorg copies the old row into block_displacement first (0003), in the same statement as the upsert. That is not optional bookkeeping: the node cannot be asked for the block later. blocks-pruning defaults to archive-canonical, so non-canonical bodies go once finality passes them — about a hundred blocks behind the tip on Planck — and difficulty is a state read against a 256-block default. Whatever record_blocks captures is all there will ever be. The two halves are one statement because every data-modifying CTE sees the same snapshot, so the copy reads the pre-update row; splitting them would need an explicit transaction to be equivalent.

A block decoded against the wrong runtime must fail, not succeed. The whole metadata-oracle approach rests on Runtime::decode_events refusing a partial read. Events are a Vec<EventRecord> and SCALE is self-describing only against a registry, so a nearly-right registry decodes a nearly-right answer and leaves bytes over. That leftover is the only signal an upgrade boundary was crossed — it is what makes trying the tip's runtime speculatively safe, and what triggers runtime_at to ask which runtime actually produced the block. Relaxing that check to tolerate trailing bytes would turn a loud failure into silently wrong rewards.

An extrinsic's first byte is not its version. The top two bits are a type tag — 0b10 signed, 0b00 bare, 0b01 general — and only the low six are the version. Mainnet emits 0x84 (signed, v4) and 0x05 (bare, v5) in the same block while the metadata declares extrinsic version 4. A decoder that reads the byte as a version and checks it against the metadata rejects every timestamp inherent on the chain.

Never store a signature. Dilithium is 5.3 KiB per transaction — two orders of magnitude larger than the call it authorises. decode_extrinsic decodes it because that is the only way to know where the call begins, then keeps the scheme's name and the byte count and discards the bytes. At a block a second, storing them would be gigabytes a month that nothing renders.

Backfill progress is a cursor, not max(height) from chain_event. A block with no indexed event and a block never looked at give the same answer, and the first is the common case — INDEXED_EVENTS is deliberately narrow. Inferring progress from the rows makes the backfill re-read the same empty stretch forever. event_scan holds one contiguous interval, and each batch is walked in the direction that keeps it contiguous if it stops early: upward when extending high, downward when extending low. Walking a downward batch upward leaves the unread remainder between what was just read and low, which the cursor cannot express and nothing would ever notice.

Historical state_getRuntimeVersion costs seconds, not milliseconds. It makes the node load and instantiate the runtime WASM out of that block's state: measured at ~4 s against Heisenberg's public endpoint versus ~0.25 s at the tip. discover_runtimes is its own task for exactly this reason — a bisection is ~100 of those calls, and inside the four-second poll loop it stalls difficulty, the summary broadcast and the telemetry attach for minutes on every start. The 10 s RPC_TIMEOUT is what makes those calls succeed at all; shortening it would make runtime history quietly stop working while everything else stayed fine.

first_seen_height = 0 means "not known", not genesis. A runtime cached on the very first poll is recorded before the header subscription has delivered a height. lower_first_seen uses least, so a zero left in place would win forever and claim every runtime began at block 0 — it special-cases zero for that reason. Do not simplify it back to a plain least.

An account is told from a hash by registry path, never by shape. After normalise an AccountId32 and an H256 are both 0x plus sixty-four hex characters. decode_typed returns the account set the decoder collected while it still knew, and that is the only sound source — filtering the rendered JSON by shape claims System::ParentHash and ZkTree::Root name accounts.

A storage entry names an account only if its value is one. System::Events is full of accounts and names none of them: they are a payload it holds for one block. The first cut of the roles index treated "contains an account" as a role and labelled half the chain's active addresses Events. The test is value being a string that is itself in the account set.

A JSON-RPC error is the node answering; a transport failure is not. RpcClient::call fails over to the next endpoint on the second and never on the first. Moving on a real error would hide a caller's own mistake behind a second node making the same complaint — and would walk the whole endpoint list to do it. DataError::Rpc means "answered, with an error" and stops; Malformed, Http and timeouts mean "did not answer" and move on.

A pruned block is a success, not an unhealthy endpoint. state_getStorage for state a node has dropped returns {"result": null}, which reaches the caller as Null. Treating that as a failure would fail over through every endpoint asking a question none of them can answer, and turn a legitimate absence into an outage.

Endpoints are stuck to, never round-robined. A storage read at an old block hash needs a node that still holds that block's state, and nodes prune on their own schedules — alternating between them returns a mixture of answers and absences that reads as sparse data rather than as a configuration problem. The cursor is shared across clones so a failover one task discovers is not rediscovered by every other task on that chain.

state_getKeysPaged rejects a count over 1,000, it does not truncate. Asking for 1,600 is an RPC error, and a caller that unwrap_or_default()s it has turned "could not ask" into "the answer is nothing" — which for a map like PendingTransfers reads exactly like a chain with nothing in flight. MAX_KEYS_PER_PAGE clamps it in rpc.rs, because the ceiling is the node's rather than a caller's preference. More generally: unwrap_or_default() on anything that talks to a node is how a wrong answer comes to look like a right one.

A wrong storage hasher reads as absent, not as an error. A key is twox128(pallet prefix) ++ twox128(item) ++ hashed keys, and each key's hash is declared per entry — System::Account is Blake2_128Concat, other maps are Twox64Concat. Get it wrong and state_getStorage returns null, which is indistinguishable from an empty entry. Runtime::storage_key therefore takes every part from the metadata, including PalletStorageMetadata.prefix, which is usually the pallet name and is not required to be. Its test pins two keys against ones read off the live chain by hand; keep it that way, because nothing else would catch a mistake here.

Absent is not zero, and only the modifier knows which. An optional entry holding nothing means nothing. A default entry holding nothing means the runtime's default, which is a value the chain never wrote. StorageTarget carries the default so a caller can tell them apart, and read_balance deliberately does not apply it: AccountInfo defaults to a zeroed struct, and Substrate reaps empty accounts, so falling back would turn "this account does not exist" into "this account holds nothing". The treasury address on mainnet is exactly that case.

Facts established by measurement

Taken from the live Planck chain, 2026-09-04. Don't re-derive or contradict without re-measuring.

Planck is no longer the tracked chain. Mainnet launched 2026-09-09 and the node on bob was switched to it; the [[chains]] entry followed the node, so quantus is now what that endpoint serves and Planck is telemetry-listed only. The decoding facts below still hold — same runtime family, same digest shape — but every timing figure was measured on Planck against a 6 s target and says nothing about mainnet, which targets 12 s.

• Header digest shape is exactly 0x06 + 706f775f + compact 0x80 (32) + 32-byte preimage. Genesis is the only header without one; 12/12 recent headers decoded.
• Telemetry propagation: first reporter stamped 0, others 50–620 ms. The ATTRIBUTION_LEAD_MS = 20 threshold only has to exclude a tie — identity over many blocks does the discriminating.
• Planck's observed block interval is ~13–15 s against a 6 s target, with consecutive gaps of 1.6 s, 4.4 s, 13.5 s, 26 s, 27 s. This is why MIN_TIP_SAMPLES is 20: at five samples the headline hashrate swings by a factor of three between refreshes.
• system_properties: PLK, 12 decimals, ss58 prefix 189. Genesis 0x4901bf5c…e65e72.
• Mainnet, read from bob 2026-09-09: system_chain "Quantus", token QTC, 12 decimals, ss58 189, genesis 0xfb5487c0…626fba. Its target block time is 12 s, not Planck's 6 — TARGET_BLOCK_TIME_MS = 12_000 at commit b017e642, which is the 0.11.1-b017e6420aa the node reports. Check the constant at the commit the node actually runs, not at the tip of the chain repo: this figure is the denominator for every hashrate shown before twenty tip samples exist, so getting it wrong publishes a headline that is wrong by exactly that factor and labelled "nominal" rather than broken.
• The telemetry feed URL is wss://feed-telemetry.quantus.cat/feed — found in /tmp/env-config.js on the telemetry site, not the /feed path on the main host, which 404s.

A miner's reward address is derivable from its preimage, offline. pallets/mining-rewards pays qp_wormhole::derive_wormhole_address(preimage), which is qp_poseidon_core::rehash_to_bytes — Poseidon2 over Goldilocks. blackbeard-core::wormhole runs the same function and SS58-encodes it with prefix 189. It takes the preimage and nothing else: no genesis, no chain parameters, so one preimage is one address on every Quantus chain. That is also what makes carrying a name across chains sound. The test pins it against a real pair (baba-gorchitsa), so a qp-poseidon-core upgrade that changed the derivation fails the build rather than silently relabelling every miner.

An unnamed miner therefore renders as its address, not its preimage — both identify it exactly, and only one is a string its owner has ever seen. Do not infer "is this a name or an address?" from the string: several real node names begin with q, so RecentBlock and LeaderboardRow carry the AttributionSource and the UI reads that.

The frontend's types are generated

web/src/api/generated/ is written by ts-rs from blackbeard-entities when cargo test -p blackbeard-entities runs. Never edit those files. CI fails the build if the committed output is stale.

u64 maps to bigint by default, which is wrong — serde_json puts a u64 on the wire as a JSON number, so the type and the runtime value disagree. Every u64 DTO field carries #[ts(type = "number")]. Add it to any new one.

A restart is visible, so warm start has to put everything back

blackbeard-api-cert.path watches the host certificate and issues a full systemctl restart when it rotates — several times a day, on top of every deploy. Two pieces of state live only in memory and used to come back empty:

• The block ticker, which is built from live blocks and has no other source. It returned holding one row and refilled over the following minute.
• The tip samples behind the measured interval, because the window replay pushed everything with at_tip: false. Right for blocks a previous process caught up on; wrong for the ones it genuinely watched arrive, which 0002 has recorded ever since. Without them the headline hashrate falls back to the target — at mainnet's interval that is an eight-fold drop, correctly labelled nominal and reported as a bug every time.

restore_ticker_and_timing puts both back, so a restart is no longer user-visible. It runs after the attributions and carried names, because the ticker rows need their display names. Restored tip samples are capped at ten minutes old: the interval stays arithmetically valid across a gap, but it stops being current, and a sample from an hour ago averaged with one from now describes the hour rather than the tip.

Worth knowing: the store already recycles pooled connections hourly precisely so a rotated certificate is a routine reconnect. The path unit's restart is therefore belt-and-braces over a case the pool already handles.

Separation of concerns is load-bearing here

• blackbeard-core has no I/O, no clock, no sockets. That is what lets the decoding and the statistics — the two parts that are easy to get subtly wrong and hard to notice — be exercised by unit tests instead of only against a live chain. Keep it that way; if something there needs the time, pass it in.
• blackbeard-data owns retries, reconnects and schema.
• blackbeard-api wires them together and owns nothing else.

Locking

ChainRuntime::inner is a std::sync::RwLock, never held across an .await. Everything under it is CPU work on in-memory collections. Do not switch it to tokio's — that variant makes it easy to accidentally hold a lock across an await and stall a worker, for no benefit here.

Database work

Queries use sqlx::query! (compile-time checked) with the offline cache in .sqlx/ committed, so CI builds with SQLX_OFFLINE=true and no database.

After changing any query, regenerate the cache or CI fails on a stale one:

podman run -d --rm --name bb-pg -e POSTGRES_PASSWORD=dev \
  -e POSTGRES_DB=blackbeard -p 55432:5432 docker.io/library/postgres:18-alpine
export DATABASE_URL='postgres://postgres:dev@127.0.0.1:55432/blackbeard'
cargo sqlx prepare --workspace -- --all-targets

Migrations are sequentially versioned and immutable once committed. Correct a mistake with a new file, never by editing one that has landed — the runner's checksum diverges and it refuses to start.

Charts

Any chart added here must be read against the dataviz skill first. The constraint already in force: bronze #bd8829 and crimson #d8453a are adjacent hues and fail CVD separation as a categorical pair. Every chart on the site is therefore single-series — magnitude, one hue — with identity carried by labels and row treatment. A second series in the accent colour is the one change that would break the palette, and it would not look broken.

This is why the stat-tile sparklines do not follow the usual "current period in the accent" convention: the newest point is emphasised with --data-bright, a lighter step of the same hue. They also do not start at zero — a sparkline is read for shape — which is why the hover readout and the aria-label both carry real numbers, and why the readout borrows the tile's note line instead of floating a tooltip: five overlays in a row of 180px tiles is not a hover layer, it is a pile.

web/src/index.css documents the validated values. Re-run the validator after touching them:

node <skill>/scripts/validate_palette.js "#bd8829" --mode dark --surface "#14110d"

Deployment gotchas learned the hard way

A sudoers grant matches the whole argument vector. The restorecon grant names three paths; running it with one path is a different command and asks for a password. Run the command exactly as infra-setup.sh grants it, which is what the workflow does.

The config check runs as blackbeard, not root — runas (blackbeard) in sudoers. A root-run check passes on a config the service account cannot read, which is the failure it exists to catch.

CorsLayer::allow_origin replaces, it does not append. Folding over a list of origins leaves only the last one allowed. It does not break the site — the frontend is same-origin and never consults CORS — so it is invisible until something else tries to call the API. Use AllowOrigin::list.

The API binds the host's mesh address, never loopback and never 0.0.0.0. The fleet has one firewalld default zone, so a wildcard bind plus the named service would publish it on every address the host carries; and the edge proxy is at another site, so loopback-only would not serve the site at all. Anything probing 127.0.0.1:25864 gets connection-refused against a perfectly healthy daemon — which is exactly how the first green build still failed its health check. Use $(hostname -f), which resolves to the mesh address.

The edge proxy is at a different site from the API. oolon (kosherinata) fronts the name; the API is on bob (hanzalova) because that is where the Planck node lives. A health probe on the API host cannot see that hop, so there is a separate check that curls the API from oolon. A firewalld service scoped to bob's own /16 would leave a live site with a dead /v1 and nothing would fail.

The edge proxy cannot reach its own public name. From inside the mesh blackbeard.observer resolves to the site's WAN address and dead-ends on the OPNsense LAN interface (reverse-proxies.md §2). Verify the vhost with --resolve blackbeard.observer:443:127.0.0.1, which still exercises the :443 stream router, SNI, the vhost, the cross-site hop and the API.

A frontend-only push skips the Rust build and the API deploy. The what changed step in deploy.yaml diffs against github.event.before and sets one output; the gate, the musl build, the ts-rs drift check and the whole deploy-api job hang off it. Two entries in that path list are not obvious and must not be trimmed: asset/, because deploy-api ships the systemd units, the firewalld service and the rendered config from it; and web/src/api/generated/, because the drift gate only runs when cargo test has regenerated those files, so a hand-edit of them must not be able to arrive labelled "frontend only" — which is precisely the change that gate exists to catch. Anything unrecognised counts as Rust: a false positive costs a slow deploy, a false negative leaves a binary on bob that does not match the commit and nothing reports it.

Runner labels. fedora-* images have no cargo — Rust builds go on rust, which is based on runner-fedora-44 and so carries node + pnpm too. Never corepack enable; pnpm is already on PATH (gitea-runners.md §4).

WebSocket upgrade tests need --http1.1. The vhost serves HTTP/2, and curl negotiates h2, where the Connection: Upgrade handshake is not how WebSockets work — you get a 400 from axum that looks like a proxy misconfiguration.

Verifying a change

systemctl is-active is not evidence this works. A daemon with a node it cannot decode, or a telemetry feed it silently ignores, is perfectly "active" while serving an empty leaderboard. Check the numbers:

cargo run -p blackbeard-cli -- probe --rpc-url http://bob.hanzalova.internal:9944
cargo run -p blackbeard-cli -- standings --chain quantus
curl -s localhost:25864/v1/chains/quantus/summary | python3 -m json.tool

The block route degrades rather than failing, so it is worth checking both ends of that: a recent height should carry difficulty, extrinsics and seconds_since_parent, while one a few days old keeps the body and loses difficulty to state pruning unless this observer recorded it at the time.

curl -s localhost:25864/v1/chains/quantus/blocks/1000 | python3 -m json.tool

A working deployment shows a non-null telemetry_nodes, distinct_miners above one, and named rows in the standings.