Failover moved on a transport failure and never on a JSON-RPC error, both
right — but a node that has pruned the state being asked for does neither. It
answers `{"result": null}`, which is a success, so whichever endpoint happened
to be sticky decided how far back the whole site could see.
The tempting fix is wrong. Null is also the correct answer for most keys read
here: an account with no balance, an item never set, the treasury's
`System::Account` entry, absent because the treasury was never funded. Retrying
every one of those against every endpoint would multiply the load to re-derive
an answer already in hand, and still end in null. The distinction is not in the
response — it is whether the node could have known, which is a property of the
endpoint.
So `classify_depth` probes each one at startup, reading `System::Number` at
block one. That key is derived in a test rather than pasted, because a mistyped
key is absent everywhere and would classify every endpoint as pruned. An
endpoint that answers holds all state after block one; one that cannot be
reached stays `Unknown` and routes as pruned, since an unreachable host must not
become the archive of record by default.
Reads naming a block hash then go through `call_deep`. It tries the sticky
endpoint first, so the common case — mainnet's loopback node, which is an
archive — costs no extra round trip. Only a null from an endpoint never shown to
hold old state buys a second opinion, and `state_call`'s refusal is treated the
same way, because that is how difficulty is read and a node that cannot execute
against dropped state errors rather than returning null.
All seven configured endpoints probe as archives right now, so this changes no
current behaviour. It is what stops history going quietly blank the day one of
them stops being one — and it says so in the log when it happens, because an
endpoint that cannot see as far back as the site is asking is a configuration
fact worth surfacing rather than a chain fact.
Closes #9
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jp6a8EDar9ueEhAxzep4V5
531 lines
30 KiB
Markdown
531 lines
30 KiB
Markdown
# CLAUDE.md
|
||
|
||
Guidance for Claude Code working in this repository.
|
||
|
||
Read `readme.md` first — it carries the mechanism (how authorship is decoded from
|
||
block headers), the reasoning behind the window semantics, and the list of
|
||
deliberate deviations from house convention. This file is the things that will
|
||
bite you.
|
||
|
||
Conventions: `~/git/architecture` — `generic.md` for the workspace shape,
|
||
`deployment-gitea-actions.md` for the deploy, `port-allocations.md` for the port,
|
||
`reverse-proxies.md` and `external-tls.md`/`internal-tls.md` for the vhosts.
|
||
|
||
## Work is filed before it is done
|
||
|
||
Most fixes and features start as a Gitea issue on
|
||
`quantus/blackbeard.observer`, and the commit that implements one closes it:
|
||
|
||
```
|
||
Closes #12
|
||
```
|
||
|
||
The point is an auditable history — why a thing was built, what was believed at
|
||
the time, and what turned out to be wrong on the way. That last part is the
|
||
valuable half, so **when an investigation contradicts the issue, comment on the
|
||
issue rather than quietly building the right thing**. Issue #1 asserted that
|
||
`execute_at` came from `PendingTransfers`; it does not, and the correction is a
|
||
comment there, dated, rather than a surprise in a diff.
|
||
|
||
An issue is not a ceremony for a typo. It is for anything where somebody later
|
||
would reasonably ask "why is it like this".
|
||
|
||
## The things that cost an afternoon
|
||
|
||
**substrate-telemetry sends its JSON in *binary* WebSocket frames, not text.** A
|
||
client that handles only `Message::Text` connects, subscribes, reports itself
|
||
healthy, and receives absolutely nothing — no error, no decode failure, an empty
|
||
feed and `telemetry_nodes: null`. Worse, a throwaway probe written in Python
|
||
*works*, because `json.loads` accepts bytes, so the two disagree for no visible
|
||
reason. Both frame types are accepted in `telemetry.rs` and `rpc.rs`. Do not
|
||
"simplify" either match arm.
|
||
|
||
**Difficulty is a little-endian U512.** `state_call` returns a SCALE-encoded
|
||
`U512`, byte-reversed relative to how a hash reads. Decoding it big-endian yields
|
||
a plausible-looking number that is wrong by ~10^150, and every hashrate derived
|
||
from it is wrong without ever looking broken. `digest::u512_le` has the test.
|
||
|
||
**A measured block interval is only valid at the tip.** A syncing node imports
|
||
history at disk speed; dividing difficulty by *that* gives a hashrate wrong by
|
||
orders of magnitude that still renders as a number. `RollingWindow::push` takes
|
||
an explicit `at_tip` flag and backfilled blocks contribute no timing sample. The
|
||
`Interval` enum exists so a caller cannot forget to say which kind it has.
|
||
|
||
**A miner's hashrate is its summed work over time, never its share of the
|
||
network.** Difficulty *is* expected hashes per block, so the difficulty of the
|
||
blocks a miner won, divided by the time they spanned, is its hashrate directly
|
||
— and each block carries the difficulty in force when it was won.
|
||
|
||
`share of blocks × network hashrate` is only correct while the network is the
|
||
same size across the whole window. On mainnet's launch day difficulty rose
|
||
**346-fold inside one 14,400-block window**, so a miner that won early — when a
|
||
block cost a three-hundredth of what it costs now — had that share multiplied
|
||
by today's network and was credited with **ten times** the hashrate it had
|
||
(25.4 GH/s reported against 2.5 GH/s actual). Both the leaderboard and
|
||
`miner_series` were wrong this way. The headline *network* figure still divides
|
||
current difficulty by the measured interval, because that one is asking "how
|
||
hard is this to win right now" rather than "what did this miner do".
|
||
|
||
The window's duration comes from `authored_at`, not `observed_at`: a stretch
|
||
this process caught up on carries observation times seconds apart for blocks
|
||
minutes apart, and dividing real work by that invents hashrate.
|
||
|
||
**`difficulty` really is expected hashes, and this has been checked three
|
||
ways.** Before "correcting" any hashrate figure, know that the formula is the
|
||
chain's own: `qpow-math::is_valid_nonce` accepts when `hash < U512::MAX /
|
||
difficulty`, `pallet_qpow::verify_nonce_internal` passes it exactly what
|
||
`get_difficulty()` returns, and the hash is uniform — 200,000 samples of
|
||
`qp_poseidon_core::hash_squeeze_twice` gave a mean of 0.4995 of 2^512 against a
|
||
uniform 0.5, and 12.48% below 2^509 against 12.5%. So acceptance is 1/difficulty
|
||
and expected trials is difficulty, exactly.
|
||
|
||
**A per-miner hashrate is a Poisson estimate and needs its error bar.** The
|
||
estimator is unbiased; that is not the same as accurate. `k` blocks carries a
|
||
relative standard error of `1/√k` — ±100% at one block, ±71% at two, and only
|
||
±20% at twenty-five. The leaderboard printed three significant figures off a
|
||
single block for a miner last seen four hours earlier. `miner_hashrate` returns
|
||
the count's error alongside the figure and the UI renders it; do not drop it.
|
||
|
||
**Our own miners disagree with the chain by ~3.5× and it is not the observer.**
|
||
Measured 2026-09-09: beast 1.116, benjy 0.461, quadbrat 0.082 GH/s — 1.66 GH/s
|
||
total, and 1.39 GH/s instantaneous over a 90 s sample. Over 29 blocks (±18%) in
|
||
6.9 h the chain credited the same preimage 1.32e14 expected hashes, i.e.
|
||
5.3 GH/s. Sampling does not explain a 14-sigma gap. `miner_cpu_hash_rate` is
|
||
**0** on all three hosts — every hash is on GPU — and while the miner's CPU path
|
||
calls `qpow_math::get_nonce_hash` directly, the CUDA kernel is a separate
|
||
implementation with its own counter. That is where to look, not here.
|
||
|
||
**Windows are block counts, never durations.** See `readme.md`. Anything that
|
||
turns `Window` into a time range is wrong.
|
||
|
||
**The difficulty of block *h* lives in the state of block *h-1*.**
|
||
`pallet_qpow::on_finalize` retargets at the *end* of every block, so
|
||
`QPoWApi_get_difficulty` at the latest state returns what the *next* block must
|
||
meet, not what the tip met. Recording that against the block just seen is off by
|
||
one, which on a gradual retarget is a number that still looks entirely right.
|
||
`RpcClient::difficulty_for_child_of` asks at `header.parent_hash`, which is the
|
||
same call the miner made when it built the block. It needs state the node may
|
||
have pruned; an error there is `None`, never a fallback to the current value —
|
||
substituting it is how a gap fill stamps today's difficulty across a stretch of
|
||
old heights and draws a flat plateau in the history chart.
|
||
|
||
**Attribution confidence is measured over votes *cast*, not over the window.**
|
||
`votes` holds one slot per block checked and a `None` for every block whose
|
||
first reporter the feed could not separate from its second. Those are
|
||
abstentions. Dividing agreement by the window length counts them as votes
|
||
against, and on a chain where clean leads are the minority that makes
|
||
`MIN_CONFIDENCE` unreachable: mainnet resolves a lead on about a quarter of
|
||
blocks, which capped every author at 0.25 against a 0.6 bar and named nobody,
|
||
however unanimous the cast votes were. `MIN_VOTES` is the companion guard, so
|
||
one lucky vote in a silent window is not 100% agreement.
|
||
|
||
Two things that look like this bug and are not: the feed sends a **190-node
|
||
roster before any live block traffic** — `ImportedBlock` does not start until
|
||
about 40 s after `subscribe:`, so any probe shorter than that sees zero and
|
||
concludes the feed is dead. And the roster's `AddedNode` entries carry a
|
||
placeholder block of height 0 with a zero hash; counting those as import
|
||
reports inflates the apparent number of nodes claiming to be first.
|
||
|
||
**`observed_at` is only a measurement when `at_tip` is true.** The flag is now a
|
||
column (`0002`). A gap fill writes a whole batch of `observed_at` within the same
|
||
second, so any query treating it as timing data must filter on it. The chain
|
||
series sidesteps this entirely by using `authored_at`, the chain's own clock.
|
||
|
||
**`chain_subscribeNewHeads` skips blocks.** It reports the *best* head, so when
|
||
several import at once the intermediates never arrive. It is a liveness signal,
|
||
not a ledger — `ingest` fills gaps against `chain_getBlockHash`. Removing that
|
||
would quietly under-count exactly the miners who won blocks during a burst.
|
||
|
||
**Blocks are keyed on `(chain, height)`, not hash.** This chain reorgs; the
|
||
upsert is what makes a replacement overwrite rather than accumulate. A schema
|
||
keyed on the hash would inflate the losing fork's author forever.
|
||
|
||
**A reorg copies the old row into `block_displacement` first** (`0003`), in the
|
||
same statement as the upsert. That is not optional bookkeeping: the node cannot
|
||
be asked for the block later. `blocks-pruning` defaults to `archive-canonical`,
|
||
so non-canonical bodies go once finality passes them — about a hundred blocks
|
||
behind the tip on Planck — and difficulty is a state read against a 256-block
|
||
default. Whatever `record_blocks` captures is all there will ever be. The two
|
||
halves are one statement because every data-modifying CTE sees the same
|
||
snapshot, so the copy reads the pre-update row; splitting them would need an
|
||
explicit transaction to be equivalent.
|
||
|
||
**A block decoded against the wrong runtime must fail, not succeed.** The whole
|
||
metadata-oracle approach rests on `Runtime::decode_events` refusing a partial
|
||
read. Events are a `Vec<EventRecord>` and SCALE is self-describing only against
|
||
a registry, so a nearly-right registry decodes a nearly-right answer and leaves
|
||
bytes over. That leftover is the *only* signal an upgrade boundary was crossed —
|
||
it is what makes trying the tip's runtime speculatively safe, and what triggers
|
||
`runtime_at` to ask which runtime actually produced the block. Relaxing that
|
||
check to tolerate trailing bytes would turn a loud failure into silently wrong
|
||
rewards.
|
||
|
||
**An extrinsic's first byte is not its version.** The top two bits are a type
|
||
tag — `0b10` signed, `0b00` bare, `0b01` general — and only the low six are the
|
||
version. Mainnet emits `0x84` (signed, v4) and `0x05` (bare, v**5**) in the same
|
||
block while the metadata declares extrinsic version 4. A decoder that reads the
|
||
byte as a version and checks it against the metadata rejects every timestamp
|
||
inherent on the chain.
|
||
|
||
**Never store a signature.** Dilithium is **5.3 KiB** per transaction — two
|
||
orders of magnitude larger than the call it authorises. `decode_extrinsic`
|
||
decodes it because that is the only way to know where the call begins, then
|
||
keeps the scheme's *name* and the byte count and discards the bytes. At a block
|
||
a second, storing them would be gigabytes a month that nothing renders.
|
||
|
||
**A length prefix out of a blob must never size an allocation.**
|
||
`scale_value` decodes a sequence by doing `Vec::with_capacity(remaining())`
|
||
*before* it decodes the first item, where `remaining()` is the compact length
|
||
read straight out of the bytes. A `Value<u32>` is 80 bytes, so a blob that
|
||
disagrees with the registry can ask for any allocation at all — and a failed
|
||
allocation aborts the process rather than returning an error. Deployed, that was
|
||
`memory allocation of 82014765760 bytes failed`, 316 times in ten hours, from a
|
||
claimed length of 1,025,184,572. The step that would have caught the mismatch is
|
||
the one that never runs, so `.ok()` at the call site catches nothing: there is no
|
||
`Err`, only SIGABRT. Every decode therefore goes through
|
||
`Runtime::decode_checked`, which walks the bytes first with `scale_decode`'s
|
||
`IgnoreVisitor` — that crate contains no `with_capacity` anywhere, so an
|
||
impossible length runs out of input on the first item and comes back as an
|
||
error. This is what makes "a block decoded against the wrong runtime must fail"
|
||
true in the case where it previously did neither. Do not route a new decode
|
||
around it.
|
||
|
||
**A catch-up that writes once at the end makes no progress at all.** `fill_gap`
|
||
used to accumulate the whole gap and call `record_blocks` after the loop, so
|
||
anything that ended the process first discarded every block collected. Each
|
||
block is four RPC round trips, one a historical state read, and this service is
|
||
restarted several times a day by `blackbeard-api-cert.path` alone — so a gap
|
||
wider than the interval between routine restarts was permanently unfillable
|
||
while logging `filling a gap in the head stream` on every start, always with the
|
||
same `from`, reading exactly like progress. Both testnets sat a day behind that
|
||
way. It flushes every `GAP_FLUSH_BLOCKS` now; anything that walks history in
|
||
this codebase has to be resumable, because the process is not long-lived.
|
||
|
||
**Backfill progress is a cursor, not `max(height) from chain_event`.** A block
|
||
with no indexed event and a block never looked at give the same answer, and the
|
||
first is the common case — `INDEXED_EVENTS` is deliberately narrow. Inferring
|
||
progress from the rows makes the backfill re-read the same empty stretch
|
||
forever. `event_scan` holds one contiguous interval, and each batch is walked in
|
||
the direction that keeps it contiguous if it stops early: upward when extending
|
||
`high`, downward when extending `low`. Walking a downward batch upward leaves the
|
||
unread remainder *between* what was just read and `low`, which the cursor cannot
|
||
express and nothing would ever notice.
|
||
|
||
**Historical `state_getRuntimeVersion` costs seconds, not milliseconds.** It
|
||
makes the node load and instantiate the runtime WASM out of that block's state:
|
||
measured at ~4 s against Heisenberg's public endpoint versus ~0.25 s at the tip.
|
||
`discover_runtimes` is its own task for exactly this reason — a bisection is
|
||
~100 of those calls, and inside the four-second poll loop it stalls difficulty,
|
||
the summary broadcast and the telemetry attach for minutes on every start. The
|
||
10 s `RPC_TIMEOUT` is what makes those calls succeed at all; shortening it would
|
||
make runtime history quietly stop working while everything else stayed fine.
|
||
|
||
**`first_seen_height = 0` means "not known", not genesis.** A runtime cached on
|
||
the very first poll is recorded before the header subscription has delivered a
|
||
height. `lower_first_seen` uses `least`, so a zero left in place would win
|
||
forever and claim every runtime began at block 0 — it special-cases zero for
|
||
that reason. Do not simplify it back to a plain `least`.
|
||
|
||
**An account is told from a hash by registry path, never by shape.** After
|
||
`normalise` an `AccountId32` and an `H256` are both `0x` plus sixty-four hex
|
||
characters. `decode_typed` returns the account set the decoder collected while
|
||
it still knew, and that is the only sound source — filtering the rendered JSON
|
||
by shape claims `System::ParentHash` and `ZkTree::Root` name accounts.
|
||
|
||
**A storage entry names an account only if its value *is* one.** `System::Events`
|
||
is full of accounts and names none of them: they are a payload it holds for one
|
||
block. The first cut of the roles index treated "contains an account" as a role
|
||
and labelled half the chain's active addresses `Events`. The test is
|
||
`value` being a string that is itself in the account set.
|
||
|
||
**A JSON-RPC error is the node answering; a transport failure is not.**
|
||
`RpcClient::call` fails over to the next endpoint on the second and never on the
|
||
first. Moving on a real error would hide a caller's own mistake behind a second
|
||
node making the same complaint — and would walk the whole endpoint list to do
|
||
it. `DataError::Rpc` means "answered, with an error" and stops; `Malformed`,
|
||
`Http` and timeouts mean "did not answer" and move on.
|
||
|
||
**A pruned block is a success, not an unhealthy endpoint.** `state_getStorage`
|
||
for state a node has dropped returns `{"result": null}`, which reaches the
|
||
caller as `Null`. Treating that as a failure would fail over through every
|
||
endpoint asking a question none of them can answer, and turn a legitimate
|
||
absence into an outage.
|
||
|
||
**Which is why depth is a property of the endpoint, not of the response.** Null
|
||
is also the right answer for most keys read here — an account with no balance,
|
||
an item never set, the treasury's `System::Account` entry. So `classify_depth`
|
||
probes each endpoint once at startup, reading `System::Number` at block one: an
|
||
endpoint that answers holds everything after it. Reads that name a block hash go
|
||
through `call_deep`, which tries the sticky endpoint first — costing nothing in
|
||
the common case — and asks an archive for a second opinion only when a
|
||
*non-archive* endpoint returned null, or refused with an RPC error, which is
|
||
what `state_call` does when it cannot execute against dropped state. A null from
|
||
an archive is the answer and stops there. An endpoint that could not be probed
|
||
stays `Unknown` and routes as pruned: depth is demonstrated, never assumed, or
|
||
an unreachable host silently becomes the site's archive of record. All seven
|
||
configured endpoints probe as archives today, so none of this changes any
|
||
current behaviour — it is what keeps history from quietly going blank the day
|
||
one of them stops being one.
|
||
|
||
**Endpoints are stuck to, never round-robined.** A storage read at an old block
|
||
hash needs a node that still holds that block's state, and nodes prune on their
|
||
own schedules — alternating between them returns a mixture of answers and
|
||
absences that reads as sparse data rather than as a configuration problem. The
|
||
cursor is shared across clones so a failover one task discovers is not
|
||
rediscovered by every other task on that chain.
|
||
|
||
**`state_getKeysPaged` rejects a count over 1,000, it does not truncate.**
|
||
Asking for 1,600 is an RPC *error*, and a caller that `unwrap_or_default()`s it
|
||
has turned "could not ask" into "the answer is nothing" — which for a map like
|
||
`PendingTransfers` reads exactly like a chain with nothing in flight.
|
||
`MAX_KEYS_PER_PAGE` clamps it in `rpc.rs`, because the ceiling is the node's
|
||
rather than a caller's preference. More generally: `unwrap_or_default()` on
|
||
anything that talks to a node is how a wrong answer comes to look like a right
|
||
one.
|
||
|
||
**A wrong storage hasher reads as absent, not as an error.** A key is
|
||
`twox128(pallet prefix) ++ twox128(item) ++ hashed keys`, and each key's hash is
|
||
declared per entry — `System::Account` is `Blake2_128Concat`, other maps are
|
||
`Twox64Concat`. Get it wrong and `state_getStorage` returns null, which is
|
||
indistinguishable from an empty entry. `Runtime::storage_key` therefore takes
|
||
every part from the metadata, including `PalletStorageMetadata.prefix`, which is
|
||
usually the pallet name and is **not required to be**. Its test pins two keys
|
||
against ones read off the live chain by hand; keep it that way, because nothing
|
||
else would catch a mistake here.
|
||
|
||
**Absent is not zero, and only the modifier knows which.** An `optional` entry
|
||
holding nothing means nothing. A `default` entry holding nothing means the
|
||
runtime's default, which is a value the chain never wrote. `StorageTarget`
|
||
carries the default so a caller can tell them apart, and `read_balance`
|
||
deliberately does *not* apply it: `AccountInfo` defaults to a zeroed struct, and
|
||
Substrate reaps empty accounts, so falling back would turn "this account does
|
||
not exist" into "this account holds nothing". The treasury address on mainnet is
|
||
exactly that case.
|
||
|
||
## Facts established by measurement
|
||
|
||
Taken from the live **Planck** chain, 2026-09-04. Don't re-derive or contradict
|
||
without re-measuring.
|
||
|
||
**Planck is no longer the tracked chain.** Mainnet launched 2026-09-09 and the
|
||
node on bob was switched to it; the `[[chains]]` entry followed the node, so
|
||
`quantus` is now what that endpoint serves and Planck is telemetry-listed only.
|
||
The decoding facts below still hold — same runtime family, same digest shape —
|
||
but every *timing* figure was measured on Planck against a 6 s target and says
|
||
nothing about mainnet, which targets 12 s.
|
||
|
||
- Header digest shape is exactly `0x06` + `706f775f` + compact `0x80` (32) +
|
||
32-byte preimage. Genesis is the only header without one; 12/12 recent headers
|
||
decoded.
|
||
- Telemetry propagation: first reporter stamped `0`, others 50–620 ms. The
|
||
`ATTRIBUTION_LEAD_MS = 20` threshold only has to exclude a tie — identity over
|
||
many blocks does the discriminating.
|
||
- Planck's observed block interval is ~13–15 s against a 6 s target, with
|
||
consecutive gaps of 1.6 s, 4.4 s, 13.5 s, 26 s, 27 s. This is why
|
||
`MIN_TIP_SAMPLES` is 20: at five samples the headline hashrate swings by a
|
||
factor of three between refreshes.
|
||
- `system_properties`: `PLK`, 12 decimals, ss58 prefix 189. Genesis
|
||
`0x4901bf5c…e65e72`.
|
||
- Mainnet, read from bob 2026-09-09: `system_chain` "Quantus", token `QTC`, 12
|
||
decimals, ss58 189, genesis `0xfb5487c0…626fba`. **Its target block time is
|
||
12 s, not Planck's 6** — `TARGET_BLOCK_TIME_MS = 12_000` at commit `b017e642`,
|
||
which is the `0.11.1-b017e6420aa` the node reports. Check the constant at the
|
||
commit the node actually runs, not at the tip of the chain repo: this figure
|
||
is the denominator for every hashrate shown before twenty tip samples exist,
|
||
so getting it wrong publishes a headline that is wrong by exactly that factor
|
||
and labelled "nominal" rather than broken.
|
||
- The telemetry feed URL is `wss://feed-telemetry.quantus.cat/feed` — found in
|
||
`/tmp/env-config.js` on the telemetry site, **not** the `/feed` path on the
|
||
main host, which 404s.
|
||
|
||
**A miner's reward address is derivable from its preimage, offline.**
|
||
`pallets/mining-rewards` pays `qp_wormhole::derive_wormhole_address(preimage)`,
|
||
which is `qp_poseidon_core::rehash_to_bytes` — Poseidon2 over Goldilocks.
|
||
`blackbeard-core::wormhole` runs the same function and SS58-encodes it with
|
||
prefix 189. It takes the preimage and **nothing else**: no genesis, no chain
|
||
parameters, so one preimage is one address on every Quantus chain. That is also
|
||
what makes carrying a name across chains sound. The test pins it against a real
|
||
pair (`baba-gorchitsa`), so a `qp-poseidon-core` upgrade that changed the
|
||
derivation fails the build rather than silently relabelling every miner.
|
||
|
||
An unnamed miner therefore renders as its **address**, not its preimage — both
|
||
identify it exactly, and only one is a string its owner has ever seen. Do not
|
||
infer "is this a name or an address?" from the string: several real node names
|
||
begin with `q`, so `RecentBlock` and `LeaderboardRow` carry the
|
||
`AttributionSource` and the UI reads that.
|
||
|
||
## The frontend's types are generated
|
||
|
||
`web/src/api/generated/` is written by `ts-rs` from `blackbeard-entities` when
|
||
`cargo test -p blackbeard-entities` runs. **Never edit those files.** CI fails
|
||
the build if the committed output is stale.
|
||
|
||
`u64` maps to `bigint` by default, which is wrong — `serde_json` puts a u64 on
|
||
the wire as a JSON *number*, so the type and the runtime value disagree. Every
|
||
u64 DTO field carries `#[ts(type = "number")]`. Add it to any new one.
|
||
|
||
## A restart is visible, so warm start has to put everything back
|
||
|
||
`blackbeard-api-cert.path` watches the host certificate and issues a full
|
||
`systemctl restart` when it rotates — several times a day, on top of every
|
||
deploy. Two pieces of state live only in memory and used to come back empty:
|
||
|
||
- The **block ticker**, which is built from live blocks and has no other source.
|
||
It returned holding one row and refilled over the following minute.
|
||
- The **tip samples** behind the measured interval, because the window replay
|
||
pushed everything with `at_tip: false`. Right for blocks a previous process
|
||
caught up on; wrong for the ones it genuinely watched arrive, which `0002`
|
||
has recorded ever since. Without them the headline hashrate falls back to the
|
||
*target* — at mainnet's interval that is an eight-fold drop, correctly
|
||
labelled nominal and reported as a bug every time.
|
||
|
||
`restore_ticker_and_timing` puts both back, so a restart is no longer
|
||
user-visible. It runs *after* the attributions and carried names, because the
|
||
ticker rows need their display names. Restored tip samples are capped at ten
|
||
minutes old: the interval stays arithmetically valid across a gap, but it stops
|
||
being *current*, and a sample from an hour ago averaged with one from now
|
||
describes the hour rather than the tip.
|
||
|
||
Worth knowing: the store already recycles pooled connections hourly precisely
|
||
so a rotated certificate is a routine reconnect. The path unit's restart is
|
||
therefore belt-and-braces over a case the pool already handles.
|
||
|
||
## Separation of concerns is load-bearing here
|
||
|
||
- `blackbeard-core` has **no I/O, no clock, no sockets**. That is what lets the
|
||
decoding and the statistics — the two parts that are easy to get subtly wrong
|
||
and hard to notice — be exercised by unit tests instead of only against a live
|
||
chain. Keep it that way; if something there needs the time, pass it in.
|
||
- `blackbeard-data` owns retries, reconnects and schema.
|
||
- `blackbeard-api` wires them together and owns nothing else.
|
||
|
||
## Locking
|
||
|
||
`ChainRuntime::inner` is a **`std::sync::RwLock`, never held across an `.await`**.
|
||
Everything under it is CPU work on in-memory collections. Do not switch it to
|
||
tokio's — that variant makes it easy to accidentally hold a lock across an await
|
||
and stall a worker, for no benefit here.
|
||
|
||
## Database work
|
||
|
||
Queries use `sqlx::query!` (compile-time checked) with the offline cache in
|
||
`.sqlx/` committed, so CI builds with `SQLX_OFFLINE=true` and no database.
|
||
|
||
**After changing any query, regenerate the cache** or CI fails on a stale one:
|
||
|
||
```sh
|
||
podman run -d --rm --name bb-pg -e POSTGRES_PASSWORD=dev \
|
||
-e POSTGRES_DB=blackbeard -p 55432:5432 docker.io/library/postgres:18-alpine
|
||
export DATABASE_URL='postgres://postgres:dev@127.0.0.1:55432/blackbeard'
|
||
cargo sqlx prepare --workspace -- --all-targets
|
||
```
|
||
|
||
Migrations are sequentially versioned and **immutable once committed**. Correct a
|
||
mistake with a new file, never by editing one that has landed — the runner's
|
||
checksum diverges and it refuses to start.
|
||
|
||
## Charts
|
||
|
||
Any chart added here must be read against the `dataviz` skill first. The
|
||
constraint already in force: **bronze `#bd8829` and crimson `#d8453a` are
|
||
adjacent hues and fail CVD separation as a categorical pair.** Every chart on the
|
||
site is therefore single-series — magnitude, one hue — with identity carried by
|
||
labels and row treatment. A second series in the accent colour is the one change
|
||
that would break the palette, and it would not look broken.
|
||
|
||
This is why the stat-tile sparklines do not follow the usual "current period in
|
||
the accent" convention: the newest point is emphasised with `--data-bright`, a
|
||
lighter step of the same hue. They also do not start at zero — a sparkline is
|
||
read for shape — which is why the hover readout and the `aria-label` both carry
|
||
real numbers, and why the readout borrows the tile's note line instead of
|
||
floating a tooltip: five overlays in a row of 180px tiles is not a hover layer,
|
||
it is a pile.
|
||
|
||
`web/src/index.css` documents the validated values. Re-run the validator after
|
||
touching them:
|
||
|
||
```sh
|
||
node <skill>/scripts/validate_palette.js "#bd8829" --mode dark --surface "#14110d"
|
||
```
|
||
|
||
## Deployment gotchas learned the hard way
|
||
|
||
**A sudoers grant matches the whole argument vector.** The `restorecon` grant
|
||
names three paths; running it with one path is a different command and asks for
|
||
a password. Run the command exactly as `infra-setup.sh` grants it, which is what
|
||
the workflow does.
|
||
|
||
**The config check runs as `blackbeard`, not root** — runas `(blackbeard)` in
|
||
sudoers. A root-run check passes on a config the service account cannot read,
|
||
which is the failure it exists to catch.
|
||
|
||
**`CorsLayer::allow_origin` replaces, it does not append.** Folding over a list
|
||
of origins leaves only the last one allowed. It does not break the site — the
|
||
frontend is same-origin and never consults CORS — so it is invisible until
|
||
something else tries to call the API. Use `AllowOrigin::list`.
|
||
|
||
**The API binds the host's mesh address, never loopback and never `0.0.0.0`.**
|
||
The fleet has one firewalld default zone, so a wildcard bind plus the named
|
||
service would publish it on every address the host carries; and the edge proxy
|
||
is at another site, so loopback-only would not serve the site at all. Anything
|
||
probing `127.0.0.1:25864` gets connection-refused against a perfectly healthy
|
||
daemon — which is exactly how the first green build still failed its health
|
||
check. Use `$(hostname -f)`, which resolves to the mesh address.
|
||
|
||
**The edge proxy is at a different site from the API.** oolon (kosherinata)
|
||
fronts the name; the API is on bob (hanzalova) because that is where the Planck
|
||
node lives. A health probe on the API host cannot see that hop, so there is a
|
||
separate check that curls the API *from oolon*. A firewalld service scoped to
|
||
bob's own /16 would leave a live site with a dead `/v1` and nothing would fail.
|
||
|
||
**The edge proxy cannot reach its own public name.** From inside the mesh
|
||
`blackbeard.observer` resolves to the site's WAN address and dead-ends on the
|
||
OPNsense LAN interface (`reverse-proxies.md` §2). Verify the vhost with
|
||
`--resolve blackbeard.observer:443:127.0.0.1`, which still exercises the `:443`
|
||
stream router, SNI, the vhost, the cross-site hop and the API.
|
||
|
||
**A frontend-only push skips the Rust build and the API deploy.** The `what
|
||
changed` step in `deploy.yaml` diffs against `github.event.before` and sets one
|
||
output; the gate, the musl build, the ts-rs drift check and the whole
|
||
`deploy-api` job hang off it. Two entries in that path list are not obvious and
|
||
must not be trimmed: `asset/`, because `deploy-api` ships the systemd units, the
|
||
firewalld service and the rendered config from it; and
|
||
`web/src/api/generated/`, because the drift gate only runs when `cargo test` has
|
||
regenerated those files, so a hand-edit of them must not be able to arrive
|
||
labelled "frontend only" — which is precisely the change that gate exists to
|
||
catch. Anything unrecognised counts as Rust: a false positive costs a slow
|
||
deploy, a false negative leaves a binary on bob that does not match the commit
|
||
and nothing reports it.
|
||
|
||
**Runner labels.** `fedora-*` images have **no cargo** — Rust builds go on
|
||
`rust`, which is based on `runner-fedora-44` and so carries node + pnpm too.
|
||
Never `corepack enable`; pnpm is already on PATH (`gitea-runners.md` §4).
|
||
|
||
**WebSocket upgrade tests need `--http1.1`.** The vhost serves HTTP/2, and curl
|
||
negotiates h2, where the `Connection: Upgrade` handshake is not how WebSockets
|
||
work — you get a 400 from axum that looks like a proxy misconfiguration.
|
||
|
||
## Verifying a change
|
||
|
||
`systemctl is-active` is not evidence this works. A daemon with a node it cannot
|
||
decode, or a telemetry feed it silently ignores, is perfectly "active" while
|
||
serving an empty leaderboard. Check the numbers:
|
||
|
||
```sh
|
||
cargo run -p blackbeard-cli -- probe --rpc-url http://bob.hanzalova.internal:9944
|
||
cargo run -p blackbeard-cli -- standings --chain quantus
|
||
curl -s localhost:25864/v1/chains/quantus/summary | python3 -m json.tool
|
||
```
|
||
|
||
The block route degrades rather than failing, so it is worth checking both ends
|
||
of that: a recent height should carry `difficulty`, `extrinsics` and
|
||
`seconds_since_parent`, while one a few days old keeps the body and loses
|
||
difficulty to state pruning unless this observer recorded it at the time.
|
||
|
||
```sh
|
||
curl -s localhost:25864/v1/chains/quantus/blocks/1000 | python3 -m json.tool
|
||
```
|
||
|
||
A working deployment shows a non-null `telemetry_nodes`, `distinct_miners` above
|
||
one, and named rows in the standings.
|