Files
observer/CLAUDE.md
T
grenadeandClaude Opus 5 9def0c8b6a
deploy / build (push) Successful in 7m53s
deploy / deploy-web (push) Successful in 5s
deploy / deploy-api (push) Successful in 15s
feat: route reads that need old state to an endpoint that has it
Failover moved on a transport failure and never on a JSON-RPC error, both
right — but a node that has pruned the state being asked for does neither. It
answers `{"result": null}`, which is a success, so whichever endpoint happened
to be sticky decided how far back the whole site could see.

The tempting fix is wrong. Null is also the correct answer for most keys read
here: an account with no balance, an item never set, the treasury's
`System::Account` entry, absent because the treasury was never funded. Retrying
every one of those against every endpoint would multiply the load to re-derive
an answer already in hand, and still end in null. The distinction is not in the
response — it is whether the node could have known, which is a property of the
endpoint.

So `classify_depth` probes each one at startup, reading `System::Number` at
block one. That key is derived in a test rather than pasted, because a mistyped
key is absent everywhere and would classify every endpoint as pruned. An
endpoint that answers holds all state after block one; one that cannot be
reached stays `Unknown` and routes as pruned, since an unreachable host must not
become the archive of record by default.

Reads naming a block hash then go through `call_deep`. It tries the sticky
endpoint first, so the common case — mainnet's loopback node, which is an
archive — costs no extra round trip. Only a null from an endpoint never shown to
hold old state buys a second opinion, and `state_call`'s refusal is treated the
same way, because that is how difficulty is read and a node that cannot execute
against dropped state errors rather than returning null.

All seven configured endpoints probe as archives right now, so this changes no
current behaviour. It is what stops history going quietly blank the day one of
them stops being one — and it says so in the log when it happens, because an
endpoint that cannot see as far back as the site is asking is a configuration
fact worth surfacing rather than a chain fact.

Closes #9

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jp6a8EDar9ueEhAxzep4V5
2026-09-10 12:12:11 +03:00

531 lines
30 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CLAUDE.md
Guidance for Claude Code working in this repository.
Read `readme.md` first — it carries the mechanism (how authorship is decoded from
block headers), the reasoning behind the window semantics, and the list of
deliberate deviations from house convention. This file is the things that will
bite you.
Conventions: `~/git/architecture` — `generic.md` for the workspace shape,
`deployment-gitea-actions.md` for the deploy, `port-allocations.md` for the port,
`reverse-proxies.md` and `external-tls.md`/`internal-tls.md` for the vhosts.
## Work is filed before it is done
Most fixes and features start as a Gitea issue on
`quantus/blackbeard.observer`, and the commit that implements one closes it:
```
Closes #12
```
The point is an auditable history — why a thing was built, what was believed at
the time, and what turned out to be wrong on the way. That last part is the
valuable half, so **when an investigation contradicts the issue, comment on the
issue rather than quietly building the right thing**. Issue #1 asserted that
`execute_at` came from `PendingTransfers`; it does not, and the correction is a
comment there, dated, rather than a surprise in a diff.
An issue is not a ceremony for a typo. It is for anything where somebody later
would reasonably ask "why is it like this".
## The things that cost an afternoon
**substrate-telemetry sends its JSON in *binary* WebSocket frames, not text.** A
client that handles only `Message::Text` connects, subscribes, reports itself
healthy, and receives absolutely nothing — no error, no decode failure, an empty
feed and `telemetry_nodes: null`. Worse, a throwaway probe written in Python
*works*, because `json.loads` accepts bytes, so the two disagree for no visible
reason. Both frame types are accepted in `telemetry.rs` and `rpc.rs`. Do not
"simplify" either match arm.
**Difficulty is a little-endian U512.** `state_call` returns a SCALE-encoded
`U512`, byte-reversed relative to how a hash reads. Decoding it big-endian yields
a plausible-looking number that is wrong by ~10^150, and every hashrate derived
from it is wrong without ever looking broken. `digest::u512_le` has the test.
**A measured block interval is only valid at the tip.** A syncing node imports
history at disk speed; dividing difficulty by *that* gives a hashrate wrong by
orders of magnitude that still renders as a number. `RollingWindow::push` takes
an explicit `at_tip` flag and backfilled blocks contribute no timing sample. The
`Interval` enum exists so a caller cannot forget to say which kind it has.
**A miner's hashrate is its summed work over time, never its share of the
network.** Difficulty *is* expected hashes per block, so the difficulty of the
blocks a miner won, divided by the time they spanned, is its hashrate directly
— and each block carries the difficulty in force when it was won.
`share of blocks × network hashrate` is only correct while the network is the
same size across the whole window. On mainnet's launch day difficulty rose
**346-fold inside one 14,400-block window**, so a miner that won early — when a
block cost a three-hundredth of what it costs now — had that share multiplied
by today's network and was credited with **ten times** the hashrate it had
(25.4 GH/s reported against 2.5 GH/s actual). Both the leaderboard and
`miner_series` were wrong this way. The headline *network* figure still divides
current difficulty by the measured interval, because that one is asking "how
hard is this to win right now" rather than "what did this miner do".
The window's duration comes from `authored_at`, not `observed_at`: a stretch
this process caught up on carries observation times seconds apart for blocks
minutes apart, and dividing real work by that invents hashrate.
**`difficulty` really is expected hashes, and this has been checked three
ways.** Before "correcting" any hashrate figure, know that the formula is the
chain's own: `qpow-math::is_valid_nonce` accepts when `hash < U512::MAX /
difficulty`, `pallet_qpow::verify_nonce_internal` passes it exactly what
`get_difficulty()` returns, and the hash is uniform — 200,000 samples of
`qp_poseidon_core::hash_squeeze_twice` gave a mean of 0.4995 of 2^512 against a
uniform 0.5, and 12.48% below 2^509 against 12.5%. So acceptance is 1/difficulty
and expected trials is difficulty, exactly.
**A per-miner hashrate is a Poisson estimate and needs its error bar.** The
estimator is unbiased; that is not the same as accurate. `k` blocks carries a
relative standard error of `1/√k` — ±100% at one block, ±71% at two, and only
±20% at twenty-five. The leaderboard printed three significant figures off a
single block for a miner last seen four hours earlier. `miner_hashrate` returns
the count's error alongside the figure and the UI renders it; do not drop it.
**Our own miners disagree with the chain by ~3.5× and it is not the observer.**
Measured 2026-09-09: beast 1.116, benjy 0.461, quadbrat 0.082 GH/s — 1.66 GH/s
total, and 1.39 GH/s instantaneous over a 90 s sample. Over 29 blocks (±18%) in
6.9 h the chain credited the same preimage 1.32e14 expected hashes, i.e.
5.3 GH/s. Sampling does not explain a 14-sigma gap. `miner_cpu_hash_rate` is
**0** on all three hosts — every hash is on GPU — and while the miner's CPU path
calls `qpow_math::get_nonce_hash` directly, the CUDA kernel is a separate
implementation with its own counter. That is where to look, not here.
**Windows are block counts, never durations.** See `readme.md`. Anything that
turns `Window` into a time range is wrong.
**The difficulty of block *h* lives in the state of block *h-1*.**
`pallet_qpow::on_finalize` retargets at the *end* of every block, so
`QPoWApi_get_difficulty` at the latest state returns what the *next* block must
meet, not what the tip met. Recording that against the block just seen is off by
one, which on a gradual retarget is a number that still looks entirely right.
`RpcClient::difficulty_for_child_of` asks at `header.parent_hash`, which is the
same call the miner made when it built the block. It needs state the node may
have pruned; an error there is `None`, never a fallback to the current value —
substituting it is how a gap fill stamps today's difficulty across a stretch of
old heights and draws a flat plateau in the history chart.
**Attribution confidence is measured over votes *cast*, not over the window.**
`votes` holds one slot per block checked and a `None` for every block whose
first reporter the feed could not separate from its second. Those are
abstentions. Dividing agreement by the window length counts them as votes
against, and on a chain where clean leads are the minority that makes
`MIN_CONFIDENCE` unreachable: mainnet resolves a lead on about a quarter of
blocks, which capped every author at 0.25 against a 0.6 bar and named nobody,
however unanimous the cast votes were. `MIN_VOTES` is the companion guard, so
one lucky vote in a silent window is not 100% agreement.
Two things that look like this bug and are not: the feed sends a **190-node
roster before any live block traffic** — `ImportedBlock` does not start until
about 40 s after `subscribe:`, so any probe shorter than that sees zero and
concludes the feed is dead. And the roster's `AddedNode` entries carry a
placeholder block of height 0 with a zero hash; counting those as import
reports inflates the apparent number of nodes claiming to be first.
**`observed_at` is only a measurement when `at_tip` is true.** The flag is now a
column (`0002`). A gap fill writes a whole batch of `observed_at` within the same
second, so any query treating it as timing data must filter on it. The chain
series sidesteps this entirely by using `authored_at`, the chain's own clock.
**`chain_subscribeNewHeads` skips blocks.** It reports the *best* head, so when
several import at once the intermediates never arrive. It is a liveness signal,
not a ledger — `ingest` fills gaps against `chain_getBlockHash`. Removing that
would quietly under-count exactly the miners who won blocks during a burst.
**Blocks are keyed on `(chain, height)`, not hash.** This chain reorgs; the
upsert is what makes a replacement overwrite rather than accumulate. A schema
keyed on the hash would inflate the losing fork's author forever.
**A reorg copies the old row into `block_displacement` first** (`0003`), in the
same statement as the upsert. That is not optional bookkeeping: the node cannot
be asked for the block later. `blocks-pruning` defaults to `archive-canonical`,
so non-canonical bodies go once finality passes them — about a hundred blocks
behind the tip on Planck — and difficulty is a state read against a 256-block
default. Whatever `record_blocks` captures is all there will ever be. The two
halves are one statement because every data-modifying CTE sees the same
snapshot, so the copy reads the pre-update row; splitting them would need an
explicit transaction to be equivalent.
**A block decoded against the wrong runtime must fail, not succeed.** The whole
metadata-oracle approach rests on `Runtime::decode_events` refusing a partial
read. Events are a `Vec<EventRecord>` and SCALE is self-describing only against
a registry, so a nearly-right registry decodes a nearly-right answer and leaves
bytes over. That leftover is the *only* signal an upgrade boundary was crossed —
it is what makes trying the tip's runtime speculatively safe, and what triggers
`runtime_at` to ask which runtime actually produced the block. Relaxing that
check to tolerate trailing bytes would turn a loud failure into silently wrong
rewards.
**An extrinsic's first byte is not its version.** The top two bits are a type
tag — `0b10` signed, `0b00` bare, `0b01` general — and only the low six are the
version. Mainnet emits `0x84` (signed, v4) and `0x05` (bare, v**5**) in the same
block while the metadata declares extrinsic version 4. A decoder that reads the
byte as a version and checks it against the metadata rejects every timestamp
inherent on the chain.
**Never store a signature.** Dilithium is **5.3 KiB** per transaction — two
orders of magnitude larger than the call it authorises. `decode_extrinsic`
decodes it because that is the only way to know where the call begins, then
keeps the scheme's *name* and the byte count and discards the bytes. At a block
a second, storing them would be gigabytes a month that nothing renders.
**A length prefix out of a blob must never size an allocation.**
`scale_value` decodes a sequence by doing `Vec::with_capacity(remaining())`
*before* it decodes the first item, where `remaining()` is the compact length
read straight out of the bytes. A `Value<u32>` is 80 bytes, so a blob that
disagrees with the registry can ask for any allocation at all — and a failed
allocation aborts the process rather than returning an error. Deployed, that was
`memory allocation of 82014765760 bytes failed`, 316 times in ten hours, from a
claimed length of 1,025,184,572. The step that would have caught the mismatch is
the one that never runs, so `.ok()` at the call site catches nothing: there is no
`Err`, only SIGABRT. Every decode therefore goes through
`Runtime::decode_checked`, which walks the bytes first with `scale_decode`'s
`IgnoreVisitor` — that crate contains no `with_capacity` anywhere, so an
impossible length runs out of input on the first item and comes back as an
error. This is what makes "a block decoded against the wrong runtime must fail"
true in the case where it previously did neither. Do not route a new decode
around it.
**A catch-up that writes once at the end makes no progress at all.** `fill_gap`
used to accumulate the whole gap and call `record_blocks` after the loop, so
anything that ended the process first discarded every block collected. Each
block is four RPC round trips, one a historical state read, and this service is
restarted several times a day by `blackbeard-api-cert.path` alone — so a gap
wider than the interval between routine restarts was permanently unfillable
while logging `filling a gap in the head stream` on every start, always with the
same `from`, reading exactly like progress. Both testnets sat a day behind that
way. It flushes every `GAP_FLUSH_BLOCKS` now; anything that walks history in
this codebase has to be resumable, because the process is not long-lived.
**Backfill progress is a cursor, not `max(height) from chain_event`.** A block
with no indexed event and a block never looked at give the same answer, and the
first is the common case — `INDEXED_EVENTS` is deliberately narrow. Inferring
progress from the rows makes the backfill re-read the same empty stretch
forever. `event_scan` holds one contiguous interval, and each batch is walked in
the direction that keeps it contiguous if it stops early: upward when extending
`high`, downward when extending `low`. Walking a downward batch upward leaves the
unread remainder *between* what was just read and `low`, which the cursor cannot
express and nothing would ever notice.
**Historical `state_getRuntimeVersion` costs seconds, not milliseconds.** It
makes the node load and instantiate the runtime WASM out of that block's state:
measured at ~4 s against Heisenberg's public endpoint versus ~0.25 s at the tip.
`discover_runtimes` is its own task for exactly this reason — a bisection is
~100 of those calls, and inside the four-second poll loop it stalls difficulty,
the summary broadcast and the telemetry attach for minutes on every start. The
10 s `RPC_TIMEOUT` is what makes those calls succeed at all; shortening it would
make runtime history quietly stop working while everything else stayed fine.
**`first_seen_height = 0` means "not known", not genesis.** A runtime cached on
the very first poll is recorded before the header subscription has delivered a
height. `lower_first_seen` uses `least`, so a zero left in place would win
forever and claim every runtime began at block 0 — it special-cases zero for
that reason. Do not simplify it back to a plain `least`.
**An account is told from a hash by registry path, never by shape.** After
`normalise` an `AccountId32` and an `H256` are both `0x` plus sixty-four hex
characters. `decode_typed` returns the account set the decoder collected while
it still knew, and that is the only sound source — filtering the rendered JSON
by shape claims `System::ParentHash` and `ZkTree::Root` name accounts.
**A storage entry names an account only if its value *is* one.** `System::Events`
is full of accounts and names none of them: they are a payload it holds for one
block. The first cut of the roles index treated "contains an account" as a role
and labelled half the chain's active addresses `Events`. The test is
`value` being a string that is itself in the account set.
**A JSON-RPC error is the node answering; a transport failure is not.**
`RpcClient::call` fails over to the next endpoint on the second and never on the
first. Moving on a real error would hide a caller's own mistake behind a second
node making the same complaint — and would walk the whole endpoint list to do
it. `DataError::Rpc` means "answered, with an error" and stops; `Malformed`,
`Http` and timeouts mean "did not answer" and move on.
**A pruned block is a success, not an unhealthy endpoint.** `state_getStorage`
for state a node has dropped returns `{"result": null}`, which reaches the
caller as `Null`. Treating that as a failure would fail over through every
endpoint asking a question none of them can answer, and turn a legitimate
absence into an outage.
**Which is why depth is a property of the endpoint, not of the response.** Null
is also the right answer for most keys read here — an account with no balance,
an item never set, the treasury's `System::Account` entry. So `classify_depth`
probes each endpoint once at startup, reading `System::Number` at block one: an
endpoint that answers holds everything after it. Reads that name a block hash go
through `call_deep`, which tries the sticky endpoint first — costing nothing in
the common case — and asks an archive for a second opinion only when a
*non-archive* endpoint returned null, or refused with an RPC error, which is
what `state_call` does when it cannot execute against dropped state. A null from
an archive is the answer and stops there. An endpoint that could not be probed
stays `Unknown` and routes as pruned: depth is demonstrated, never assumed, or
an unreachable host silently becomes the site's archive of record. All seven
configured endpoints probe as archives today, so none of this changes any
current behaviour — it is what keeps history from quietly going blank the day
one of them stops being one.
**Endpoints are stuck to, never round-robined.** A storage read at an old block
hash needs a node that still holds that block's state, and nodes prune on their
own schedules — alternating between them returns a mixture of answers and
absences that reads as sparse data rather than as a configuration problem. The
cursor is shared across clones so a failover one task discovers is not
rediscovered by every other task on that chain.
**`state_getKeysPaged` rejects a count over 1,000, it does not truncate.**
Asking for 1,600 is an RPC *error*, and a caller that `unwrap_or_default()`s it
has turned "could not ask" into "the answer is nothing" — which for a map like
`PendingTransfers` reads exactly like a chain with nothing in flight.
`MAX_KEYS_PER_PAGE` clamps it in `rpc.rs`, because the ceiling is the node's
rather than a caller's preference. More generally: `unwrap_or_default()` on
anything that talks to a node is how a wrong answer comes to look like a right
one.
**A wrong storage hasher reads as absent, not as an error.** A key is
`twox128(pallet prefix) ++ twox128(item) ++ hashed keys`, and each key's hash is
declared per entry — `System::Account` is `Blake2_128Concat`, other maps are
`Twox64Concat`. Get it wrong and `state_getStorage` returns null, which is
indistinguishable from an empty entry. `Runtime::storage_key` therefore takes
every part from the metadata, including `PalletStorageMetadata.prefix`, which is
usually the pallet name and is **not required to be**. Its test pins two keys
against ones read off the live chain by hand; keep it that way, because nothing
else would catch a mistake here.
**Absent is not zero, and only the modifier knows which.** An `optional` entry
holding nothing means nothing. A `default` entry holding nothing means the
runtime's default, which is a value the chain never wrote. `StorageTarget`
carries the default so a caller can tell them apart, and `read_balance`
deliberately does *not* apply it: `AccountInfo` defaults to a zeroed struct, and
Substrate reaps empty accounts, so falling back would turn "this account does
not exist" into "this account holds nothing". The treasury address on mainnet is
exactly that case.
## Facts established by measurement
Taken from the live **Planck** chain, 2026-09-04. Don't re-derive or contradict
without re-measuring.
**Planck is no longer the tracked chain.** Mainnet launched 2026-09-09 and the
node on bob was switched to it; the `[[chains]]` entry followed the node, so
`quantus` is now what that endpoint serves and Planck is telemetry-listed only.
The decoding facts below still hold — same runtime family, same digest shape —
but every *timing* figure was measured on Planck against a 6 s target and says
nothing about mainnet, which targets 12 s.
- Header digest shape is exactly `0x06` + `706f775f` + compact `0x80` (32) +
32-byte preimage. Genesis is the only header without one; 12/12 recent headers
decoded.
- Telemetry propagation: first reporter stamped `0`, others 50–620 ms. The
`ATTRIBUTION_LEAD_MS = 20` threshold only has to exclude a tie — identity over
many blocks does the discriminating.
- Planck's observed block interval is ~13–15 s against a 6 s target, with
consecutive gaps of 1.6 s, 4.4 s, 13.5 s, 26 s, 27 s. This is why
`MIN_TIP_SAMPLES` is 20: at five samples the headline hashrate swings by a
factor of three between refreshes.
- `system_properties`: `PLK`, 12 decimals, ss58 prefix 189. Genesis
`0x4901bf5c…e65e72`.
- Mainnet, read from bob 2026-09-09: `system_chain` "Quantus", token `QTC`, 12
decimals, ss58 189, genesis `0xfb5487c0…626fba`. **Its target block time is
12 s, not Planck's 6** — `TARGET_BLOCK_TIME_MS = 12_000` at commit `b017e642`,
which is the `0.11.1-b017e6420aa` the node reports. Check the constant at the
commit the node actually runs, not at the tip of the chain repo: this figure
is the denominator for every hashrate shown before twenty tip samples exist,
so getting it wrong publishes a headline that is wrong by exactly that factor
and labelled "nominal" rather than broken.
- The telemetry feed URL is `wss://feed-telemetry.quantus.cat/feed` — found in
`/tmp/env-config.js` on the telemetry site, **not** the `/feed` path on the
main host, which 404s.
**A miner's reward address is derivable from its preimage, offline.**
`pallets/mining-rewards` pays `qp_wormhole::derive_wormhole_address(preimage)`,
which is `qp_poseidon_core::rehash_to_bytes` — Poseidon2 over Goldilocks.
`blackbeard-core::wormhole` runs the same function and SS58-encodes it with
prefix 189. It takes the preimage and **nothing else**: no genesis, no chain
parameters, so one preimage is one address on every Quantus chain. That is also
what makes carrying a name across chains sound. The test pins it against a real
pair (`baba-gorchitsa`), so a `qp-poseidon-core` upgrade that changed the
derivation fails the build rather than silently relabelling every miner.
An unnamed miner therefore renders as its **address**, not its preimage — both
identify it exactly, and only one is a string its owner has ever seen. Do not
infer "is this a name or an address?" from the string: several real node names
begin with `q`, so `RecentBlock` and `LeaderboardRow` carry the
`AttributionSource` and the UI reads that.
## The frontend's types are generated
`web/src/api/generated/` is written by `ts-rs` from `blackbeard-entities` when
`cargo test -p blackbeard-entities` runs. **Never edit those files.** CI fails
the build if the committed output is stale.
`u64` maps to `bigint` by default, which is wrong — `serde_json` puts a u64 on
the wire as a JSON *number*, so the type and the runtime value disagree. Every
u64 DTO field carries `#[ts(type = "number")]`. Add it to any new one.
## A restart is visible, so warm start has to put everything back
`blackbeard-api-cert.path` watches the host certificate and issues a full
`systemctl restart` when it rotates — several times a day, on top of every
deploy. Two pieces of state live only in memory and used to come back empty:
- The **block ticker**, which is built from live blocks and has no other source.
It returned holding one row and refilled over the following minute.
- The **tip samples** behind the measured interval, because the window replay
pushed everything with `at_tip: false`. Right for blocks a previous process
caught up on; wrong for the ones it genuinely watched arrive, which `0002`
has recorded ever since. Without them the headline hashrate falls back to the
*target* — at mainnet's interval that is an eight-fold drop, correctly
labelled nominal and reported as a bug every time.
`restore_ticker_and_timing` puts both back, so a restart is no longer
user-visible. It runs *after* the attributions and carried names, because the
ticker rows need their display names. Restored tip samples are capped at ten
minutes old: the interval stays arithmetically valid across a gap, but it stops
being *current*, and a sample from an hour ago averaged with one from now
describes the hour rather than the tip.
Worth knowing: the store already recycles pooled connections hourly precisely
so a rotated certificate is a routine reconnect. The path unit's restart is
therefore belt-and-braces over a case the pool already handles.
## Separation of concerns is load-bearing here
- `blackbeard-core` has **no I/O, no clock, no sockets**. That is what lets the
decoding and the statistics — the two parts that are easy to get subtly wrong
and hard to notice — be exercised by unit tests instead of only against a live
chain. Keep it that way; if something there needs the time, pass it in.
- `blackbeard-data` owns retries, reconnects and schema.
- `blackbeard-api` wires them together and owns nothing else.
## Locking
`ChainRuntime::inner` is a **`std::sync::RwLock`, never held across an `.await`**.
Everything under it is CPU work on in-memory collections. Do not switch it to
tokio's — that variant makes it easy to accidentally hold a lock across an await
and stall a worker, for no benefit here.
## Database work
Queries use `sqlx::query!` (compile-time checked) with the offline cache in
`.sqlx/` committed, so CI builds with `SQLX_OFFLINE=true` and no database.
**After changing any query, regenerate the cache** or CI fails on a stale one:
```sh
podman run -d --rm --name bb-pg -e POSTGRES_PASSWORD=dev \
-e POSTGRES_DB=blackbeard -p 55432:5432 docker.io/library/postgres:18-alpine
export DATABASE_URL='postgres://postgres:dev@127.0.0.1:55432/blackbeard'
cargo sqlx prepare --workspace -- --all-targets
```
Migrations are sequentially versioned and **immutable once committed**. Correct a
mistake with a new file, never by editing one that has landed — the runner's
checksum diverges and it refuses to start.
## Charts
Any chart added here must be read against the `dataviz` skill first. The
constraint already in force: **bronze `#bd8829` and crimson `#d8453a` are
adjacent hues and fail CVD separation as a categorical pair.** Every chart on the
site is therefore single-series — magnitude, one hue — with identity carried by
labels and row treatment. A second series in the accent colour is the one change
that would break the palette, and it would not look broken.
This is why the stat-tile sparklines do not follow the usual "current period in
the accent" convention: the newest point is emphasised with `--data-bright`, a
lighter step of the same hue. They also do not start at zero — a sparkline is
read for shape — which is why the hover readout and the `aria-label` both carry
real numbers, and why the readout borrows the tile's note line instead of
floating a tooltip: five overlays in a row of 180px tiles is not a hover layer,
it is a pile.
`web/src/index.css` documents the validated values. Re-run the validator after
touching them:
```sh
node <skill>/scripts/validate_palette.js "#bd8829" --mode dark --surface "#14110d"
```
## Deployment gotchas learned the hard way
**A sudoers grant matches the whole argument vector.** The `restorecon` grant
names three paths; running it with one path is a different command and asks for
a password. Run the command exactly as `infra-setup.sh` grants it, which is what
the workflow does.
**The config check runs as `blackbeard`, not root** — runas `(blackbeard)` in
sudoers. A root-run check passes on a config the service account cannot read,
which is the failure it exists to catch.
**`CorsLayer::allow_origin` replaces, it does not append.** Folding over a list
of origins leaves only the last one allowed. It does not break the site — the
frontend is same-origin and never consults CORS — so it is invisible until
something else tries to call the API. Use `AllowOrigin::list`.
**The API binds the host's mesh address, never loopback and never `0.0.0.0`.**
The fleet has one firewalld default zone, so a wildcard bind plus the named
service would publish it on every address the host carries; and the edge proxy
is at another site, so loopback-only would not serve the site at all. Anything
probing `127.0.0.1:25864` gets connection-refused against a perfectly healthy
daemon — which is exactly how the first green build still failed its health
check. Use `$(hostname -f)`, which resolves to the mesh address.
**The edge proxy is at a different site from the API.** oolon (kosherinata)
fronts the name; the API is on bob (hanzalova) because that is where the Planck
node lives. A health probe on the API host cannot see that hop, so there is a
separate check that curls the API *from oolon*. A firewalld service scoped to
bob's own /16 would leave a live site with a dead `/v1` and nothing would fail.
**The edge proxy cannot reach its own public name.** From inside the mesh
`blackbeard.observer` resolves to the site's WAN address and dead-ends on the
OPNsense LAN interface (`reverse-proxies.md` §2). Verify the vhost with
`--resolve blackbeard.observer:443:127.0.0.1`, which still exercises the `:443`
stream router, SNI, the vhost, the cross-site hop and the API.
**A frontend-only push skips the Rust build and the API deploy.** The `what
changed` step in `deploy.yaml` diffs against `github.event.before` and sets one
output; the gate, the musl build, the ts-rs drift check and the whole
`deploy-api` job hang off it. Two entries in that path list are not obvious and
must not be trimmed: `asset/`, because `deploy-api` ships the systemd units, the
firewalld service and the rendered config from it; and
`web/src/api/generated/`, because the drift gate only runs when `cargo test` has
regenerated those files, so a hand-edit of them must not be able to arrive
labelled "frontend only" — which is precisely the change that gate exists to
catch. Anything unrecognised counts as Rust: a false positive costs a slow
deploy, a false negative leaves a binary on bob that does not match the commit
and nothing reports it.
**Runner labels.** `fedora-*` images have **no cargo** — Rust builds go on
`rust`, which is based on `runner-fedora-44` and so carries node + pnpm too.
Never `corepack enable`; pnpm is already on PATH (`gitea-runners.md` §4).
**WebSocket upgrade tests need `--http1.1`.** The vhost serves HTTP/2, and curl
negotiates h2, where the `Connection: Upgrade` handshake is not how WebSockets
work — you get a 400 from axum that looks like a proxy misconfiguration.
## Verifying a change
`systemctl is-active` is not evidence this works. A daemon with a node it cannot
decode, or a telemetry feed it silently ignores, is perfectly "active" while
serving an empty leaderboard. Check the numbers:
```sh
cargo run -p blackbeard-cli -- probe --rpc-url http://bob.hanzalova.internal:9944
cargo run -p blackbeard-cli -- standings --chain quantus
curl -s localhost:25864/v1/chains/quantus/summary | python3 -m json.tool
```
The block route degrades rather than failing, so it is worth checking both ends
of that: a recent height should carry `difficulty`, `extrinsics` and
`seconds_since_parent`, while one a few days old keeps the body and loses
difficulty to state pruning unless this observer recorded it at the time.
```sh
curl -s localhost:25864/v1/chains/quantus/blocks/1000 | python3 -m json.tool
```
A working deployment shows a non-null `telemetry_nodes`, `distinct_miners` above
one, and named rows in the standings.