Every share and rank is computed over a fraction of the window the page advertises #13

New Issue

grenade · 2026-09-10T09:30:16Z

grenade commented

2026-09-10 09:30:16 +00:00

Found while confirming #10/#11 had taken effect. Unresolved — filing the evidence rather than a guess.

Symptom

observed_blocks is far below window_blocks on all three chains at once, minutes after a restart:

chain	window_blocks	observed_blocks	rows
quantus	3600	1040	14
heisenberg	3600	602	3
planck	3600	211	16

This is not a display quirk. observed_blocks is the sum of the returned rows' blocks, and it equals the tally's total, which is the denominator of every share:

planck top row: blocks 143, share 0.6875   →  143 / 208 = 0.6875 ✓

So the standings panel is labelled LAST 3,600 BLOCKS · ~11 H while the ranks and shares behind it come from ~208. The per-miner hashrates divide summed work by span_seconds over that same short tail, so they describe a much shorter period than the header claims.

Mainnet makes this concrete: it holds 19,928 blocks and the six-hour window is 3,600, so there is no shortage of history to tally — and it still reports 1,040.

What does not explain it

Not the row set. blackbeard_core::window::leaderboard filters nothing; every miner in the tally becomes a row. leaderboard_max_age is a cache freshness parameter in ChainRuntime::leaderboard, not a row filter.
Not the cache. housekeeping recomputes every leaderboard_refresh_seconds, which is 5.
Not capacity. RollingWindow::new(Window::Week.blocks()) is 100,800, well above both the restore size and 3,600.
Not tail. tail(n) is iter().skip(len - n), so it yields min(n, len). total = 208 therefore means window.len() = 208.

What contradicts it

warm_start logs what it pushed, and it is two orders of magnitude larger:

warm start: window restored  chain=planck  blocks=39881

Those go into inner.window via push(block, false) under a single write lock, on the same Arc<ChainRuntime> that spawn and the registry then share — registry.rs builds one runtime, warm-starts it, spawns against Arc::clone, and inserts that same clone into by_slug. Nothing anywhere assigns to .window, clears it, or rebuilds ChainInner.

So the log says the window holds 39,881 and the arithmetic in the served response says it holds 208. One of those is wrong, and the served one is what readers see.

Where to start

Instrument window.len() at the end of warm_start, immediately before the first recompute_leaderboard, and inside tally. That distinguishes "the restore never landed" from "something drains it between warm start and the first tally" without guessing which.

Worth checking recent_blocks' ordering at the same time: warm start pushes rows in whatever order the query returns them, and tail is insertion-ordered, not height-ordered. That does not explain a short window, but it decides which blocks a partial window keeps, and a window restored newest-first would tally the oldest end of the restore.

Not new

The screenshot from before #10/#11 landed shows the same thing on mainnet — a top row of 828 blocks at 83%, so a denominator of ~998 rather than 3,600. This predates today's fixes; it was simply invisible next to a frozen chain.

Found while confirming #10/#11 had taken effect. Unresolved — filing the evidence rather than a guess. ## Symptom `observed_blocks` is far below `window_blocks` on all three chains at once, minutes after a restart: | chain | window_blocks | observed_blocks | rows | |---|---|---|---| | quantus | 3600 | 1040 | 14 | | heisenberg | 3600 | 602 | 3 | | planck | 3600 | 211 | 16 | This is not a display quirk. `observed_blocks` is the sum of the returned rows' `blocks`, and it equals the tally's `total`, which is the **denominator of every share**: ``` planck top row: blocks 143, share 0.6875 → 143 / 208 = 0.6875 ✓ ``` So the standings panel is labelled `LAST 3,600 BLOCKS · ~11 H` while the ranks and shares behind it come from ~208. The per-miner hashrates divide summed work by `span_seconds` over that same short tail, so they describe a much shorter period than the header claims. Mainnet makes this concrete: it holds 19,928 blocks and the six-hour window is 3,600, so there is no shortage of history to tally — and it still reports 1,040. ## What does not explain it - **Not the row set.** `blackbeard_core::window::leaderboard` filters nothing; every miner in the tally becomes a row. `leaderboard_max_age` is a *cache freshness* parameter in `ChainRuntime::leaderboard`, not a row filter. - **Not the cache.** `housekeeping` recomputes every `leaderboard_refresh_seconds`, which is 5. - **Not capacity.** `RollingWindow::new(Window::Week.blocks())` is 100,800, well above both the restore size and 3,600. - **Not `tail`.** `tail(n)` is `iter().skip(len - n)`, so it yields `min(n, len)`. `total = 208` therefore means `window.len() = 208`. ## What contradicts it `warm_start` logs what it pushed, and it is two orders of magnitude larger: ``` warm start: window restored chain=planck blocks=39881 ``` Those go into `inner.window` via `push(block, false)` under a single write lock, on the same `Arc<ChainRuntime>` that `spawn` and the registry then share — `registry.rs` builds one `runtime`, warm-starts it, spawns against `Arc::clone`, and inserts that same clone into `by_slug`. Nothing anywhere assigns to `.window`, clears it, or rebuilds `ChainInner`. So the log says the window holds 39,881 and the arithmetic in the served response says it holds 208. One of those is wrong, and the served one is what readers see. ## Where to start Instrument `window.len()` at the end of `warm_start`, immediately before the first `recompute_leaderboard`, and inside `tally`. That distinguishes "the restore never landed" from "something drains it between warm start and the first tally" without guessing which. Worth checking `recent_blocks`' ordering at the same time: warm start pushes rows in whatever order the query returns them, and `tail` is insertion-ordered, not height-ordered. That does not explain a short window, but it decides *which* blocks a partial window keeps, and a window restored newest-first would tally the oldest end of the restore. ## Not new The screenshot from before #10/#11 landed shows the same thing on mainnet — a top row of 828 blocks at 83%, so a denominator of ~998 rather than 3,600. This predates today's fixes; it was simply invisible next to a frozen chain.

grenade referenced this issue from a commit

2026-09-10 09:34:48 +00:00

chore: say what the window actually holds, for #13

grenade referenced this issue from a commit

2026-09-10 09:53:30 +00:00

chore: log what the leaderboard cache serves, for #13

grenade commented

2026-09-10 09:55:24 +00:00

Invalid — I measured the wrong process. There is no bug. Closing.

The deployed API, asked on bob itself:

chain	window_blocks	observed_blocks	rows
quantus	3600	3600	30
planck	3600	3600	95
heisenberg	3600	3600	2

A full window on every chain. The shares divide by 3,600 exactly as designed.

What actually happened

Every number in the issue came from curl http://127.0.0.1:25864/... on my workstation, through what I believed was an SSH tunnel to bob. It was not:

LISTEN  127.0.0.1:25864  users:(("blackbeard-api",pid=970552))   <- a local debug build
LISTEN      [::1]:25864  users:(("ssh",pid=1018647))             <- the tunnel

A local ./target/debug/blackbeard-api from earlier in the session already held IPv4 loopback, so ssh -L fell back to binding IPv6 only — with ExitOnForwardFailure=yes satisfied, because one of the two binds succeeded. curl 127.0.0.1 prefers IPv4, so every request went to the local process. It had been up 99 minutes with a window filling from live blocks, which is precisely the "window far short of 3,600" signature I then attributed to production.

The tell was in the issue and I read past it: the log line said tallied: 3600 at the same instant the response said 1,177. Two irreconcilable answers from one process is not a subtle bug, it is a sign you are talking to two processes. I kept looking for a mechanism inside the code instead.

What stands

held beside blocks on warm start: window restored — a real invariant, cheap, and it is what proved the restore lands.
chain on the recompute line. Its absence made three chains' recomputes indistinguishable and cost real time.
Both tests: a_restored_window_tallies_the_whole_requested_span in core, and the_board_divides_by_the_whole_restored_window driving push → recompute_leaderboard. They pin behaviour that was previously only assumed, and they passed throughout — which was itself evidence the fault was not where I was looking.

The temporary debug drop-in on bob has been removed and the stray local process stopped.

The part worth keeping

The claim in "Not new" — that the pre-fix mainnet screenshot showed 828 blocks at 83%, implying a ~998 denominator — was the same measurement error. That screenshot was taken through the same tunnel, against the same local process. It was never evidence of anything about production.

**Invalid — I measured the wrong process.** There is no bug. Closing. The deployed API, asked on bob itself: | chain | window_blocks | observed_blocks | rows | |---|---|---|---| | quantus | 3600 | **3600** | 30 | | planck | 3600 | **3600** | 95 | | heisenberg | 3600 | **3600** | 2 | A full window on every chain. The shares divide by 3,600 exactly as designed. ## What actually happened Every number in the issue came from `curl http://127.0.0.1:25864/...` on my workstation, through what I believed was an SSH tunnel to bob. It was not: ``` LISTEN 127.0.0.1:25864 users:(("blackbeard-api",pid=970552)) <- a local debug build LISTEN [::1]:25864 users:(("ssh",pid=1018647)) <- the tunnel ``` A local `./target/debug/blackbeard-api` from earlier in the session already held **IPv4** loopback, so `ssh -L` fell back to binding **IPv6** only — with `ExitOnForwardFailure=yes` satisfied, because one of the two binds succeeded. `curl 127.0.0.1` prefers IPv4, so every request went to the local process. It had been up 99 minutes with a window filling from live blocks, which is precisely the "window far short of 3,600" signature I then attributed to production. The tell was in the issue and I read past it: the log line said `tallied: 3600` at the same instant the response said 1,177. Two irreconcilable answers from one process is not a subtle bug, it is a sign you are talking to two processes. I kept looking for a mechanism inside the code instead. ## What stands - `held` beside `blocks` on `warm start: window restored` — a real invariant, cheap, and it is what proved the restore lands. - `chain` on the recompute line. Its absence made three chains' recomputes indistinguishable and cost real time. - Both tests: `a_restored_window_tallies_the_whole_requested_span` in core, and `the_board_divides_by_the_whole_restored_window` driving push → `recompute_leaderboard`. They pin behaviour that was previously only assumed, and they passed throughout — which was itself evidence the fault was not where I was looking. The temporary debug drop-in on bob has been removed and the stray local process stopped. ## The part worth keeping The claim in "Not new" — that the pre-fix mainnet screenshot showed 828 blocks at 83%, implying a ~998 denominator — was the same measurement error. That screenshot was taken through the same tunnel, against the same local process. It was never evidence of anything about production.

grenade closed this issue

2026-09-10 09:55:29 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: blackbeard/observer#13