Found while confirming #10/#11 had taken effect. Unresolved — filing the evidence rather than a guess.
Symptom
observed_blocks is far below window_blocks on all three chains at once, minutes after a restart:
chain
window_blocks
observed_blocks
rows
quantus
3600
1040
14
heisenberg
3600
602
3
planck
3600
211
16
This is not a display quirk. observed_blocks is the sum of the returned rows' blocks, and it equals the tally's total, which is the denominator of every share:
So the standings panel is labelled LAST 3,600 BLOCKS · ~11 H while the ranks and shares behind it come from ~208. The per-miner hashrates divide summed work by span_seconds over that same short tail, so they describe a much shorter period than the header claims.
Mainnet makes this concrete: it holds 19,928 blocks and the six-hour window is 3,600, so there is no shortage of history to tally — and it still reports 1,040.
What does not explain it
Not the row set.blackbeard_core::window::leaderboard filters nothing; every miner in the tally becomes a row. leaderboard_max_age is a cache freshness parameter in ChainRuntime::leaderboard, not a row filter.
Not the cache.housekeeping recomputes every leaderboard_refresh_seconds, which is 5.
Not capacity.RollingWindow::new(Window::Week.blocks()) is 100,800, well above both the restore size and 3,600.
Not tail.tail(n) is iter().skip(len - n), so it yields min(n, len). total = 208 therefore means window.len() = 208.
What contradicts it
warm_start logs what it pushed, and it is two orders of magnitude larger:
Those go into inner.window via push(block, false) under a single write lock, on the same Arc<ChainRuntime> that spawn and the registry then share — registry.rs builds one runtime, warm-starts it, spawns against Arc::clone, and inserts that same clone into by_slug. Nothing anywhere assigns to .window, clears it, or rebuilds ChainInner.
So the log says the window holds 39,881 and the arithmetic in the served response says it holds 208. One of those is wrong, and the served one is what readers see.
Where to start
Instrument window.len() at the end of warm_start, immediately before the first recompute_leaderboard, and inside tally. That distinguishes "the restore never landed" from "something drains it between warm start and the first tally" without guessing which.
Worth checking recent_blocks' ordering at the same time: warm start pushes rows in whatever order the query returns them, and tail is insertion-ordered, not height-ordered. That does not explain a short window, but it decides which blocks a partial window keeps, and a window restored newest-first would tally the oldest end of the restore.
Not new
The screenshot from before #10/#11 landed shows the same thing on mainnet — a top row of 828 blocks at 83%, so a denominator of ~998 rather than 3,600. This predates today's fixes; it was simply invisible next to a frozen chain.
Found while confirming #10/#11 had taken effect. Unresolved — filing the evidence rather than a guess.
## Symptom
`observed_blocks` is far below `window_blocks` on all three chains at once, minutes after a restart:
| chain | window_blocks | observed_blocks | rows |
|---|---|---|---|
| quantus | 3600 | 1040 | 14 |
| heisenberg | 3600 | 602 | 3 |
| planck | 3600 | 211 | 16 |
This is not a display quirk. `observed_blocks` is the sum of the returned rows' `blocks`, and it equals the tally's `total`, which is the **denominator of every share**:
```
planck top row: blocks 143, share 0.6875 → 143 / 208 = 0.6875 ✓
```
So the standings panel is labelled `LAST 3,600 BLOCKS · ~11 H` while the ranks and shares behind it come from ~208. The per-miner hashrates divide summed work by `span_seconds` over that same short tail, so they describe a much shorter period than the header claims.
Mainnet makes this concrete: it holds 19,928 blocks and the six-hour window is 3,600, so there is no shortage of history to tally — and it still reports 1,040.
## What does not explain it
- **Not the row set.** `blackbeard_core::window::leaderboard` filters nothing; every miner in the tally becomes a row. `leaderboard_max_age` is a *cache freshness* parameter in `ChainRuntime::leaderboard`, not a row filter.
- **Not the cache.** `housekeeping` recomputes every `leaderboard_refresh_seconds`, which is 5.
- **Not capacity.** `RollingWindow::new(Window::Week.blocks())` is 100,800, well above both the restore size and 3,600.
- **Not `tail`.** `tail(n)` is `iter().skip(len - n)`, so it yields `min(n, len)`. `total = 208` therefore means `window.len() = 208`.
## What contradicts it
`warm_start` logs what it pushed, and it is two orders of magnitude larger:
```
warm start: window restored chain=planck blocks=39881
```
Those go into `inner.window` via `push(block, false)` under a single write lock, on the same `Arc<ChainRuntime>` that `spawn` and the registry then share — `registry.rs` builds one `runtime`, warm-starts it, spawns against `Arc::clone`, and inserts that same clone into `by_slug`. Nothing anywhere assigns to `.window`, clears it, or rebuilds `ChainInner`.
So the log says the window holds 39,881 and the arithmetic in the served response says it holds 208. One of those is wrong, and the served one is what readers see.
## Where to start
Instrument `window.len()` at the end of `warm_start`, immediately before the first `recompute_leaderboard`, and inside `tally`. That distinguishes "the restore never landed" from "something drains it between warm start and the first tally" without guessing which.
Worth checking `recent_blocks`' ordering at the same time: warm start pushes rows in whatever order the query returns them, and `tail` is insertion-ordered, not height-ordered. That does not explain a short window, but it decides *which* blocks a partial window keeps, and a window restored newest-first would tally the oldest end of the restore.
## Not new
The screenshot from before #10/#11 landed shows the same thing on mainnet — a top row of 828 blocks at 83%, so a denominator of ~998 rather than 3,600. This predates today's fixes; it was simply invisible next to a frozen chain.
Invalid — I measured the wrong process. There is no bug. Closing.
The deployed API, asked on bob itself:
chain
window_blocks
observed_blocks
rows
quantus
3600
3600
30
planck
3600
3600
95
heisenberg
3600
3600
2
A full window on every chain. The shares divide by 3,600 exactly as designed.
What actually happened
Every number in the issue came from curl http://127.0.0.1:25864/... on my workstation, through what I believed was an SSH tunnel to bob. It was not:
LISTEN 127.0.0.1:25864 users:(("blackbeard-api",pid=970552)) <- a local debug build
LISTEN [::1]:25864 users:(("ssh",pid=1018647)) <- the tunnel
A local ./target/debug/blackbeard-api from earlier in the session already held IPv4 loopback, so ssh -L fell back to binding IPv6 only — with ExitOnForwardFailure=yes satisfied, because one of the two binds succeeded. curl 127.0.0.1 prefers IPv4, so every request went to the local process. It had been up 99 minutes with a window filling from live blocks, which is precisely the "window far short of 3,600" signature I then attributed to production.
The tell was in the issue and I read past it: the log line said tallied: 3600 at the same instant the response said 1,177. Two irreconcilable answers from one process is not a subtle bug, it is a sign you are talking to two processes. I kept looking for a mechanism inside the code instead.
What stands
held beside blocks on warm start: window restored — a real invariant, cheap, and it is what proved the restore lands.
chain on the recompute line. Its absence made three chains' recomputes indistinguishable and cost real time.
Both tests: a_restored_window_tallies_the_whole_requested_span in core, and the_board_divides_by_the_whole_restored_window driving push → recompute_leaderboard. They pin behaviour that was previously only assumed, and they passed throughout — which was itself evidence the fault was not where I was looking.
The temporary debug drop-in on bob has been removed and the stray local process stopped.
The part worth keeping
The claim in "Not new" — that the pre-fix mainnet screenshot showed 828 blocks at 83%, implying a ~998 denominator — was the same measurement error. That screenshot was taken through the same tunnel, against the same local process. It was never evidence of anything about production.
**Invalid — I measured the wrong process.** There is no bug. Closing.
The deployed API, asked on bob itself:
| chain | window_blocks | observed_blocks | rows |
|---|---|---|---|
| quantus | 3600 | **3600** | 30 |
| planck | 3600 | **3600** | 95 |
| heisenberg | 3600 | **3600** | 2 |
A full window on every chain. The shares divide by 3,600 exactly as designed.
## What actually happened
Every number in the issue came from `curl http://127.0.0.1:25864/...` on my workstation, through what I believed was an SSH tunnel to bob. It was not:
```
LISTEN 127.0.0.1:25864 users:(("blackbeard-api",pid=970552)) <- a local debug build
LISTEN [::1]:25864 users:(("ssh",pid=1018647)) <- the tunnel
```
A local `./target/debug/blackbeard-api` from earlier in the session already held **IPv4** loopback, so `ssh -L` fell back to binding **IPv6** only — with `ExitOnForwardFailure=yes` satisfied, because one of the two binds succeeded. `curl 127.0.0.1` prefers IPv4, so every request went to the local process. It had been up 99 minutes with a window filling from live blocks, which is precisely the "window far short of 3,600" signature I then attributed to production.
The tell was in the issue and I read past it: the log line said `tallied: 3600` at the same instant the response said 1,177. Two irreconcilable answers from one process is not a subtle bug, it is a sign you are talking to two processes. I kept looking for a mechanism inside the code instead.
## What stands
- `held` beside `blocks` on `warm start: window restored` — a real invariant, cheap, and it is what proved the restore lands.
- `chain` on the recompute line. Its absence made three chains' recomputes indistinguishable and cost real time.
- Both tests: `a_restored_window_tallies_the_whole_requested_span` in core, and `the_board_divides_by_the_whole_restored_window` driving push → `recompute_leaderboard`. They pin behaviour that was previously only assumed, and they passed throughout — which was itself evidence the fault was not where I was looking.
The temporary debug drop-in on bob has been removed and the stray local process stopped.
## The part worth keeping
The claim in "Not new" — that the pre-fix mainnet screenshot showed 828 blocks at 83%, implying a ~998 denominator — was the same measurement error. That screenshot was taken through the same tunnel, against the same local process. It was never evidence of anything about production.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Found while confirming #10/#11 had taken effect. Unresolved — filing the evidence rather than a guess.
Symptom
observed_blocksis far belowwindow_blockson all three chains at once, minutes after a restart:This is not a display quirk.
observed_blocksis the sum of the returned rows'blocks, and it equals the tally'stotal, which is the denominator of every share:So the standings panel is labelled
LAST 3,600 BLOCKS · ~11 Hwhile the ranks and shares behind it come from ~208. The per-miner hashrates divide summed work byspan_secondsover that same short tail, so they describe a much shorter period than the header claims.Mainnet makes this concrete: it holds 19,928 blocks and the six-hour window is 3,600, so there is no shortage of history to tally — and it still reports 1,040.
What does not explain it
blackbeard_core::window::leaderboardfilters nothing; every miner in the tally becomes a row.leaderboard_max_ageis a cache freshness parameter inChainRuntime::leaderboard, not a row filter.housekeepingrecomputes everyleaderboard_refresh_seconds, which is 5.RollingWindow::new(Window::Week.blocks())is 100,800, well above both the restore size and 3,600.tail.tail(n)isiter().skip(len - n), so it yieldsmin(n, len).total = 208therefore meanswindow.len() = 208.What contradicts it
warm_startlogs what it pushed, and it is two orders of magnitude larger:Those go into
inner.windowviapush(block, false)under a single write lock, on the sameArc<ChainRuntime>thatspawnand the registry then share —registry.rsbuilds oneruntime, warm-starts it, spawns againstArc::clone, and inserts that same clone intoby_slug. Nothing anywhere assigns to.window, clears it, or rebuildsChainInner.So the log says the window holds 39,881 and the arithmetic in the served response says it holds 208. One of those is wrong, and the served one is what readers see.
Where to start
Instrument
window.len()at the end ofwarm_start, immediately before the firstrecompute_leaderboard, and insidetally. That distinguishes "the restore never landed" from "something drains it between warm start and the first tally" without guessing which.Worth checking
recent_blocks' ordering at the same time: warm start pushes rows in whatever order the query returns them, andtailis insertion-ordered, not height-ordered. That does not explain a short window, but it decides which blocks a partial window keeps, and a window restored newest-first would tally the oldest end of the restore.Not new
The screenshot from before #10/#11 landed shows the same thing on mainnet — a top row of 828 blocks at 83%, so a denominator of ~998 rather than 3,600. This predates today's fixes; it was simply invisible next to a frozen chain.
Invalid — I measured the wrong process. There is no bug. Closing.
The deployed API, asked on bob itself:
A full window on every chain. The shares divide by 3,600 exactly as designed.
What actually happened
Every number in the issue came from
curl http://127.0.0.1:25864/...on my workstation, through what I believed was an SSH tunnel to bob. It was not:A local
./target/debug/blackbeard-apifrom earlier in the session already held IPv4 loopback, sossh -Lfell back to binding IPv6 only — withExitOnForwardFailure=yessatisfied, because one of the two binds succeeded.curl 127.0.0.1prefers IPv4, so every request went to the local process. It had been up 99 minutes with a window filling from live blocks, which is precisely the "window far short of 3,600" signature I then attributed to production.The tell was in the issue and I read past it: the log line said
tallied: 3600at the same instant the response said 1,177. Two irreconcilable answers from one process is not a subtle bug, it is a sign you are talking to two processes. I kept looking for a mechanism inside the code instead.What stands
heldbesideblocksonwarm start: window restored— a real invariant, cheap, and it is what proved the restore lands.chainon the recompute line. Its absence made three chains' recomputes indistinguishable and cost real time.a_restored_window_tallies_the_whole_requested_spanin core, andthe_board_divides_by_the_whole_restored_windowdriving push →recompute_leaderboard. They pin behaviour that was previously only assumed, and they passed throughout — which was itself evidence the fault was not where I was looking.The temporary debug drop-in on bob has been removed and the stray local process stopped.
The part worth keeping
The claim in "Not new" — that the pre-fix mainnet screenshot showed 828 blocks at 83%, implying a ~998 denominator — was the same measurement error. That screenshot was taken through the same tunnel, against the same local process. It was never evidence of anything about production.