blackbeard/observer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Planck's measured block interval is 170x too fast, so its whole page reads as fiction #14

New Issue
Closed
opened 2026-09-10 11:20:20 +00:00 by grenade · 2 comments
grenade commented 2026-09-10 11:20:20 +00:00
Owner
Copy Link

Reported from the standings: baba-gorchitsa shows 3 blocks in Planck's "six hours" window, having moved to mainnet more than six hours ago. Its miner page is correct. Two separate defects behind that, and the first is much worse than the complaint.

1. The measured interval is the observer's catch-up rate, not the chain's

chain        block_interval_seconds   nominal   network_hashrate
quantus      11.07                    false     20.5 TH/s
planck        0.175                   false      877 GH/s
heisenberg    (none)                  true       14.7 MH/s

Planck's 0.175 s is flagged measured, not nominal. The truth, from the chain's own timestamp inherent across exactly the blocks in that window:

#1,079,242  authored 2026-09-09T06:09:19Z
#1,082,764  authored 2026-09-10T11:17:58Z
            3,522 blocks in 29.1 hours  =  29.8 s/block

So the figure is wrong by a factor of 170, and everything derived from it follows: the headline block time, the network hashrate (877 GH/s for a testnet nobody is mining), and the window label.

measured_interval is (last_at - first_at) / (last_h - first_h) over the tip samples. That is correct only if every height between the two samples was watched arriving. Planck has spent the day catching up, so it isn't. record() marks every head-stream block at_tip: true, including the head that lands immediately after fill_gap closed a 1,251-block gap — and a sample pair straddling that measures how fast this observer caught up, not how fast the chain produced.

CLAUDE.md already documents this failure mode — "A measured block interval is only valid at the tip" — and the at_tip flag exists precisely to prevent it. The flag just doesn't cover this path: a head arriving after a gap fill is a catch-up, not a tip observation, and nothing says so. The same applies to an ordinary burst import, where several blocks land at once and the next head is +5 with no elapsed time.

Fix: the ingest loop already knows whether it called fill_gap for this head. Pass at_tip: false when it did. Worth a second guard in measured_interval for a sample pair whose height delta cannot be explained by the elapsed time, since a wrong interval here is never visibly wrong — it renders as a number.

2. The window label is computed from that interval instead of from the window

windowSpan(blocks, interval) multiplies the block count by the current interval. Even with a correct interval that is the wrong quantity: it describes the rate right now, not the span the window actually covers, and on a chain whose rate moved 9x when its miners left, those are different by more than a factor of two.

The honest number already exists. RollingWindow::span_seconds(n) is the authored-time span of exactly the blocks being tallied, and it is already trusted as the denominator of every per-miner hashrate. Surface it and label with it:

last 3,600 blocks · 29 h        (not "~6 h", and no "~" — it is measured, not estimated)

That makes the label true by construction and immune to any future interval bug.

What this does not change

Windows stay block counts. readme.md §"Windows are block counts, not durations" gives the reasons — a duration is meaningless while a node is catching up, and height-bucketed sparklines give every point equal statistical weight — and both still hold. The complaint is not that the window is 3,600 blocks; it is that the page claims 3,600 Planck blocks is six hours when it is twenty-nine.

With both fixed, baba-gorchitsa's row becomes correct and legible: three blocks inside a window the page openly says reaches back 29 hours.

Optional third

The board does not show last_seen, though every row carries it. A muted "27 h ago" against a row whose last block predates the window's nominal duration would answer "did this miner mine recently?" directly, rather than leaving it to be inferred from the header.

Reported from the standings: `baba-gorchitsa` shows 3 blocks in Planck's "six hours" window, having moved to mainnet more than six hours ago. Its miner page is correct. Two separate defects behind that, and the first is much worse than the complaint. ## 1. The measured interval is the observer's catch-up rate, not the chain's ``` chain block_interval_seconds nominal network_hashrate quantus 11.07 false 20.5 TH/s planck 0.175 false 877 GH/s heisenberg (none) true 14.7 MH/s ``` Planck's `0.175 s` is flagged **measured**, not nominal. The truth, from the chain's own timestamp inherent across exactly the blocks in that window: ``` #1,079,242 authored 2026-09-09T06:09:19Z #1,082,764 authored 2026-09-10T11:17:58Z 3,522 blocks in 29.1 hours = 29.8 s/block ``` So the figure is wrong by a factor of **170**, and everything derived from it follows: the headline block time, the network hashrate (877 GH/s for a testnet nobody is mining), and the window label. `measured_interval` is `(last_at - first_at) / (last_h - first_h)` over the tip samples. That is correct only if every height between the two samples was *watched arriving*. Planck has spent the day catching up, so it isn't. `record()` marks every head-stream block `at_tip: true`, including the head that lands immediately after `fill_gap` closed a 1,251-block gap — and a sample pair straddling that measures how fast **this observer** caught up, not how fast the chain produced. `CLAUDE.md` already documents this failure mode — *"A measured block interval is only valid at the tip"* — and the `at_tip` flag exists precisely to prevent it. The flag just doesn't cover this path: a head arriving after a gap fill is a catch-up, not a tip observation, and nothing says so. The same applies to an ordinary burst import, where several blocks land at once and the next head is `+5` with no elapsed time. **Fix:** the ingest loop already knows whether it called `fill_gap` for this head. Pass `at_tip: false` when it did. Worth a second guard in `measured_interval` for a sample pair whose height delta cannot be explained by the elapsed time, since a wrong interval here is never visibly wrong — it renders as a number. ## 2. The window label is computed from that interval instead of from the window `windowSpan(blocks, interval)` multiplies the block count by the *current* interval. Even with a correct interval that is the wrong quantity: it describes the rate right now, not the span the window actually covers, and on a chain whose rate moved 9x when its miners left, those are different by more than a factor of two. The honest number already exists. `RollingWindow::span_seconds(n)` is the authored-time span of exactly the blocks being tallied, and it is already trusted as the denominator of every per-miner hashrate. Surface it and label with it: ``` last 3,600 blocks · 29 h (not "~6 h", and no "~" — it is measured, not estimated) ``` That makes the label true by construction and immune to any future interval bug. ## What this does *not* change Windows stay block counts. `readme.md` §"Windows are block counts, not durations" gives the reasons — a duration is meaningless while a node is catching up, and height-bucketed sparklines give every point equal statistical weight — and both still hold. The complaint is not that the window is 3,600 blocks; it is that the page claims 3,600 Planck blocks is six hours when it is twenty-nine. With both fixed, `baba-gorchitsa`'s row becomes correct *and* legible: three blocks inside a window the page openly says reaches back 29 hours. ## Optional third The board does not show `last_seen`, though every row carries it. A muted "27 h ago" against a row whose last block predates the window's nominal duration would answer "did this miner mine recently?" directly, rather than leaving it to be inferred from the header.
grenade reopened this issue 2026-09-10 15:15:02 +00:00
grenade reopened this issue 2026-09-10 15:15:06 +00:00
grenade commented 2026-09-10 15:15:22 +00:00
Author
Owner
Copy Link

Reopened. I closed this on an incomplete fix and the evidence says so.

Where it stands

The label half is done and verified in a browser against the live chain:

/quantus/six_hours   -> /quantus/3600-blocks     last 3,600 blocks · 11 h
/planck/3600-blocks  -> unchanged                last 3,600 blocks · 32 h

Plus the rename (600-blocks … 100800-blocks, with the old duration names still resolving), and the miner history chart, which was the last place still assuming a block count implies a duration.

The interval half took three passes, and the first two each looked complete:

planck heisenberg true
before 0.175 s — ~30 s / ~300 s
after the gap-fill guard 4.9 s 0.052 s
after receipt-time stamping 45.8 s ✓ 0.054 s

Three distinct causes, not one.

  1. A head that closes a gap was pushed as a tip observation, so a sample pair straddling a catch-up measured the catch-up.
  2. observed_at was stamped where the head was processed. Each head costs two RPC round trips, so the 64-deep channel backs up and drains in a burst — a batch carrying near-identical times while heights march on. Stamped at receipt now.
  3. Fixing the code did not fix the rows already written. at_tip in the table is whatever the writing process believed, and every process before (2) believed a drained channel was a tip observation. restore_ticker_and_timing replays those rows on every start, so the bug comes back from storage. Heisenberg makes a block every five minutes — far too slow to accumulate twenty fresh samples before the next restart — so it never got the chance to correct itself, and reported 0.054 s/block with the running code already correct.

(3) is the one worth remembering: a fix to a pipeline does not fix the data the broken pipeline wrote, and a warm start is a pipeline that reads it back.

Restored samples are now checked against the one clock the observer cannot fake. At the tip a block is seen roughly when it is authored, so the observer's span should resemble the chain's span for the same blocks. Propagation and skew are seconds; a catch-up is a factor. A batch that fails is left unrestored and timing starts nominal.

Not closing this again until it is measured

All three chains holding a plausible interval across a restart — not just now, and not because the code looks right. Heisenberg is the useful case: it is slow enough that every restart depends on the restore path rather than on fresh samples.

Worth considering separately

Every one of these three was an artefact of observer-side timing. The chain's own authored_at is immune to all of them, and the chain series already uses it for exactly that reason (readme.md: "their block interval comes from the author's own timestamp inherent rather than from when this observer saw the block"). Deriving the headline interval from authored_at over recent blocks would delete this entire class of bug rather than guarding against each instance.

I have not done that, because CLAUDE.md records a deliberate decision that the headline uses the measured observation interval, and changing what the site's most prominent number means is not a thing to slip into a bug fix. Filing it as its own question if it is wanted.

Reopened. I closed this on an incomplete fix and the evidence says so. ## Where it stands The label half is done and verified in a browser against the live chain: ``` /quantus/six_hours -> /quantus/3600-blocks last 3,600 blocks · 11 h /planck/3600-blocks -> unchanged last 3,600 blocks · 32 h ``` Plus the rename (`600-blocks` … `100800-blocks`, with the old duration names still resolving), and the miner history chart, which was the last place still assuming a block count implies a duration. The interval half took three passes, and the first two each looked complete: | | planck | heisenberg | true | |---|---|---|---| | before | 0.175 s | — | ~30 s / ~300 s | | after the gap-fill guard | 4.9 s | 0.052 s | | | after receipt-time stamping | 45.8 s ✓ | 0.054 s | | **Three distinct causes, not one.** 1. A head that closes a gap was pushed as a tip observation, so a sample pair straddling a catch-up measured the catch-up. 2. `observed_at` was stamped where the head was *processed*. Each head costs two RPC round trips, so the 64-deep channel backs up and drains in a burst — a batch carrying near-identical times while heights march on. Stamped at receipt now. 3. **Fixing the code did not fix the rows already written.** `at_tip` in the table is whatever the writing process believed, and every process before (2) believed a drained channel was a tip observation. `restore_ticker_and_timing` replays those rows on every start, so the bug comes back from storage. Heisenberg makes a block every five minutes — far too slow to accumulate twenty fresh samples before the next restart — so it never got the chance to correct itself, and reported 0.054 s/block with the running code already correct. (3) is the one worth remembering: a fix to a pipeline does not fix the data the broken pipeline wrote, and a warm start is a pipeline that reads it back. Restored samples are now checked against the one clock the observer cannot fake. At the tip a block is seen roughly when it is authored, so the observer's span should resemble the chain's span for the same blocks. Propagation and skew are seconds; a catch-up is a factor. A batch that fails is left unrestored and timing starts nominal. ## Not closing this again until it is measured All three chains holding a plausible interval **across a restart** — not just now, and not because the code looks right. Heisenberg is the useful case: it is slow enough that every restart depends on the restore path rather than on fresh samples. ## Worth considering separately Every one of these three was an artefact of *observer-side* timing. The chain's own `authored_at` is immune to all of them, and the chain series already uses it for exactly that reason (`readme.md`: "their block interval comes from the author's own timestamp inherent rather than from when this observer saw the block"). Deriving the headline interval from `authored_at` over recent blocks would delete this entire class of bug rather than guarding against each instance. I have not done that, because `CLAUDE.md` records a deliberate decision that the headline uses the measured observation interval, and changing what the site's most prominent number means is not a thing to slip into a bug fix. Filing it as its own question if it is wanted.
grenade commented 2026-09-10 20:40:39 +00:00
Author
Owner
Copy Link

Closing on measurement, as promised.

A circuit breaker tripped at kosherinata, taking out both the database host and the edge proxy. blackbeard-api on bob failed its startup connection and looped 378 times before the power came back — which handed me exactly the test this issue was held open for: a genuine cold start on the chain most exposed to the restore path.

Heisenberg is that chain. It produces roughly a block every five minutes, far too slowly to accumulate MIN_TIP_SAMPLES fresh observations before the next restart, so its interval comes entirely from whatever restore_ticker_and_timing replays out of the table. That is why it was the last one still wrong.

chain before the guard after a real restart target
quantus 13.3 s 12.85 s 12 s
planck 45.8 s 29.6 s 6 s (running far slower)
heisenberg 0.054 s 41.6 s 6 s (running far slower)

All three measured rather than nominal, all three plausible against a chain that is producing. Heisenberg moved by a factor of 770.

The label half is verified too — Planck's 3,600-block window now reads 37.1 h, up from 32 h earlier today, because the chain keeps slowing and the label keeps tracking it instead of asserting six hours.

What this took, for the record

Four causes behind one symptom, and the first three each looked like the whole thing:

  1. A head closing a gap was pushed as a tip observation, so a sample pair straddling a catch-up measured the catch-up.
  2. observed_at was stamped where the head was processed, behind two RPC round trips and a 64-deep channel, so a drained backlog carried near-identical times while heights marched on.
  3. Fixing the code did not fix the rows already written. at_tip in the table is whatever the writing process believed, and the warm start replayed those beliefs on every boot.
  4. The label was blocks x current interval rather than the window's own authored span — which was wrong even once the interval was right.

Plus the rename that came out of it, and the miner history chart, which was the last place still assuming a block count implies a duration.

The lesson worth keeping is (3): a fix to a pipeline does not fix the data the broken pipeline wrote, and a warm start is a pipeline that reads it back. Two of these fixes were verified as "the number looks right now" and were not — the only check that settled it was a restart on the slowest chain.

Still open as its own question

Every one of these was an artefact of observer-side timing. The chain's own authored_at is immune to all four, and the chain series already uses it for that reason. Deriving the headline interval from it would delete the class rather than guarding instances — but it changes what the site's most prominent number means, and CLAUDE.md records the current choice deliberately, so it deserves its own decision rather than being folded into a bug fix.

Closing on measurement, as promised. A circuit breaker tripped at kosherinata, taking out both the database host and the edge proxy. `blackbeard-api` on bob failed its startup connection and looped 378 times before the power came back — which handed me exactly the test this issue was held open for: **a genuine cold start on the chain most exposed to the restore path.** Heisenberg is that chain. It produces roughly a block every five minutes, far too slowly to accumulate `MIN_TIP_SAMPLES` fresh observations before the next restart, so its interval comes entirely from whatever `restore_ticker_and_timing` replays out of the table. That is why it was the last one still wrong. | chain | before the guard | after a real restart | target | |---|---|---|---| | quantus | 13.3 s | **12.85 s** | 12 s | | planck | 45.8 s | **29.6 s** | 6 s (running far slower) | | **heisenberg** | **0.054 s** | **41.6 s** | 6 s (running far slower) | All three measured rather than nominal, all three plausible against a chain that is producing. Heisenberg moved by a factor of 770. The label half is verified too — Planck's 3,600-block window now reads **37.1 h**, up from 32 h earlier today, because the chain keeps slowing and the label keeps tracking it instead of asserting six hours. ## What this took, for the record Four causes behind one symptom, and the first three each looked like the whole thing: 1. A head closing a gap was pushed as a tip observation, so a sample pair straddling a catch-up measured the catch-up. 2. `observed_at` was stamped where the head was *processed*, behind two RPC round trips and a 64-deep channel, so a drained backlog carried near-identical times while heights marched on. 3. Fixing the code did not fix the rows already written. `at_tip` in the table is whatever the writing process believed, and the warm start replayed those beliefs on every boot. 4. The label was `blocks x current interval` rather than the window's own authored span — which was wrong even once the interval was right. Plus the rename that came out of it, and the miner history chart, which was the last place still assuming a block count implies a duration. The lesson worth keeping is (3): a fix to a pipeline does not fix the data the broken pipeline wrote, and a warm start is a pipeline that reads it back. Two of these fixes were verified as "the number looks right now" and were not — the only check that settled it was a restart on the slowest chain. ## Still open as its own question Every one of these was an artefact of observer-side timing. The chain's own `authored_at` is immune to all four, and the chain series already uses it for that reason. Deriving the headline interval from it would delete the class rather than guarding instances — but it changes what the site's most prominent number means, and `CLAUDE.md` records the current choice deliberately, so it deserves its own decision rather than being folded into a bug fix.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
grenade
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: blackbeard/observer#14
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?