fill_gap is all-or-nothing, so a gap it cannot finish in one process lifetime never shrinks at all #11

New Issue

grenade · 2026-09-10T08:56:29Z

grenade commented

2026-09-10 08:56:29 +00:00

fill_gap accumulates the whole gap in memory and writes once, after the loop:

async fn fill_gap(chain: &Arc<ChainRuntime>, store: &Store, from: u64, to: u64) {
    let mut batch = Vec::new();
    let mut observed = Vec::new();
    for height in from..to {
        ...four RPC round trips per block...
        batch.push(BlockRecord { .. });
    }
    if let Err(e) = store.record_blocks(&batch).await { .. }

Its doc comment says it is best effort — "a node that has pruned, or that fails midway, costs the window those blocks and nothing more" — and that is true of a node that answers badly. It is not true of the process going away: partial progress is discarded entirely, so a gap that cannot be walked end to end inside one process lifetime yields zero recorded blocks, forever, no matter how many times it is retried.

Observed

Both testnets are frozen at the height the gap opened at, and have been for about 24 hours:

chain	our newest block	chain tip	behind
planck	1,080,906 (authored 2026-09-09T09:01:17Z)	1,082,602	1,696
heisenberg	1,021,214	1,024,006	2,792
quantus	tip	tip	0

The log shows the attempt on every single start, always with the same from:

filling a gap in the head stream  chain=planck      from=1080907 to=1082602
filling a gap in the head stream  chain=heisenberg  from=1021215 to=1024006

Each pass is ~4 RPC calls per block against a remote endpoint, one of which is difficulty_for_child_of — a historical state read. 1,696 blocks is several thousand round trips, and #10 caps the process at ~114 seconds, so it never reaches record_blocks.

Mainnet is unaffected for two compounding reasons: its endpoint is loopback, and it has no gap to fill.

Why this is worth fixing separately from #10

#10 is what makes the lifetime short today, and fixing it will let these two gaps close. But the interaction is the real defect: an all-or-nothing batch turns any interruption into no progress, and interruptions are routine here — blackbeard-api-cert.path restarts the service on certificate rotation several times a day, and every deploy restarts it too. A gap large enough to outlast the interval between routine restarts is permanently unfillable, with nothing in the logs saying so. The line it does print, filling a gap in the head stream, reads as progress.

Shape of a fix

Flush in batches — a few hundred blocks — so each completed chunk is durable and the next pass starts from a higher from. The window push can move with them. That makes the work resumable and monotonic, and it removes the unbounded Vec at the same time: max_gap_fill_blocks currently bounds it, but the bound exists to limit how far back to reach, not to limit memory.

Worth logging the outcome too, with counts. Right now a pass that records nothing and a pass that records everything produce the same single line.

`fill_gap` accumulates the whole gap in memory and writes once, after the loop: ```rust async fn fill_gap(chain: &Arc<ChainRuntime>, store: &Store, from: u64, to: u64) { let mut batch = Vec::new(); let mut observed = Vec::new(); for height in from..to { ...four RPC round trips per block... batch.push(BlockRecord { .. }); } if let Err(e) = store.record_blocks(&batch).await { .. } ``` Its doc comment says it is best effort — *"a node that has pruned, or that fails midway, costs the window those blocks and nothing more"* — and that is true of a node that answers badly. It is **not** true of the process going away: partial progress is discarded entirely, so a gap that cannot be walked end to end inside one process lifetime yields zero recorded blocks, forever, no matter how many times it is retried. ## Observed Both testnets are frozen at the height the gap opened at, and have been for about 24 hours: | chain | our newest block | chain tip | behind | |---|---|---|---| | planck | 1,080,906 (authored 2026-09-09T09:01:17Z) | 1,082,602 | 1,696 | | heisenberg | 1,021,214 | 1,024,006 | 2,792 | | quantus | tip | tip | 0 | The log shows the attempt on every single start, always with the same `from`: ``` filling a gap in the head stream chain=planck from=1080907 to=1082602 filling a gap in the head stream chain=heisenberg from=1021215 to=1024006 ``` Each pass is ~4 RPC calls per block against a remote endpoint, one of which is `difficulty_for_child_of` — a historical state read. 1,696 blocks is several thousand round trips, and #10 caps the process at ~114 seconds, so it never reaches `record_blocks`. Mainnet is unaffected for two compounding reasons: its endpoint is loopback, and it has no gap to fill. ## Why this is worth fixing separately from #10 #10 is what makes the lifetime short today, and fixing it will let these two gaps close. But the interaction is the real defect: **an all-or-nothing batch turns any interruption into no progress**, and interruptions are routine here — `blackbeard-api-cert.path` restarts the service on certificate rotation several times a day, and every deploy restarts it too. A gap large enough to outlast the interval between routine restarts is permanently unfillable, with nothing in the logs saying so. The line it does print, `filling a gap in the head stream`, reads as progress. ## Shape of a fix Flush in batches — a few hundred blocks — so each completed chunk is durable and the next pass starts from a higher `from`. The window push can move with them. That makes the work resumable and monotonic, and it removes the unbounded `Vec` at the same time: `max_gap_fill_blocks` currently bounds it, but the bound exists to limit how far back to reach, not to limit memory. Worth logging the outcome too, with counts. Right now a pass that records nothing and a pass that records everything produce the same single line.

grenade referenced this issue from a commit

2026-09-10 09:02:12 +00:00

fix: a bogus length prefix aborted the daemon, and the gap fill never resumed

grenade closed this issue

2026-09-10 09:02:12 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: blackbeard/observer#11