Its doc comment says it is best effort — "a node that has pruned, or that fails midway, costs the window those blocks and nothing more" — and that is true of a node that answers badly. It is not true of the process going away: partial progress is discarded entirely, so a gap that cannot be walked end to end inside one process lifetime yields zero recorded blocks, forever, no matter how many times it is retried.
Observed
Both testnets are frozen at the height the gap opened at, and have been for about 24 hours:
chain
our newest block
chain tip
behind
planck
1,080,906 (authored 2026-09-09T09:01:17Z)
1,082,602
1,696
heisenberg
1,021,214
1,024,006
2,792
quantus
tip
tip
0
The log shows the attempt on every single start, always with the same from:
filling a gap in the head stream chain=planck from=1080907 to=1082602
filling a gap in the head stream chain=heisenberg from=1021215 to=1024006
Each pass is ~4 RPC calls per block against a remote endpoint, one of which is difficulty_for_child_of — a historical state read. 1,696 blocks is several thousand round trips, and #10 caps the process at ~114 seconds, so it never reaches record_blocks.
Mainnet is unaffected for two compounding reasons: its endpoint is loopback, and it has no gap to fill.
#10 is what makes the lifetime short today, and fixing it will let these two gaps close. But the interaction is the real defect: an all-or-nothing batch turns any interruption into no progress, and interruptions are routine here — blackbeard-api-cert.path restarts the service on certificate rotation several times a day, and every deploy restarts it too. A gap large enough to outlast the interval between routine restarts is permanently unfillable, with nothing in the logs saying so. The line it does print, filling a gap in the head stream, reads as progress.
Shape of a fix
Flush in batches — a few hundred blocks — so each completed chunk is durable and the next pass starts from a higher from. The window push can move with them. That makes the work resumable and monotonic, and it removes the unbounded Vec at the same time: max_gap_fill_blocks currently bounds it, but the bound exists to limit how far back to reach, not to limit memory.
Worth logging the outcome too, with counts. Right now a pass that records nothing and a pass that records everything produce the same single line.
`fill_gap` accumulates the whole gap in memory and writes once, after the loop:
```rust
async fn fill_gap(chain: &Arc<ChainRuntime>, store: &Store, from: u64, to: u64) {
let mut batch = Vec::new();
let mut observed = Vec::new();
for height in from..to {
...four RPC round trips per block...
batch.push(BlockRecord { .. });
}
if let Err(e) = store.record_blocks(&batch).await { .. }
```
Its doc comment says it is best effort — *"a node that has pruned, or that fails midway, costs the window those blocks and nothing more"* — and that is true of a node that answers badly. It is **not** true of the process going away: partial progress is discarded entirely, so a gap that cannot be walked end to end inside one process lifetime yields zero recorded blocks, forever, no matter how many times it is retried.
## Observed
Both testnets are frozen at the height the gap opened at, and have been for about 24 hours:
| chain | our newest block | chain tip | behind |
|---|---|---|---|
| planck | 1,080,906 (authored 2026-09-09T09:01:17Z) | 1,082,602 | 1,696 |
| heisenberg | 1,021,214 | 1,024,006 | 2,792 |
| quantus | tip | tip | 0 |
The log shows the attempt on every single start, always with the same `from`:
```
filling a gap in the head stream chain=planck from=1080907 to=1082602
filling a gap in the head stream chain=heisenberg from=1021215 to=1024006
```
Each pass is ~4 RPC calls per block against a remote endpoint, one of which is `difficulty_for_child_of` — a historical state read. 1,696 blocks is several thousand round trips, and #10 caps the process at ~114 seconds, so it never reaches `record_blocks`.
Mainnet is unaffected for two compounding reasons: its endpoint is loopback, and it has no gap to fill.
## Why this is worth fixing separately from #10
#10 is what makes the lifetime short today, and fixing it will let these two gaps close. But the interaction is the real defect: **an all-or-nothing batch turns any interruption into no progress**, and interruptions are routine here — `blackbeard-api-cert.path` restarts the service on certificate rotation several times a day, and every deploy restarts it too. A gap large enough to outlast the interval between routine restarts is permanently unfillable, with nothing in the logs saying so. The line it does print, `filling a gap in the head stream`, reads as progress.
## Shape of a fix
Flush in batches — a few hundred blocks — so each completed chunk is durable and the next pass starts from a higher `from`. The window push can move with them. That makes the work resumable and monotonic, and it removes the unbounded `Vec` at the same time: `max_gap_fill_blocks` currently bounds it, but the bound exists to limit how far back to reach, not to limit memory.
Worth logging the outcome too, with counts. Right now a pass that records nothing and a pass that records everything produce the same single line.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
fill_gapaccumulates the whole gap in memory and writes once, after the loop:Its doc comment says it is best effort — "a node that has pruned, or that fails midway, costs the window those blocks and nothing more" — and that is true of a node that answers badly. It is not true of the process going away: partial progress is discarded entirely, so a gap that cannot be walked end to end inside one process lifetime yields zero recorded blocks, forever, no matter how many times it is retried.
Observed
Both testnets are frozen at the height the gap opened at, and have been for about 24 hours:
The log shows the attempt on every single start, always with the same
from:Each pass is ~4 RPC calls per block against a remote endpoint, one of which is
difficulty_for_child_of— a historical state read. 1,696 blocks is several thousand round trips, and #10 caps the process at ~114 seconds, so it never reachesrecord_blocks.Mainnet is unaffected for two compounding reasons: its endpoint is loopback, and it has no gap to fill.
Why this is worth fixing separately from #10
#10 is what makes the lifetime short today, and fixing it will let these two gaps close. But the interaction is the real defect: an all-or-nothing batch turns any interruption into no progress, and interruptions are routine here —
blackbeard-api-cert.pathrestarts the service on certificate rotation several times a day, and every deploy restarts it too. A gap large enough to outlast the interval between routine restarts is permanently unfillable, with nothing in the logs saying so. The line it does print,filling a gap in the head stream, reads as progress.Shape of a fix
Flush in batches — a few hundred blocks — so each completed chunk is durable and the next pass starts from a higher
from. The window push can move with them. That makes the work resumable and monotonic, and it removes the unboundedVecat the same time:max_gap_fill_blockscurrently bounds it, but the bound exists to limit how far back to reach, not to limit memory.Worth logging the outcome too, with counts. Right now a pass that records nothing and a pass that records everything produce the same single line.