Files
observer/crates/blackbeard-api
grenadeandClaude Opus 5 4fb856827c
deploy / build (push) Successful in 8m3s
deploy / deploy-web (push) Successful in 5s
deploy / deploy-api (push) Successful in 17s
fix: a bogus length prefix aborted the daemon, and the gap fill never resumed
Two faults, one visible symptom: both testnet leaderboards frozen for a day at
the height their gap opened at, while mainnet was fine.

The daemon had aborted 316 times in ten hours, always identically —
`memory allocation of 82014765760 bytes failed`, 76.4 GiB. That is
1,025,184,572 x 80, and 80 is `size_of::<scale_value::Value<u32>>()`:
scale-value sizes a sequence's `Vec` from the compact length in the blob before
decoding a single item, so a blob that disagrees with the registry asks for an
allocation of any size at all and Rust aborts on the failed one. The step that
would have caught the mismatch is the one that never runs, which is why
`decode_events(&blob).ok()` could not help — there was no `Err`, only SIGABRT.

Every decode now goes through `Runtime::decode_checked`, which walks the bytes
first with scale-decode's `IgnoreVisitor`. That crate has no `with_capacity`
anywhere, so an impossible length runs out of input on the first item and comes
back as an error. The regression test is the exact four-byte blob, reproduced
against the mainnet fixture before the guard went in — the abort message matched
production byte for byte.

`fill_gap` then turned a short process lifetime into no progress whatsoever: it
accumulated the whole gap and wrote once after the loop, so every interruption
discarded everything read. Each block is four RPC round trips, one a historical
state read; 1,696 blocks against a remote endpoint does not fit in the two
minutes the aborts allowed. And this is not only about the aborts —
`blackbeard-api-cert.path` restarts the service several times a day, so any gap
wider than that interval was already unfillable, logging
`filling a gap in the head stream` on every start with the same `from` and
looking like progress. It flushes every 256 blocks now, and says how many it
recorded.

Both traps are in CLAUDE.md, because neither looks broken from the outside: one
presents as a service that is merely restarting, the other as a catch-up that is
merely slow.

Closes #10
Closes #11

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jp6a8EDar9ueEhAxzep4V5
2026-09-10 12:02:07 +03:00
..