The API aborts every ~2 minutes on a 76 GiB allocation: scale-value preallocates a sequence from its declared length #10

New Issue

grenade · 2026-09-10T08:56:10Z

grenade commented

2026-09-10 08:56:10 +00:00

The deployed API has been in a crash loop since 2026-09-09T22:30:24Z. As of 2026-09-10T09:00Z, 316 aborts, all identical:

blackbeard-api[902683]: memory allocation of 82014765760 bytes failed
systemd[1]: blackbeard-api.service: Main process exited, code=dumped, status=6/ABRT
systemd[1]: blackbeard-api.service: Scheduled restart job, restart counter is at 10.

Every one is the same size — 82,014,765,760 bytes, 76.4 GiB — and the process survives about 114 seconds from start to abort.

Where the number comes from

std::mem::size_of::<scale_value::Value<u32>>() is 80, and

82_014_765_760 / 80 = 1_025_184_572   (exactly, no remainder)

So this is Vec::<Value<u32>>::with_capacity(1_025_184_572). In scale-value 0.18.2, scale_impls/decode.rs:

macro_rules! to_unnamed_composite {
    ($value:ident, $type_id:ident) => {{
        let mut vals = Vec::with_capacity($value.remaining());
        while let Some(val) = $value.decode_item(DecodeValueVisitor::<R, F>::new()) {

For a sequence, remaining() is the SCALE compact length read straight out of the blob. The allocation happens before a single item is decoded, so the very step that would have caught the mismatch never runs. A blob that disagrees with the registry does not return Err — it takes the process down.

This is the case CLAUDE.md says cannot happen:

A block decoded against the wrong runtime must fail, not succeed.

It does neither. And decode_events(&blob).ok() at ingest.rs:758 cannot help: .ok() catches an Err, not an abort.

What it breaks

Not itself the standings — see the companion issue on fill_gap — but it caps every process lifetime at ~114 s, and any work that needs longer than that never finishes.

Not yet known

Which call site feeds it the mismatched blob. Ruled out by local reproduction against the live chains:

heisenberg 1,021,215–1,021,217 and planck 1,080,907–1,080,909 (the first blocks past each frozen cursor) all decode cleanly against their own metadata
planck's descending backfill region is spec 148 throughout, the same runtime the tip uses, so it is not crossing an upgrade boundary
heisenberg 977,079 — the block the log warns about every cycle — fails, but cleanly, with Decode error: Codec error, and reproduces locally as a plain Err

The abort follows an rpc endpoint failed over line by between 3 and 60 seconds in every cycle, which is suggestive and not yet conclusive.

Release builds are stripped, so the cores are not usable: eu-stack reports No DWARF information found, and systemd's own unwinder produces two junk frames. Getting the call site needs either RUST_BACKTRACE=1 on the unit or a debug build.

Fix, regardless of the call site

Do not let a length read out of a blob size an allocation. The blob is at most a few hundred KiB, and no sequence in it can have more items than it has bytes, so the honest cap is the remaining input length. Either clamp before calling decode_as_type, or — cleaner — check the declared length against the input and return RuntimeError::Decode when it cannot possibly fit. That turns this into exactly the loud, local failure the decoder is supposed to produce.

Upstream is worth a report too, but we should not wait for it.

The deployed API has been in a crash loop since **2026-09-09T22:30:24Z**. As of 2026-09-10T09:00Z, **316 aborts**, all identical: ``` blackbeard-api[902683]: memory allocation of 82014765760 bytes failed systemd[1]: blackbeard-api.service: Main process exited, code=dumped, status=6/ABRT systemd[1]: blackbeard-api.service: Scheduled restart job, restart counter is at 10. ``` Every one is the same size — 82,014,765,760 bytes, 76.4 GiB — and the process survives about 114 seconds from start to abort. ## Where the number comes from `std::mem::size_of::<scale_value::Value<u32>>()` is **80**, and ``` 82_014_765_760 / 80 = 1_025_184_572 (exactly, no remainder) ``` So this is `Vec::<Value<u32>>::with_capacity(1_025_184_572)`. In `scale-value 0.18.2`, `scale_impls/decode.rs`: ```rust macro_rules! to_unnamed_composite { ($value:ident, $type_id:ident) => {{ let mut vals = Vec::with_capacity($value.remaining()); while let Some(val) = $value.decode_item(DecodeValueVisitor::<R, F>::new()) { ``` For a sequence, `remaining()` is the SCALE compact length read straight out of the blob. The allocation happens **before a single item is decoded**, so the very step that would have caught the mismatch never runs. A blob that disagrees with the registry does not return `Err` — it takes the process down. This is the case `CLAUDE.md` says cannot happen: > **A block decoded against the wrong runtime must fail, not succeed.** It does neither. And `decode_events(&blob).ok()` at `ingest.rs:758` cannot help: `.ok()` catches an `Err`, not an abort. ## What it breaks Not itself the standings — see the companion issue on `fill_gap` — but it caps every process lifetime at ~114 s, and any work that needs longer than that never finishes. ## Not yet known **Which call site feeds it the mismatched blob.** Ruled out by local reproduction against the live chains: - heisenberg 1,021,215–1,021,217 and planck 1,080,907–1,080,909 (the first blocks past each frozen cursor) all decode cleanly against their own metadata - planck's descending backfill region is spec 148 throughout, the same runtime the tip uses, so it is not crossing an upgrade boundary - heisenberg 977,079 — the block the log warns about every cycle — fails, but *cleanly*, with `Decode error: Codec error`, and reproduces locally as a plain `Err` The abort follows an `rpc endpoint failed over` line by between 3 and 60 seconds in every cycle, which is suggestive and not yet conclusive. Release builds are stripped, so the cores are not usable: `eu-stack` reports `No DWARF information found`, and systemd's own unwinder produces two junk frames. Getting the call site needs either `RUST_BACKTRACE=1` on the unit or a debug build. ## Fix, regardless of the call site Do not let a length read out of a blob size an allocation. The blob is at most a few hundred KiB, and no sequence in it can have more items than it has bytes, so the honest cap is the remaining input length. Either clamp before calling `decode_as_type`, or — cleaner — check the declared length against the input and return `RuntimeError::Decode` when it cannot possibly fit. That turns this into exactly the loud, local failure the decoder is supposed to produce. Upstream is worth a report too, but we should not wait for it.

grenade referenced this issue

2026-09-10 08:56:29 +00:00

fill_gap is all-or-nothing, so a gap it cannot finish in one process lifetime never shrinks at all #11

grenade referenced this issue from a commit

2026-09-10 09:02:11 +00:00

fix: a bogus length prefix aborted the daemon, and the gap fill never resumed

grenade closed this issue

2026-09-10 09:02:12 +00:00

Sign in to join this conversation.