The account page for baba-gorchitsa (qzp2AxZw8szXc5qDe1aU8P5kt1dkf1JFYktvV7N6cVbFZrs8L, account 0xca0f…60f1) shows mined 14.97, but its balance is 14.68. The account is keyless and has only ever mined. On this chain a mining address's balance can only go up (see CLAUDE.md, distribution table), so its balance can never be less than what it was paid.
The index's total credits come to 14.99: 47 × MinerRewarded plus 2 × MinerVolumeFeePaid of 0.01. That is 0.31 more than the balance.
Where
I read the account's free balance from bob's archive node either side of each of the 47 reward heights. 46 go up by the reward. 61041 does not:
The canonical block 61041 is 0x8eacd1a7…cd57e0. Our block row has the same hash and names QUANPOOL (0x02dacf59…) as the miner. The chain's System::Events at that hash contains the exit to 0x9f88… but nothing mentioning 0xca0f…. also_seen for 61041 is empty.
Why
chain_event is keyed on (chain, height, event_index) and stores no block hash. Nothing deletes a height's events when its block is replaced. Here the live pass indexed baba-gorchitsa's fork block at 61041, the chain reorged to QUANPOOL's block, and the block row followed. The event rows did not, and event_scan already claimed 61041, so the backfill never reads it again.
The result is a mix of the two forks' events at that height. The losing miner is over-credited, the winner's real reward is missing, and nothing looks wrong. This affects every sum over chain_event: account mined, the distribution table, account_flows/flows_for, wormhole_routes, and daily activity.
Fix
Record on each event row which block hash it was decoded from.
When a height is indexed, replace that height's whole event set, not individual indices.
When a reorg replaces a block, index the new block's events again, whichever path recorded it.
For history: rows written before this change carry no hash. Add a migration that collapses the event_scan cursor so the backfill re-reads them, which brings every height back in line with its canonical block (cf. 0008_rescan_events.sql).
## Observed
The account page for `baba-gorchitsa` (`qzp2AxZw8szXc5qDe1aU8P5kt1dkf1JFYktvV7N6cVbFZrs8L`, account `0xca0f…60f1`) shows **mined 14.97**, but its balance is **14.68**. The account is keyless and has only ever mined. On this chain a mining address's balance can only go up (see CLAUDE.md, distribution table), so its balance can never be less than what it was paid.
The index's total credits come to 14.99: 47 × `MinerRewarded` plus 2 × `MinerVolumeFeePaid` of 0.01. That is 0.31 more than the balance.
## Where
I read the account's free balance from bob's archive node either side of each of the 47 reward heights. 46 go up by the reward. **61041** does not:
```
61040 9.65
61041 9.65 <- index: MinerRewarded 0.31, Balances::Minted 0.31, NativeTransferred 0.31 (leaf 122192)
```
The canonical block 61041 is `0x8eacd1a7…cd57e0`. Our `block` row has the same hash and names QUANPOOL (`0x02dacf59…`) as the miner. The chain's `System::Events` at that hash contains the exit to `0x9f88…` but **nothing mentioning `0xca0f…`**. `also_seen` for 61041 is empty.
## Why
`chain_event` is keyed on `(chain, height, event_index)` and stores no block hash. Nothing deletes a height's events when its block is replaced. Here the live pass indexed baba-gorchitsa's fork block at 61041, the chain reorged to QUANPOOL's block, and the `block` row followed. The event rows did not, and `event_scan` already claimed 61041, so the backfill never reads it again.
The result is a mix of the two forks' events at that height. The losing miner is over-credited, the winner's real reward is missing, and nothing looks wrong. This affects every sum over `chain_event`: account `mined`, the distribution table, `account_flows`/`flows_for`, `wormhole_routes`, and daily activity.
## Fix
- Record on each event row which block hash it was decoded from.
- When a height is indexed, replace that height's whole event set, not individual indices.
- When a reorg replaces a block, index the new block's events again, whichever path recorded it.
- For history: rows written before this change carry no hash. Add a migration that collapses the `event_scan` cursor so the backfill re-reads them, which brings every height back in line with its canonical block (cf. `0008_rescan_events.sql`).
Correction, 2026-10-14: the issue says the block row followed the reorg. It did not, and that is worse.
/blocks/61041 shows observed_at: null beside recorded neighbours: 61040 and 61042 were both recorded at the tip, at 11:16:25 and 11:17:09. The route lifts observation times only from a recorded row whose hash matches the canonical block, and also_seen is empty. So the row at 61041 holds a hash that is not the canonical block and was never displaced. It is almost certainly baba-gorchitsa's losing block. (I couldn't query the table directly from here, because bob has no psql.)
Mechanism: chain_subscribeNewHeads reports the best head. When QUANPOOL's fork won, it arrived as its 61042, whose parent was not our 61041. ingest only checks height gaps, never parent continuity, so the replacement at 61041 was never recorded at all. That means the standings and miner series were wrong at reorged heights too, not just the event sums.
The fix covers three paths:
Live:record compares each head's parent with the stored row beneath it and walks the fork back, re-recording the block and re-reading its events.
Storage: event and extrinsic rows carry block_hash. A height is cleared of other hashes before it is written.
Backfill: it stays 32 blocks behind the tip and re-records any block row that is not canonical. Migration 0024 collapses its cursor, so all history gets re-checked.
Separately, RollingWindow::push appended a block at a height it already held, which counted both forks. It now replaces in place.
**Correction, 2026-10-14:** the issue says the `block` row followed the reorg. It did not, and that is worse.
`/blocks/61041` shows `observed_at: null` beside recorded neighbours: 61040 and 61042 were both recorded at the tip, at 11:16:25 and 11:17:09. The route lifts observation times only from a recorded row whose hash matches the canonical block, and `also_seen` is empty. So the row at 61041 holds a hash that is not the canonical block and was never displaced. It is almost certainly baba-gorchitsa's losing block. (I couldn't query the table directly from here, because bob has no `psql`.)
Mechanism: `chain_subscribeNewHeads` reports the best head. When QUANPOOL's fork won, it arrived as its 61042, whose parent was not our 61041. `ingest` only checks height gaps, never parent continuity, so the replacement at 61041 was never recorded at all. That means the standings and miner series were wrong at reorged heights too, not just the event sums.
The fix covers three paths:
- **Live:** `record` compares each head's parent with the stored row beneath it and walks the fork back, re-recording the block and re-reading its events.
- **Storage:** event and extrinsic rows carry `block_hash`. A height is cleared of other hashes before it is written.
- **Backfill:** it stays 32 blocks behind the tip and re-records any block row that is not canonical. Migration 0024 collapses its cursor, so all history gets re-checked.
Separately, `RollingWindow::push` appended a block at a height it already held, which counted both forks. It now replaces in place.
Confirmed from the table, which is reachable from roosta over mTLS: block at quantus 61041 is 0xb00f9771…114d9a, with miner 0x134e73f0…edf59 (baba-gorchitsa's preimage). The canonical block is 0x8eacd1a7…cd57e0, mined by QUANPOOL. So the stale row was the losing fork's, as the previous comment inferred.
Confirmed from the table, which is reachable from roosta over mTLS: `block` at quantus 61041 is `0xb00f9771…114d9a`, with miner `0x134e73f0…edf59` (baba-gorchitsa's preimage). The canonical block is `0x8eacd1a7…cd57e0`, mined by QUANPOOL. So the stale row was the losing fork's, as the previous comment inferred.
There are two gaps left in 739a2f5's backfill, both found by reading it back against the code.
An unreadable height keeps a known-wrong row. When events_at returns nothing, or no runtime decodes the events, index_events returns before forget_other_events. The height is claimed anyway, so rows read from a known different block survive. The same happens to extrinsics whenever the body read fails. Fix: once the canonical hash is known, always drop rows whose block_hash names another block. Rows with a null hash are still dropped only when a replacement is in hand. A legacy row is usually right, so an unreadable height keeps it rather than losing history nobody can re-read.
A permanently unreplaceable block row wedges the backfill. If the stored row is stale and the canonical block can't be described (no header, no author digest), replace_block returns false and the backfill breaks at that height on every pass, forever. Fix: tell transient from permanent. A transient failure retries as now. A permanent one is logged at warn with both hashes and the height is claimed, so the rescan carries on below it.
There are two gaps left in 739a2f5's backfill, both found by reading it back against the code.
1. **An unreadable height keeps a known-wrong row.** When `events_at` returns nothing, or no runtime decodes the events, `index_events` returns before `forget_other_events`. The height is claimed anyway, so rows read from a *known different* block survive. The same happens to extrinsics whenever the body read fails. Fix: once the canonical hash is known, always drop rows whose `block_hash` names another block. Rows with a null hash are still dropped only when a replacement is in hand. A legacy row is usually right, so an unreadable height keeps it rather than losing history nobody can re-read.
2. **A permanently unreplaceable block row wedges the backfill.** If the stored row is stale and the canonical block can't be described (no header, no author digest), `replace_block` returns false and the backfill `break`s at that height on every pass, forever. Fix: tell transient from permanent. A transient failure retries as now. A permanent one is logged at `warn` with both hashes and the height is claimed, so the rescan carries on below it.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Observed
The account page for
baba-gorchitsa(qzp2AxZw8szXc5qDe1aU8P5kt1dkf1JFYktvV7N6cVbFZrs8L, account0xca0f…60f1) shows mined 14.97, but its balance is 14.68. The account is keyless and has only ever mined. On this chain a mining address's balance can only go up (see CLAUDE.md, distribution table), so its balance can never be less than what it was paid.The index's total credits come to 14.99: 47 ×
MinerRewardedplus 2 ×MinerVolumeFeePaidof 0.01. That is 0.31 more than the balance.Where
I read the account's free balance from bob's archive node either side of each of the 47 reward heights. 46 go up by the reward. 61041 does not:
The canonical block 61041 is
0x8eacd1a7…cd57e0. Ourblockrow has the same hash and names QUANPOOL (0x02dacf59…) as the miner. The chain'sSystem::Eventsat that hash contains the exit to0x9f88…but nothing mentioning0xca0f….also_seenfor 61041 is empty.Why
chain_eventis keyed on(chain, height, event_index)and stores no block hash. Nothing deletes a height's events when its block is replaced. Here the live pass indexed baba-gorchitsa's fork block at 61041, the chain reorged to QUANPOOL's block, and theblockrow followed. The event rows did not, andevent_scanalready claimed 61041, so the backfill never reads it again.The result is a mix of the two forks' events at that height. The losing miner is over-credited, the winner's real reward is missing, and nothing looks wrong. This affects every sum over
chain_event: accountmined, the distribution table,account_flows/flows_for,wormhole_routes, and daily activity.Fix
event_scancursor so the backfill re-reads them, which brings every height back in line with its canonical block (cf.0008_rescan_events.sql).Correction, 2026-10-14: the issue says the
blockrow followed the reorg. It did not, and that is worse./blocks/61041showsobserved_at: nullbeside recorded neighbours: 61040 and 61042 were both recorded at the tip, at 11:16:25 and 11:17:09. The route lifts observation times only from a recorded row whose hash matches the canonical block, andalso_seenis empty. So the row at 61041 holds a hash that is not the canonical block and was never displaced. It is almost certainly baba-gorchitsa's losing block. (I couldn't query the table directly from here, because bob has nopsql.)Mechanism:
chain_subscribeNewHeadsreports the best head. When QUANPOOL's fork won, it arrived as its 61042, whose parent was not our 61041.ingestonly checks height gaps, never parent continuity, so the replacement at 61041 was never recorded at all. That means the standings and miner series were wrong at reorged heights too, not just the event sums.The fix covers three paths:
recordcompares each head's parent with the stored row beneath it and walks the fork back, re-recording the block and re-reading its events.block_hash. A height is cleared of other hashes before it is written.Separately,
RollingWindow::pushappended a block at a height it already held, which counted both forks. It now replaces in place.Confirmed from the table, which is reachable from roosta over mTLS:
blockat quantus 61041 is0xb00f9771…114d9a, with miner0x134e73f0…edf59(baba-gorchitsa's preimage). The canonical block is0x8eacd1a7…cd57e0, mined by QUANPOOL. So the stale row was the losing fork's, as the previous comment inferred.There are two gaps left in 739a2f5's backfill, both found by reading it back against the code.
events_atreturns nothing, or no runtime decodes the events,index_eventsreturns beforeforget_other_events. The height is claimed anyway, so rows read from a known different block survive. The same happens to extrinsics whenever the body read fails. Fix: once the canonical hash is known, always drop rows whoseblock_hashnames another block. Rows with a null hash are still dropped only when a replacement is in hand. A legacy row is usually right, so an unreadable height keeps it rather than losing history nobody can re-read.replace_blockreturns false and the backfillbreaks at that height on every pass, forever. Fix: tell transient from permanent. A transient failure retries as now. A permanent one is logged atwarnwith both hashes and the height is claimed, so the rescan carries on below it.