Failover buys availability, not depth: a pruned node answers null, and null is a success #9

New Issue

grenade · 2026-09-10T08:37:51Z

grenade commented

2026-09-10 08:37:51 +00:00

Follow-up to #8, which gave mainnet two public archive nodes behind our own.

The gap

RpcClient::call moves to the next endpoint on a transport failure and never on a JSON-RPC error, which is right — an error is the node answering, and asking a second node the same bad question just wastes a round trip.

But a node that has pruned the state you asked for does not error. It answers:

{"jsonrpc":"2.0","id":1,"result":null}

That is a success. state_getStorage at a pruned height, state_getKeysPaged over a pruned prefix, QPoWApi_get_difficulty against a pruned parent — all of them come back empty and the client is satisfied. So the fallbacks do nothing for depth: whichever node we happen to be pinned to decides how far back the site can see, and if that is ours, the answer is 256 blocks once it starts pruning.

Right now this is invisible because bob runs archive state pruning and is first in the list, so it always has the answer. It becomes real the moment bob's disk gets tight and somebody drops that flag, and it will present as "history quietly went blank" rather than as an error.

Why it is not just "retry on null"

Null is a legitimate answer. Most of the storage keys we read genuinely do not exist — an account with no balance, an item never set, the treasury's System::Account entry, which is absent because the treasury was never funded (that is a real finding the site currently reports correctly). Retrying every one of those against three nodes in turn would triple the read volume on the public nodes to re-derive an answer we already had, and would still end in null.

The distinction is not in the response. It is in whether the node could have known:

a read at the tip, or at a height inside every node's pruning window, has a trustworthy null
a read at a historical height only has a trustworthy null from a node that holds that state

Sketch

Some options, roughly in increasing order of cost:

Ask the node what it keeps. system_nodeRoles does not say, but archive_v1_* in rpc_methods is a strong signal, and a state_getStorage for a key we know exists at genesis is a direct test. Probe each endpoint once at startup, mark it archival or not, and route historical reads only to the archival ones. Cheap — one probe per endpoint — and it makes the pinning explicit rather than incidental.
A depth-aware call, call_deep or similar, used by the reads that are known to be historical (difficulty_for_child_of, the genesis and block-zero state reads, storage_keys_paged at an old hash). A null from a non-archival endpoint means "ask the next one"; a null from an archival one is the answer. Leaves every tip read on the fast path untouched.
Record the pruning boundary we observe and surface it, so the block route can say "state before #N was pruned by every endpoint we have" instead of rendering a block with no difficulty and no explanation. The route already degrades gracefully — it just cannot currently distinguish "pruned" from "we never asked".

(1) and (2) compose: the probe decides which endpoints are eligible, the depth-aware call rotates among them.

Not urgent

Nothing is broken today. Filing it because the failure mode is silent, the cause will be a config change on an unrelated host months from now, and the symptom — old blocks losing their difficulty — is one this codebase has already produced once for a different reason, so it would be easy to misdiagnose. CLAUDE.md already warns that substituting the current difficulty for a pruned one is how a gap fill draws a flat plateau across a stretch of history; this is the same trap approached from the other side.

Follow-up to #8, which gave mainnet two public archive nodes behind our own. ## The gap `RpcClient::call` moves to the next endpoint on a transport failure and never on a JSON-RPC error, which is right — an error is the node answering, and asking a second node the same bad question just wastes a round trip. But a node that has pruned the state you asked for does not error. It answers: ```json {"jsonrpc":"2.0","id":1,"result":null} ``` That is a success. `state_getStorage` at a pruned height, `state_getKeysPaged` over a pruned prefix, `QPoWApi_get_difficulty` against a pruned parent — all of them come back empty and the client is satisfied. So the fallbacks do nothing for depth: whichever node we happen to be pinned to decides how far back the site can see, and if that is ours, the answer is 256 blocks once it starts pruning. Right now this is invisible because bob runs archive state pruning and is first in the list, so it always has the answer. It becomes real the moment bob's disk gets tight and somebody drops that flag, and it will present as "history quietly went blank" rather than as an error. ## Why it is not just "retry on null" Null is a legitimate answer. Most of the storage keys we read genuinely do not exist — an account with no balance, an item never set, the treasury's `System::Account` entry, which is absent because the treasury was never funded (that is a real finding the site currently reports correctly). Retrying every one of those against three nodes in turn would triple the read volume on the public nodes to re-derive an answer we already had, and would still end in null. The distinction is not in the response. It is in whether the node *could* have known: - a read **at the tip**, or at a height inside every node's pruning window, has a trustworthy null - a read at a **historical** height only has a trustworthy null from a node that holds that state ## Sketch Some options, roughly in increasing order of cost: 1. **Ask the node what it keeps.** `system_nodeRoles` does not say, but `archive_v1_*` in `rpc_methods` is a strong signal, and a `state_getStorage` for a key we know exists at genesis is a direct test. Probe each endpoint once at startup, mark it archival or not, and route historical reads only to the archival ones. Cheap — one probe per endpoint — and it makes the pinning explicit rather than incidental. 2. **A depth-aware call**, `call_deep` or similar, used by the reads that are known to be historical (`difficulty_for_child_of`, the genesis and block-zero state reads, `storage_keys_paged` at an old hash). A null from a non-archival endpoint means "ask the next one"; a null from an archival one is the answer. Leaves every tip read on the fast path untouched. 3. **Record the pruning boundary** we observe and surface it, so the block route can say "state before #N was pruned by every endpoint we have" instead of rendering a block with no difficulty and no explanation. The route already degrades gracefully — it just cannot currently distinguish "pruned" from "we never asked". (1) and (2) compose: the probe decides which endpoints are eligible, the depth-aware call rotates among them. ## Not urgent Nothing is broken today. Filing it because the failure mode is silent, the cause will be a config change on an unrelated host months from now, and the symptom — old blocks losing their difficulty — is one this codebase has already produced once for a different reason, so it would be easy to misdiagnose. `CLAUDE.md` already warns that substituting the current difficulty for a pruned one is how a gap fill draws a flat plateau across a stretch of history; this is the same trap approached from the other side.

grenade referenced this issue from a commit

2026-09-10 08:38:16 +00:00

config: give mainnet the two public archive nodes as fallbacks

grenade commented

2026-09-10 08:39:26 +00:00

Worth pinning before anyone picks this up: CLAUDE.md already documents null-on-pruned as deliberate, under "A pruned block is a success, not an unhealthy endpoint" — treating it as a failure would walk the whole endpoint list asking a question none of them can answer, and turn a legitimate absence into an outage. The neighbouring note, "Endpoints are stuck to, never round-robined", is the other half: alternating between nodes that prune on their own schedules returns a mixture of answers and absences that reads as sparse data rather than as a configuration problem.

So this issue is not "make null fail over". Both of those decisions stand. What is missing is the third thing neither covers: knowing, before the read, which endpoints could possibly hold the state — so a historical read can be routed to an archival node rather than retried across all of them. Option (1) in the description is the load-bearing part; (2) is only useful once (1) exists.

If a fix here ever ends up deleting either of those CLAUDE.md paragraphs, it has gone wrong.

Worth pinning before anyone picks this up: `CLAUDE.md` already documents null-on-pruned as **deliberate**, under *"A pruned block is a success, not an unhealthy endpoint"* — treating it as a failure would walk the whole endpoint list asking a question none of them can answer, and turn a legitimate absence into an outage. The neighbouring note, *"Endpoints are stuck to, never round-robined"*, is the other half: alternating between nodes that prune on their own schedules returns a mixture of answers and absences that reads as sparse data rather than as a configuration problem. So this issue is **not** "make null fail over". Both of those decisions stand. What is missing is the third thing neither covers: knowing, before the read, which endpoints could possibly hold the state — so a historical read can be *routed* to an archival node rather than *retried* across all of them. Option (1) in the description is the load-bearing part; (2) is only useful once (1) exists. If a fix here ever ends up deleting either of those `CLAUDE.md` paragraphs, it has gone wrong.

grenade referenced this issue from a commit

2026-09-10 09:12:15 +00:00

feat: route reads that need old state to an endpoint that has it

grenade closed this issue

2026-09-10 09:12:15 +00:00

grenade commented

2026-09-10 09:12:31 +00:00

Implemented in 9def0c8, with one deliberate departure from what this issue proposed.

The description said "route historical reads only to the archival ones". That is worse than what landed. It would have taken every historical read off the sticky endpoint — including mainnet's, which is our own loopback node and pays no network hop — and sent it over the wire to a public node, on every read, to defend against a case that is not currently true of any endpoint.

What landed instead: call_deep tries the sticky endpoint first and only buys a second opinion when a non-archive endpoint returned null. A null from an archive is the answer and stops there. So the common case costs nothing at all, and the extra round trip is paid only in the situation this issue exists for.

One thing the description missed. It framed the whole problem as null-that-looks-like-an-answer. But state_call — which is how difficulty_for_child_of reads difficulty — does not return null when it cannot execute against dropped state. It refuses, with a JSON-RPC error. call is right not to fail over on that (an error is the node answering), so the archive would never have been asked, and the most depth-sensitive read on the site would have been the one still broken. call_deep treats an Rpc error from a non-archive endpoint the same as a null.

Probe. System::Number at block one — always written, four bytes. The key is derived from twox_128 in a test rather than pasted, because a mistyped storage key is absent everywhere and would have classified every endpoint as pruned. :code would also work and is megabytes; state_getRuntimeVersion would work and takes ~4 s per endpoint.

Measured against all seven configured endpoints today, every one returns 0x01000000 — block one's number, so all seven are archives:

endpoint	depth
`http://127.0.0.1:9944` (ours)	archive
`rpc1-mainnet.quantus.com`, `rpc2-`	archive
`a1-heisenberg.quantus.cat`, `a2-`	archive
`a1-planck.quantus.cat`, `a2-`	archive

So this changes no current behaviour and is dormant insurance, which is what it was filed as. An endpoint that cannot be probed stays Unknown and routes as pruned — depth is demonstrated, never assumed, or an unreachable host silently becomes the archive of record.

Option (3) from the description — recording and surfacing the pruning boundary — is not implemented. call_deep logs when an archive answers what the current endpoint could not, which is the operational signal; showing a boundary on the block route is a separate piece of work and nobody needs it while every endpoint is an archive.

Both CLAUDE.md paragraphs the earlier comment asked to protect are still there, with the new behaviour written as a third alongside them.

Implemented in 9def0c8, with one deliberate departure from what this issue proposed. **The description said "route historical reads only to the archival ones".** That is worse than what landed. It would have taken every historical read off the sticky endpoint — including mainnet's, which is our own loopback node and pays no network hop — and sent it over the wire to a public node, on every read, to defend against a case that is not currently true of any endpoint. What landed instead: `call_deep` tries the sticky endpoint first and only buys a second opinion when a **non-archive** endpoint returned null. A null from an archive is the answer and stops there. So the common case costs nothing at all, and the extra round trip is paid only in the situation this issue exists for. **One thing the description missed.** It framed the whole problem as null-that-looks-like-an-answer. But `state_call` — which is how `difficulty_for_child_of` reads difficulty — does not return null when it cannot execute against dropped state. It **refuses, with a JSON-RPC error**. `call` is right not to fail over on that (an error is the node answering), so the archive would never have been asked, and the most depth-sensitive read on the site would have been the one still broken. `call_deep` treats an `Rpc` error from a non-archive endpoint the same as a null. **Probe.** `System::Number` at block one — always written, four bytes. The key is derived from `twox_128` in a test rather than pasted, because a mistyped storage key is absent everywhere and would have classified every endpoint as pruned. `:code` would also work and is megabytes; `state_getRuntimeVersion` would work and takes ~4 s per endpoint. Measured against all seven configured endpoints today, every one returns `0x01000000` — block one's number, so all seven are archives: | endpoint | depth | |---|---| | `http://127.0.0.1:9944` (ours) | archive | | `rpc1-mainnet.quantus.com`, `rpc2-` | archive | | `a1-heisenberg.quantus.cat`, `a2-` | archive | | `a1-planck.quantus.cat`, `a2-` | archive | So this changes no current behaviour and is dormant insurance, which is what it was filed as. An endpoint that cannot be probed stays `Unknown` and routes as pruned — depth is demonstrated, never assumed, or an unreachable host silently becomes the archive of record. Option (3) from the description — recording and surfacing the pruning boundary — is not implemented. `call_deep` logs when an archive answers what the current endpoint could not, which is the operational signal; showing a boundary on the block route is a separate piece of work and nobody needs it while every endpoint is an archive. Both `CLAUDE.md` paragraphs the earlier comment asked to protect are still there, with the new behaviour written as a third alongside them.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: blackbeard/observer#9