A chain should hold several endpoints, so one going down cannot hold the observer hostage #7

New Issue

grenade · 2026-09-10T08:10:24Z

grenade commented

2026-09-10 08:10:24 +00:00

Queued deliberately. Not to be picked up while anything else is in flight.

What

A [[chains]] entry holds exactly one rpc_url and one ws_url. When that
endpoint goes down, everything the observer does for that chain stops: no
headers, no difficulty, no backfill, no state reads, no runtime discovery. The
chain reports unreachable and keeps serving its last standings, which is
graceful — but it is one host's outage deciding what a whole chain looks like.

The endpoints already exist in pairs. a1-planck.quantus.cat and
a2-planck.quantus.cat both answer today, and only the first is configured. The
same a1/a2 pattern holds for Heisenberg. We are choosing single points of
failure that the network went to some trouble to avoid.

Shape

[[chains]]
id = "planck"
rpc_urls = ["https://a1-planck.quantus.cat", "https://a2-planck.quantus.cat"]
ws_urls  = ["wss://a1-planck.quantus.cat",  "wss://a2-planck.quantus.cat"]

Keep rpc_url / ws_url working as a one-element list so existing configs do
not break.

Things worth getting right

Failover, not round-robin, for state reads. A storage read at a block hash
needs a node that has that block's state; spreading consecutive reads across
nodes with different pruning would give a mix of answers and absences that
looks like sparse data. Prefer one endpoint until it fails, then move.
A pruning node is not a broken node. state_getStorage returning null for
an old block is a legitimate answer from a node that has pruned it, and must
not mark the endpoint unhealthy. Only transport failures and malformed
responses should.
The WebSocket is a different problem. subscribe_new_heads is a long-lived
connection with reconnect logic already; picking a different endpoint on
reconnect is the natural place for failover, rather than racing two
subscriptions and deduplicating heads.
Say which endpoint answered, at least in logs. "The chain is unreachable"
and "one of three endpoints is unreachable" are different operational facts
and currently look identical.
Backfill could genuinely fan out across endpoints, since each block is
independent and it is the bulk of the traffic. Worth doing after the failover
case works, not instead of it.

Not in scope

Load balancing for its own sake. The point is that no single host can stop the
observer, not that requests are evenly spread.

> **Queued deliberately.** Not to be picked up while anything else is in flight. ## What A `[[chains]]` entry holds exactly one `rpc_url` and one `ws_url`. When that endpoint goes down, everything the observer does for that chain stops: no headers, no difficulty, no backfill, no state reads, no runtime discovery. The chain reports `unreachable` and keeps serving its last standings, which is graceful — but it is one host's outage deciding what a whole chain looks like. The endpoints already exist in pairs. `a1-planck.quantus.cat` and `a2-planck.quantus.cat` both answer today, and only the first is configured. The same `a1`/`a2` pattern holds for Heisenberg. We are choosing single points of failure that the network went to some trouble to avoid. ## Shape ```toml [[chains]] id = "planck" rpc_urls = ["https://a1-planck.quantus.cat", "https://a2-planck.quantus.cat"] ws_urls = ["wss://a1-planck.quantus.cat", "wss://a2-planck.quantus.cat"] ``` Keep `rpc_url` / `ws_url` working as a one-element list so existing configs do not break. ## Things worth getting right - **Failover, not round-robin, for state reads.** A storage read at a block hash needs a node that has that block's state; spreading consecutive reads across nodes with different pruning would give a mix of answers and absences that looks like sparse data. Prefer one endpoint until it fails, then move. - **A pruning node is not a broken node.** `state_getStorage` returning null for an old block is a legitimate answer from a node that has pruned it, and must not mark the endpoint unhealthy. Only transport failures and malformed responses should. - **The WebSocket is a different problem.** `subscribe_new_heads` is a long-lived connection with reconnect logic already; picking a different endpoint on reconnect is the natural place for failover, rather than racing two subscriptions and deduplicating heads. - **Say which endpoint answered**, at least in logs. "The chain is unreachable" and "one of three endpoints is unreachable" are different operational facts and currently look identical. - Backfill could genuinely fan out across endpoints, since each block is independent and it is the bulk of the traffic. Worth doing *after* the failover case works, not instead of it. ## Not in scope Load balancing for its own sake. The point is that no single host can stop the observer, not that requests are evenly spread.

grenade referenced this issue from a commit

2026-09-10 08:17:08 +00:00

feat: several endpoints per chain, so one host cannot stop it

grenade closed this issue

2026-09-10 08:17:08 +00:00

grenade referenced this issue

2026-09-10 08:33:04 +00:00

Mainnet has only our own endpoint; two public archive nodes exist #8

Sign in to join this conversation.