A chain entry held exactly one `rpc_url` and one `ws_url`, and when that host
went down everything for that chain stopped — headers, difficulty, backfill,
state reads, runtime discovery. The endpoints already existed in pairs:
`a2-planck` and `a2-heisenberg` both answer and neither was configured. We were
choosing single points of failure the network went to some trouble to avoid.
`rpc_urls` and `ws_urls` are lists now, and `rpc_url`/`ws_url` still work as a
one-element list — a config naming one endpoint is still a valid config, and
making every deployment rewrite its entry would be this feature breaking the
thing it exists to make reliable.
Endpoints are stuck to rather than balanced across, which is the design and not
laziness: a storage read at an old block hash needs a node that still holds that
block's state, and nodes prune on their own schedules, so alternating would
return a mixture of answers and absences that reads as sparse data rather than a
configuration problem. The cursor is shared across clones so a failover one task
finds is not rediscovered by every other task on the chain.
Failing over on the wrong thing was the trap worth avoiding. A JSON-RPC error is
the node answering — moving on `count exceeds maximum value` would hide a
caller's mistake behind a second node making the same complaint — so a new
`Malformed` variant separates "did not answer" from "answered, with an error".
A pruned block returns `{"result": null}`, a success, and never looks unhealthy.
The WebSocket rotates at reconnect, where the loop already was; racing
subscriptions across endpoints and deduplicating heads buys nothing, since heads
are a liveness signal and ingest fills gaps against `chain_getBlockHash` anyway.
Verified live with a dead endpoint configured first: Planck stayed `full` at its
real height, the RPC logged one `failed over` with from and to, and the head
subscription logged the loss with the endpoint count beside it — because "the
chain is unreachable" and "one of three endpoints is unreachable" are different
operational facts and used to look identical.
Closes#7
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jp6a8EDar9ueEhAxzep4V5
rob thijssen
1a59b2d398
