rob thijssen
1a59b2d398
A chain entry held exactly one `rpc_url` and one `ws_url`, and when that host
went down everything for that chain stopped — headers, difficulty, backfill,
state reads, runtime discovery. The endpoints already existed in pairs:
`a2-planck` and `a2-heisenberg` both answer and neither was configured. We were
choosing single points of failure the network went to some trouble to avoid.
`rpc_urls` and `ws_urls` are lists now, and `rpc_url`/`ws_url` still work as a
one-element list — a config naming one endpoint is still a valid config, and
making every deployment rewrite its entry would be this feature breaking the
thing it exists to make reliable.
Endpoints are stuck to rather than balanced across, which is the design and not
laziness: a storage read at an old block hash needs a node that still holds that
block's state, and nodes prune on their own schedules, so alternating would
return a mixture of answers and absences that reads as sparse data rather than a
configuration problem. The cursor is shared across clones so a failover one task
finds is not rediscovered by every other task on the chain.
Failing over on the wrong thing was the trap worth avoiding. A JSON-RPC error is
the node answering — moving on `count exceeds maximum value` would hide a
caller's mistake behind a second node making the same complaint — so a new
`Malformed` variant separates "did not answer" from "answered, with an error".
A pruned block returns `{"result": null}`, a success, and never looks unhealthy.
The WebSocket rotates at reconnect, where the loop already was; racing
subscriptions across endpoints and deduplicating heads buys nothing, since heads
are a liveness signal and ingest fills gaps against `chain_getBlockHash` anyway.
Verified live with a dead endpoint configured first: Planck stayed `full` at its
real height, the RPC logged one `failed over` with from and to, and the head
subscription logged the loss with the endpoint count beside it — because "the
chain is unreachable" and "one of three endpoints is unreachable" are different
operational facts and used to look identical.
Closes #7
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jp6a8EDar9ueEhAxzep4V5
71 lines
2.4 KiB
Rust
71 lines
2.4 KiB
Rust
//! Everything that talks to something outside this process: the chain's
|
|
//! JSON-RPC, the substrate-telemetry feed, and Postgres.
|
|
//!
|
|
//! The split is deliberate. `blackbeard-core` decodes and tallies with no I/O
|
|
//! at all and is exercised entirely by unit tests; this crate does the talking
|
|
//! and holds the retry, reconnect and schema concerns. `blackbeard-api` wires
|
|
//! the two together.
|
|
|
|
#![deny(missing_docs)]
|
|
|
|
pub mod rpc;
|
|
pub mod store;
|
|
pub mod telemetry;
|
|
|
|
/// Failures reaching a node, the telemetry feed, or the database.
|
|
#[derive(Debug, thiserror::Error)]
|
|
pub enum DataError {
|
|
/// The node answered with a JSON-RPC error, or an unusable result.
|
|
#[error("rpc {method}: {message}")]
|
|
Rpc {
|
|
/// Which method.
|
|
method: String,
|
|
/// What the node said.
|
|
message: String,
|
|
},
|
|
|
|
/// The HTTP request itself failed — connection refused, timeout, TLS.
|
|
#[error("http transport: {0}")]
|
|
Http(#[from] reqwest::Error),
|
|
|
|
/// A WebSocket connection failed or dropped.
|
|
///
|
|
/// Boxed: tungstenite's error is by far the largest variant here, and an
|
|
/// unboxed copy would be paid for on the stack by every `Result` in this
|
|
/// crate — including the RPC calls made several times a second.
|
|
#[error("websocket: {0}")]
|
|
WebSocket(#[from] Box<tokio_tungstenite::tungstenite::Error>),
|
|
|
|
/// A response did not have the shape we expected.
|
|
#[error("decoding a response: {0}")]
|
|
Decode(#[from] serde_json::Error),
|
|
|
|
/// A node answered with something that was not a JSON-RPC response at all.
|
|
///
|
|
/// Distinct from [`Self::Rpc`], which is the node answering *correctly*
|
|
/// with an error. This one is a host misbehaving, and so is worth trying
|
|
/// another endpoint for — the difference is what keeps a caller's own
|
|
/// mistake from walking the whole endpoint list.
|
|
#[error("malformed response to {method}: {message}")]
|
|
Malformed {
|
|
/// Which method.
|
|
method: String,
|
|
/// What was wrong with it.
|
|
message: String,
|
|
},
|
|
|
|
/// A database query failed.
|
|
#[error("database: {0}")]
|
|
Database(#[from] sqlx::Error),
|
|
|
|
/// A migration failed to apply.
|
|
#[error("migration: {0}")]
|
|
Migration(#[from] sqlx::migrate::MigrateError),
|
|
}
|
|
|
|
impl From<tokio_tungstenite::tungstenite::Error> for DataError {
|
|
fn from(e: tokio_tungstenite::tungstenite::Error) -> Self {
|
|
DataError::WebSocket(Box::new(e))
|
|
}
|
|
}
|