feat(telemetry): decode the whole of NodeDetails, and locate nodes by country

We were reading two fields of eleven — the name and the peer id — and dropping
the rest, including the hardware every node already announces. `NodeMetadata`
now carries the slow-moving half: implementation, version, the target triple,
CPU, cores, memory, kernel, distro, whether it is virtualised, and the process
start time.

The volatile half stays out on purpose. Peer count, transaction queue,
bandwidth and state cache change every few seconds, are already a rolling series
in the feed, and would be a lot of rows for something nobody would page back
through.

`startup_time` is kept rather than derived, so node uptime becomes a fact about
the node rather than about how long this observer has been watching it.

**Location is resolved to a country and no finer.** The feed sends
`[latitude, longitude, city]` and no country, so `blackbeard_core::geo` derives
one offline from a 508 KiB boundary set — no network call, and no third party
told where these nodes are. The city is read and discarded: those coordinates
come from an IP geolocation database and at city resolution are wrong often
enough that publishing one would assert something we do not know. Nothing
downstream can render a city because nothing downstream is given one. The
lookup returns subdivisions before countries, and `GB-ENG` is exactly the extra
precision this is avoiding, so only the two-letter code survives.

`LocatedNode` is a separate frame because the telemetry server geolocates
asynchronously — `AddedNode` almost always carries a null location and the
country follows a moment later. Which means a reconnect, which re-sends every
node with a null location again, must not erase a country already resolved.
There is a test for that, because it would present as countries silently
vanishing hours after a restart.

The mapping is pinned by tests built from frames captured off the live feed
rather than from a spec: the format is positional and differs between telemetry
releases, so bytes the feed actually sent are the only honest fixture. If a
future release renumbers a field, those tests are what says so.

Refs #15

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jp6a8EDar9ueEhAxzep4V5
This commit is contained in:
2026-09-13 17:10:32 +03:00
co-authored by Claude Opus 5
parent 380a0e138b
commit 0b3c1b2e1b
6 changed files with 473 additions and 0 deletions
Generated
+74
View File
@@ -286,11 +286,13 @@ dependencies = [
"blackbeard-entities",
"blake2",
"chrono",
"country-boundaries",
"frame-metadata",
"hex",
"parity-scale-codec",
"primitive-types",
"qp-poseidon-core",
"rust_iso3166",
"scale-decode",
"scale-info",
"scale-value",
@@ -534,6 +536,12 @@ version = "0.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
[[package]]
name = "country-boundaries"
version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a949524fd9d83cd48e5046cd6212c1e533477d376b82570b6c0bcc4c6775625"
[[package]]
name = "cpufeatures"
version = "0.2.17"
@@ -719,6 +727,12 @@ dependencies = [
"pin-project-lite",
]
[[package]]
name = "fastrand"
version = "2.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223"
[[package]]
name = "figment"
version = "0.10.19"
@@ -1586,6 +1600,49 @@ version = "2.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
[[package]]
name = "phf"
version = "0.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "010378780309880b08997fae13be7834dba947d36393bd372f2b1556deb2a2f6"
dependencies = [
"phf_macros",
"phf_shared",
"serde",
]
[[package]]
name = "phf_generator"
version = "0.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "aeb62e0959d5a1bebc965f4d15d9e2b7cea002b6b0f5ba8cde6cc26738467100"
dependencies = [
"fastrand",
"phf_shared",
]
[[package]]
name = "phf_macros"
version = "0.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5fa8d0ca26d424d27630da600c6624696e7dec8bf7b3b492b383c5dc49e5e085"
dependencies = [
"phf_generator",
"phf_shared",
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "phf_shared"
version = "0.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c6fd9027e2d9319be6349febd1db4e8d02aa544921200c9b777720ac34a3aa89"
dependencies = [
"siphasher",
]
[[package]]
name = "pin-project-lite"
version = "0.2.17"
@@ -1959,6 +2016,17 @@ dependencies = [
"zeroize",
]
[[package]]
name = "rust_iso3166"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "14527538ee75e1937267e5d9c9808bce0680e6051d4ee05eb9efca77e6ee801a"
dependencies = [
"js-sys",
"phf",
"wasm-bindgen",
]
[[package]]
name = "rustc-hash"
version = "2.1.3"
@@ -2287,6 +2355,12 @@ version = "0.3.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea"
[[package]]
name = "siphasher"
version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649"
[[package]]
name = "slab"
version = "0.4.12"
+4
View File
@@ -33,6 +33,10 @@ scale-value = { version = "0.18", default-features = false }
# that keeps a bogus length prefix from sizing an allocation. Pinned to the
# version `scale-value` itself resolves, so both see one type registry.
scale-decode = { version = "0.16", default-features = false }
# Offline reverse geocoding, country level only. The embedded 180x90 dataset is
# 508 KiB and needs no network at all — see `blackbeard_core::geo`.
country-boundaries = "1"
rust_iso3166 = "0.2"
# The chain's own reward-address derivation, so the observer computes the same
# account `pallets/mining-rewards` pays rather than guessing or asking.
qp-poseidon-core = { version = "3.1.0", default-features = false }
+2
View File
@@ -16,6 +16,8 @@ twox-hash.workspace = true
frame-metadata.workspace = true
hex.workspace = true
parity-scale-codec.workspace = true
country-boundaries.workspace = true
rust_iso3166.workspace = true
scale-decode.workspace = true
scale-info.workspace = true
scale-value.workspace = true
+97
View File
@@ -0,0 +1,97 @@
//! Where a node is, to the nearest country and no nearer.
//!
//! The telemetry feed sends `[latitude, longitude, city]` and **no country**,
//! so the country has to be derived from the coordinates. That is the only
//! geography this crate produces, deliberately: the coordinates come from an IP
//! geolocation database, and at city resolution those are wrong often enough
//! that publishing one would be asserting something we do not know. A city is
//! frequently the centroid of an ISP's address block rather than anywhere the
//! machine is. The country is the level at which the same lookup is usually
//! right.
//!
//! So the city the feed sends is read and discarded, and nothing downstream can
//! render it because nothing downstream is given it.
//!
//! The lookup is offline — a 508 KiB boundary set compiled into the binary — so
//! it costs no network call and cannot leak a node's coordinates to a third
//! party by asking about them.
use std::sync::OnceLock;
use country_boundaries::{BOUNDARIES_ODBL_180X90, CountryBoundaries, LatLon};
/// A country, as ISO 3166-1 alpha-2 and a name.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Country {
/// Two-letter code, e.g. `FR`.
pub code: String,
/// The ISO name, e.g. `France`. Long for some countries — the UI shows the
/// code and keeps this for the tooltip.
pub name: String,
}
fn boundaries() -> Option<&'static CountryBoundaries> {
static BOUNDARIES: OnceLock<Option<CountryBoundaries>> = OnceLock::new();
BOUNDARIES
.get_or_init(|| CountryBoundaries::from_reader(BOUNDARIES_ODBL_180X90).ok())
.as_ref()
}
/// The country containing a point, if the boundary set knows one.
///
/// `None` for the open ocean, for Antarctica's unclaimed sectors, and for
/// coordinates the feed sent that are not on Earth — all of which are absences
/// rather than errors, and all of which render as "unknown" rather than as a
/// guess.
///
/// The lookup returns subdivisions before countries (`GB-ENG` before `GB`), and
/// only the country is wanted: a subdivision is exactly the extra precision
/// this module exists not to claim.
pub fn country_at(latitude: f64, longitude: f64) -> Option<Country> {
let point = LatLon::new(latitude, longitude).ok()?;
let code = boundaries()?
.ids(point)
.into_iter()
.find(|id| id.len() == 2)?;
let name = rust_iso3166::from_alpha2(code).map(|c| c.name.to_owned());
Some(Country {
code: code.to_owned(),
name: name.unwrap_or_else(|| code.to_owned()),
})
}
#[cfg(test)]
mod tests {
use super::*;
/// Coordinates taken from real `LocatedNode` frames on the live feed.
#[test]
fn real_feed_coordinates_resolve_to_their_countries() {
for (lat, lon, expected) in [
(49.4199, 2.823, "FR"), // the feed said "Compiègne"
(52.4994, -2.0188, "GB"), // "Oldbury"
(37.9521, 126.8853, "KR"), // "Paju"
(37.1692, -3.5764, "ES"), // "Granada"
] {
let found = country_at(lat, lon).expect("a country");
assert_eq!(found.code, expected, "{lat},{lon}");
assert!(!found.name.is_empty());
}
}
/// A subdivision is the precision this module exists not to claim.
#[test]
fn only_the_country_survives_not_the_subdivision() {
let found = country_at(52.4994, -2.0188).expect("a country");
assert_eq!(found.code, "GB", "GB-ENG is a subdivision and must not win");
assert_eq!(found.code.len(), 2);
}
/// Absences, not errors: rendered as unknown rather than as a guess.
#[test]
fn points_with_no_country_are_absent_rather_than_wrong() {
assert_eq!(country_at(0.0, -140.0), None, "the middle of the Pacific");
assert_eq!(country_at(91.0, 0.0), None, "not a latitude");
assert_eq!(country_at(0.0, 181.0), None, "not a longitude");
}
}
+1
View File
@@ -15,6 +15,7 @@
pub mod attribution;
pub mod digest;
pub mod geo;
pub mod hashrate;
pub mod runtime;
pub mod scale;
+295
View File
@@ -43,6 +43,12 @@ use crate::DataError;
const ACTION_ADDED_NODE: u64 = 3;
/// `RemovedNode`: `node_id`
const ACTION_REMOVED_NODE: u64 = 4;
/// `LocatedNode`: `[node_id, latitude, longitude, city]`
///
/// Sent separately from `AddedNode` because the geolocation lookup is
/// asynchronous on the telemetry server — a node's `AddedNode` frame usually
/// carries `null` for location and this arrives a moment later.
const ACTION_LOCATED_NODE: u64 = 5;
/// `ImportedBlock`: `[node_id, [height, hash, block_time, timestamp, propagation_ms]]`
const ACTION_IMPORTED_BLOCK: u64 = 6;
/// `BestBlock`: `[height, timestamp_ms, average_block_time_ms]` — chain-wide,
@@ -76,6 +82,59 @@ struct ImportRecord {
second_ms: Option<i64>,
}
/// What the feed says a node *is*, as opposed to what it is doing.
///
/// The slow-moving half of a node's telemetry: identity, build, hardware and
/// country. Peer count, transaction queue, bandwidth and state cache all change
/// every few seconds and are deliberately not here — they belong to the live
/// feed, not to a cache.
///
/// Keyed by peer id everywhere, never by the feed's `node_id`, which is
/// per-connection and changes on every reconnect.
#[derive(Debug, Clone, Default, PartialEq)]
pub struct NodeMetadata {
/// The node's self-reported name. Not an identity claim.
pub name: String,
/// `"Quantus Node"`.
pub implementation: Option<String>,
/// `"1.0.1-f1176cea6a6"` — the client version with its commit.
pub version: Option<String>,
/// `"linux"`, `"windows"`.
pub target_os: Option<String>,
/// `"x86_64"`.
pub target_arch: Option<String>,
/// `"gnu"`, `"musl"`.
pub target_env: Option<String>,
/// `"AMD Ryzen 5 5600 6-Core Processor"`.
pub cpu: Option<String>,
/// Physical cores the node reports.
pub core_count: Option<u32>,
/// Bytes of RAM.
pub memory: Option<u64>,
/// `"7.0.0-31-generic"`.
pub linux_kernel: Option<String>,
/// `"Ubuntu 24.04.4 LTS"`.
pub linux_distro: Option<String>,
/// Whether the node believes it is virtualised.
pub is_virtual_machine: Option<bool>,
/// ISO 3166-1 alpha-2, derived from the coordinates the feed sent.
///
/// **Country and no finer.** The feed also sends a city, which is read and
/// dropped: those coordinates come from an IP geolocation database, and at
/// city resolution they are wrong often enough that publishing one would
/// assert something we do not know. Nothing downstream can render a city
/// because nothing downstream is given one.
pub country_code: Option<String>,
/// The ISO name for `country_code`.
pub country_name: Option<String>,
/// When the node process started, milliseconds since the epoch.
///
/// Stored rather than derived, so "node uptime" is a fact about the node
/// rather than about how long this observer has been watching it — and so
/// it survives our own restarts.
pub startup_time_ms: Option<u64>,
}
#[derive(Debug, Default)]
struct State {
connected: bool,
@@ -90,6 +149,11 @@ struct State {
/// Peer id to its most recent name. Survives reconnects and renames, so an
/// attribution made before a feed hiccup still renders a name after it.
peer_names: HashMap<String, String>,
/// Peer id to everything else the feed said about that node.
metadata: HashMap<String, NodeMetadata>,
/// Node id to peer id, so a later `LocatedNode` — which names only the
/// per-connection id — can find the metadata `AddedNode` created.
located: HashMap<u64, String>,
imports: HashMap<String, ImportRecord>,
import_order: VecDeque<String>,
/// Chain-wide best height, from `BestBlock`.
@@ -189,6 +253,20 @@ impl TelemetryFeed {
(!name.is_empty()).then_some(name)
}
/// Everything the feed has said about one node, by peer id.
pub fn metadata_of(&self, peer_id: &str) -> Option<NodeMetadata> {
self.read().metadata.get(peer_id).cloned()
}
/// Every node's metadata, for persisting.
pub fn all_metadata(&self) -> Vec<(String, NodeMetadata)> {
self.read()
.metadata
.iter()
.map(|(peer, m)| (peer.clone(), m.clone()))
.collect()
}
/// Seed the peer-id-to-name map from persisted state, so names restored
/// from the database resolve before the feed has re-announced their nodes.
pub fn seed_name(&self, peer_id: String, name: String) {
@@ -274,6 +352,7 @@ impl TelemetryFeed {
};
match code {
ACTION_ADDED_NODE => state.added_node(payload),
ACTION_LOCATED_NODE => state.located_node(payload),
ACTION_REMOVED_NODE => {
if let Some(id) = payload.as_u64() {
state.names.remove(&id);
@@ -345,6 +424,15 @@ impl State {
self.errors += 1;
return;
};
// `[node_id, details, stats, io, hardware, block, location, startup]`.
let location = payload.get(6).and_then(Value::as_array).and_then(|l| {
Some([
l.first().and_then(Value::as_f64)?,
l.get(1).and_then(Value::as_f64)?,
])
});
let startup = payload.get(7).and_then(Value::as_u64);
let field = |i: usize| {
details
.get(i)
@@ -357,9 +445,107 @@ impl State {
if !peer.is_empty() && !name.is_empty() {
self.peer_names.insert(peer.clone(), name.clone());
}
// Everything past the name and the peer id. Positional, because the
// feed is: see the field map in this module's header, and the tests,
// which are pinned against frames captured from the live feed.
if !peer.is_empty() {
let text = |i: usize| {
details
.get(i)
.and_then(Value::as_str)
.filter(|s| !s.is_empty())
.map(str::to_owned)
};
let sysinfo = details.get(9);
let sys = |k: &str| sysinfo.and_then(|v| v.get(k));
// `AddedNode` usually carries a null location — the telemetry
// server's geolocation lookup is asynchronous and a `LocatedNode`
// follows. Whatever is here is only a head start.
let (country_code, country_name) = match location {
Some([lat, lon]) => match blackbeard_core::geo::country_at(lat, lon) {
Some(c) => (Some(c.code), Some(c.name)),
None => (None, None),
},
None => (None, None),
};
let metadata = NodeMetadata {
name: name.clone(),
implementation: text(1),
version: text(2),
target_os: text(5),
target_arch: text(6),
target_env: text(7),
cpu: sys("cpu").and_then(Value::as_str).map(str::to_owned),
core_count: sys("core_count").and_then(Value::as_u64).map(|c| c as u32),
memory: sys("memory").and_then(Value::as_u64),
linux_kernel: sys("linux_kernel")
.and_then(Value::as_str)
.map(str::to_owned),
linux_distro: sys("linux_distro")
.and_then(Value::as_str)
.map(str::to_owned),
is_virtual_machine: sys("is_virtual_machine").and_then(Value::as_bool),
country_code,
country_name,
startup_time_ms: startup,
};
// A country already resolved by a `LocatedNode` must not be undone
// by a later `AddedNode` that carries no location — a reconnect
// re-sends every node, and the location lookup lags behind it.
let kept = self
.metadata
.get(&peer)
.filter(|_| metadata.country_code.is_none())
.and_then(|previous| {
previous
.country_code
.clone()
.map(|code| (code, previous.country_name.clone()))
});
let metadata = match kept {
Some((code, name)) => NodeMetadata {
country_code: Some(code),
country_name: name,
..metadata
},
None => metadata,
};
self.metadata.insert(peer.clone(), metadata);
self.located.insert(id, peer.clone());
}
self.names.insert(id, (name, peer));
}
/// `LocatedNode`: the country, arriving after the node it describes.
///
/// The city in this frame is read and discarded — see `NodeMetadata`.
fn located_node(&mut self, payload: &Value) {
let (Some(id), Some(lat), Some(lon)) = (
payload.get(0).and_then(Value::as_u64),
payload.get(1).and_then(Value::as_f64),
payload.get(2).and_then(Value::as_f64),
) else {
self.errors += 1;
return;
};
let Some(peer) = self.located.get(&id).cloned() else {
return;
};
let Some(entry) = self.metadata.get_mut(&peer) else {
return;
};
// A point off every boundary the set knows is an absence, not an error:
// the node keeps whatever country it already had, or none.
if let Some(country) = blackbeard_core::geo::country_at(lat, lon) {
entry.country_code = Some(country.code);
entry.country_name = Some(country.name);
}
}
fn imported_block(&mut self, payload: &Value) {
let (Some(node_id), Some(block)) = (
payload.get(0).and_then(Value::as_u64),
@@ -414,6 +600,115 @@ mod tests {
use super::*;
use serde_json::json;
/// A real `AddedNode` frame, captured from the live feed.
///
/// Verbatim apart from the name: the format is positional and undocumented
/// per version, so the only honest test is one built from bytes the feed
/// actually sent. If a future telemetry release renumbers a field, this is
/// what says so.
fn real_added_node() -> Vec<Value> {
vec![
json!(ACTION_ADDED_NODE),
json!([
7,
[
"Wallace-Vast-ODB-G",
"Quantus Node",
"1.0.1-f1176cea6a6",
null,
"QmcjjPmnyM6KYXp1U4Rdy6v2stskYQnTpgk4FH1cMfhMpB",
"linux",
"x86_64",
"gnu",
null,
{
"cpu": "AMD Ryzen 5 5600 6-Core Processor",
"memory": 65192095744u64,
"core_count": 6,
"linux_kernel": "7.0.0-31-generic",
"linux_distro": "Ubuntu 24.04.4 LTS",
"is_virtual_machine": false
},
null
],
[8, 0],
[[582678140.0]],
[[13055.4375]],
[37863, "0x2c57760aea85466e81b751d121bc7024817607ed64a770d1f7db513a19c8ae7f", 20114, 1789307752369u64, 139],
null,
1789225142469u64
]),
]
}
#[test]
fn a_node_declares_its_hardware_and_we_keep_it() {
let feed = TelemetryFeed::new();
feed.ingest(&real_added_node());
let m = feed
.metadata_of("QmcjjPmnyM6KYXp1U4Rdy6v2stskYQnTpgk4FH1cMfhMpB")
.expect("metadata by peer id");
assert_eq!(m.name, "Wallace-Vast-ODB-G");
assert_eq!(m.implementation.as_deref(), Some("Quantus Node"));
assert_eq!(m.version.as_deref(), Some("1.0.1-f1176cea6a6"));
assert_eq!(m.target_os.as_deref(), Some("linux"));
assert_eq!(m.target_arch.as_deref(), Some("x86_64"));
assert_eq!(m.target_env.as_deref(), Some("gnu"));
assert_eq!(m.cpu.as_deref(), Some("AMD Ryzen 5 5600 6-Core Processor"));
assert_eq!(m.core_count, Some(6));
assert_eq!(m.memory, Some(65_192_095_744));
assert_eq!(m.linux_kernel.as_deref(), Some("7.0.0-31-generic"));
assert_eq!(m.linux_distro.as_deref(), Some("Ubuntu 24.04.4 LTS"));
assert_eq!(m.is_virtual_machine, Some(false));
assert_eq!(m.startup_time_ms, Some(1_789_225_142_469));
// No location in the frame, and none invented.
assert_eq!(m.country_code, None);
}
/// The country arrives after the node, and only ever as a country.
#[test]
fn a_later_location_resolves_to_a_country_and_no_city() {
let feed = TelemetryFeed::new();
feed.ingest(&real_added_node());
feed.ingest(&[
json!(ACTION_LOCATED_NODE),
json!([7, 49.4199, 2.823, "Compiègne"]),
]);
let m = feed
.metadata_of("QmcjjPmnyM6KYXp1U4Rdy6v2stskYQnTpgk4FH1cMfhMpB")
.expect("metadata");
assert_eq!(m.country_code.as_deref(), Some("FR"));
assert_eq!(m.country_name.as_deref(), Some("France"));
// The city was in the frame and is nowhere in the result: nothing
// downstream can render it because nothing downstream is given it.
let rendered = format!("{m:?}");
assert!(!rendered.contains("Compi"), "{rendered}");
}
/// A reconnect re-sends every node, and the location lookup lags it.
#[test]
fn a_reconnect_does_not_undo_a_country_already_resolved() {
let feed = TelemetryFeed::new();
feed.ingest(&real_added_node());
feed.ingest(&[
json!(ACTION_LOCATED_NODE),
json!([7, 49.4199, 2.823, "Compiègne"]),
]);
// The same node announced again, still with a null location.
feed.ingest(&real_added_node());
let m = feed
.metadata_of("QmcjjPmnyM6KYXp1U4Rdy6v2stskYQnTpgk4FH1cMfhMpB")
.expect("metadata");
assert_eq!(
m.country_code.as_deref(),
Some("FR"),
"a null location must not erase a country we already knew"
);
}
fn added_node(id: u64, name: &str, peer: &str) -> Vec<Value> {
vec![
json!(ACTION_ADDED_NODE),