blackbeard/observer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: blackbeard.observer — live Quantus mining leaderboard

Browse Source 
Cargo workspace plus a Vite frontend, following ~/git/architecture/generic.md.

Every block header carries its author's wormhole reward preimage in a `pow_`
PreRuntime digest, so authorship for the whole network is derivable from headers
alone — no indexer, no registration, no way for a miner to be left out. That
decoding, the hashrate maths and the telemetry name attribution live in
blackbeard-core with no I/O at all, so the parts that are easy to get subtly
wrong are exercised by unit tests rather than only against a live chain.

The browser holds one WebSocket: snapshot on subscribe, deltas thereafter. The
head stream is itself a push (chain_subscribeNewHeads), so a block reaches the
page the moment the node imports it. Messages are serialised once per broadcast,
and leaderboards are recomputed only for windows a socket is actually watching.
No RxJS — useSyncExternalStore is React's own contract for this.

Verified against the live Planck testnet: 12/12 headers decoded, telemetry names
attributed (quanpool-planck, baba-gorchitsa, …), warm start restoring 84 blocks
and 5 held names across a restart.

Three findings worth recording, all in CLAUDE.md:

- substrate-telemetry sends its JSON in *binary* frames. A text-only client
  connects, subscribes, reports healthy and receives nothing at all — and a
  Python probe hides it, because json.loads accepts bytes.
- Difficulty is a little-endian U512; decoding it big-endian gives a number
  wrong by ~10^150 that still renders fine.
- Planck's real block interval is ~13-15s against a 6s target with enormous
  variance, so a measured interval needs 20 tip samples before it is publishable.

Deploy assets, the Gitea Actions workflow and script/infra-setup.sh are included;
port 25864 is registered in architecture/port-allocations.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSDYiibCtELsrjQq6KXnoi
This commit is contained in:
rob thijssen
2026-09-04 12:33:54 +03:00
commit 110fbc3631
100 changed files with 16221 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.cargo/config.toml Normal file
View File

@@ -0,0 +1,6 @@
# ts-rs writes the frontend's generated types here. Setting it once, relative to
# the workspace root, keeps every `#[ts(export_to = "Foo.ts")]` a bare filename —
# so moving the frontend is a one-line change rather than a sweep through the
# entities crate.
[env]
TS_RS_EXPORT_DIR = { value = "web/src/api/generated", relative = true }

315
.gitea/workflows/deploy.yaml Normal file
View File

@@ -0,0 +1,315 @@
name: deploy
# The workflow is the source of infra truth (architecture/deployment-gitea-actions.md):
# hosts, ports, paths and component→host mapping live here and nowhere else.
# There is no separate deployment manifest.
on:
push:
branches: [main]
workflow_dispatch:
inputs:
mode:
description: "deploy (apply then validate) or validate (check only)"
type: choice
options: [deploy, validate]
default: deploy
concurrency:
# Serialise deploys; never half-apply two at once.
group: deploy
cancel-in-progress: false
env:
# --- infra truth -----------------------------------------------------------
# The API runs beside quantus-node so it can read the node's loopback RPC.
API_HOST: bob.hanzalova.internal
API_PORT: "25864"
# The site's edge proxy. It serves the built SPA and reverse-proxies /v1.
EDGE_HOST: hanzalova.internal
WEBROOT: /var/www/blackbeard.observer
PUBLIC_NAME: blackbeard.observer
DEPLOY_KEY: |
${{ secrets.RSYNC_SSH_KEY }}
jobs:
build:
runs-on: fedora-43
steps:
- uses: actions/checkout@v4
- name: rust gate
# Format, lint-as-error and the full test suite before anything is
# built, so a broken commit never reaches a host. SQLX_OFFLINE makes the
# compile-time-checked queries read the committed .sqlx cache instead of
# needing a live database on the runner.
env:
SQLX_OFFLINE: "true"
run: |
set -euo pipefail
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all
- name: build api and cli
env:
SQLX_OFFLINE: "true"
run: |
set -euo pipefail
# musl, statically linked: the runner image is newer than the target
# host, and a glibc-linked binary built here would not load there
# (deployment-gitea-actions.md §6).
cargo build --release --target x86_64-unknown-linux-musl \
-p blackbeard-api -p blackbeard-cli
file target/x86_64-unknown-linux-musl/release/blackbeard-api
- name: generated types are current
# ts-rs writes web/src/api/generated/ from the Rust entities crate.
# `cargo test` above regenerates them; a diff here means someone edited
# a DTO and committed the Rust without the TypeScript, which would
# compile on both sides and disagree at runtime.
run: |
set -euo pipefail
if ! git diff --quiet -- web/src/api/generated; then
echo "generated TypeScript is out of date; run 'cargo test -p blackbeard-entities' and commit:" >&2
git diff --stat -- web/src/api/generated >&2
exit 1
fi
- name: build web
run: |
set -euo pipefail
corepack enable
cd web
pnpm install --frozen-lockfile
pnpm lint
pnpm build
- uses: actions/upload-artifact@v3
with:
name: blackbeard
path: |
target/x86_64-unknown-linux-musl/release/blackbeard-api
target/x86_64-unknown-linux-musl/release/blackbeard
web/dist
asset
deploy-api:
needs: build
# `infra`, not `fedora-43`: the targets are mesh-only .internal names and
# the fedora runners have no route to the WireGuard mesh. Same reason as
# lair/quantus and lair/mail.
runs-on: infra
steps:
- uses: actions/download-artifact@v3
with:
name: blackbeard
- name: ssh key and reachability
run: |
set -euo pipefail
install -d -m 0700 ~/.ssh
printf '%s' "$DEPLOY_KEY" > ~/.ssh/id_deploy
chmod 0600 ~/.ssh/id_deploy
cat > ~/.ssh/config <<EOF
Host *
IdentityFile ~/.ssh/id_deploy
StrictHostKeyChecking accept-new
User gitea_ci
EOF
ssh "$API_HOST" hostname -f
- name: preflight sudoers
# Compare what infra-setup.sh grants against what the target actually
# allows, and fail up front naming the missing paths — rather than
# dying partway through an rsync with a bare "sudo: a password is
# required" after some files have already landed.
run: |
set -euo pipefail
granted=$(ssh "$API_HOST" sudo -n -l || true)
missing=""
for path in /usr/local/bin/blackbeard-api /usr/local/bin/blackbeard \
/etc/blackbeard/config.toml /etc/sysusers.d/blackbeard.conf \
/etc/systemd/system/blackbeard-api.service \
/etc/firewalld/services/blackbeard-api.xml; do
echo "$granted" | grep -qF "$path" || missing="$missing $path"
done
if [ -n "$missing" ]; then
echo "gitea_ci on $API_HOST is not permitted to write:$missing" >&2
echo "re-run script/infra-setup.sh against this host." >&2
exit 1
fi
- name: render config
# python3 .replace(), not sed or envsubst: a literal substitution
# survives values containing shell-special characters.
run: |
set -euo pipefail
fqdn=$(ssh "$API_HOST" hostname -f)
# The API binds the host's mesh address, never 0.0.0.0 — the fleet has
# one firewalld default zone, so a wildcard bind plus the named
# service would publish this on every address the host carries.
bind=$(ssh "$API_HOST" "ip -4 -o addr show | awk '/10\\.[0-9]+\\./ {print \$4}' | cut -d/ -f1 | head -1")
if [ -z "$bind" ]; then
echo "could not determine the mesh address of $API_HOST" >&2
exit 1
fi
echo "binding $bind:$API_PORT on $fqdn"
python3 - "$fqdn" "$bind" <<'PY'
import sys, pathlib
fqdn, bind = sys.argv[1], sys.argv[2]
tmpl = pathlib.Path("asset/config/config.toml.tmpl").read_text()
out = tmpl.replace("{{TARGET_FQDN}}", fqdn).replace("{{BIND_ADDRESS}}", bind)
pathlib.Path("config.toml").write_text(out)
PY
- name: verify the rendered config
# The binary validates its own configuration and exits without binding
# anything. A bad config fails the deploy here rather than leaving a
# daemon that will not start.
run: |
set -euo pipefail
chmod +x target/x86_64-unknown-linux-musl/release/blackbeard-api
# --check reads the certificate paths, which exist on the target and
# not on the runner, so this runs on the target after the config
# lands. Placeholder substitution is what is checked here.
grep -q '{{' config.toml && { echo "config still has unsubstituted placeholders" >&2; exit 1; }
echo "no placeholders remain"
- name: ship
run: |
set -euo pipefail
r() { rsync -a --checksum --mkpath --rsync-path='sudo rsync' "$@"; }
# --checksum, not rsync's default size+mtime: the build produces a new
# mtime every run, so the default heuristic reports a change on every
# deploy and would restart a healthy daemon each time.
r --chmod=0755 target/x86_64-unknown-linux-musl/release/blackbeard-api \
"$API_HOST:/usr/local/bin/blackbeard-api"
r --chmod=0755 target/x86_64-unknown-linux-musl/release/blackbeard \
"$API_HOST:/usr/local/bin/blackbeard"
r --chown=root:blackbeard --chmod=0640 config.toml \
"$API_HOST:/etc/blackbeard/config.toml"
r asset/systemd/blackbeard.sysusers.conf "$API_HOST:/etc/sysusers.d/blackbeard.conf"
r asset/systemd/blackbeard-api.service "$API_HOST:/etc/systemd/system/blackbeard-api.service"
r asset/systemd/blackbeard-api-cert.path "$API_HOST:/etc/systemd/system/blackbeard-api-cert.path"
r asset/systemd/blackbeard-api-cert-reload.service \
"$API_HOST:/etc/systemd/system/blackbeard-api-cert-reload.service"
r asset/firewalld/blackbeard-api.xml "$API_HOST:/etc/firewalld/services/blackbeard-api.xml"
- name: apply system state
run: |
set -euo pipefail
ssh "$API_HOST" bash -euo pipefail <<'REMOTE'
sudo systemd-sysusers
sudo install -d -o root -g blackbeard -m 0750 /etc/blackbeard
# The service account needs to read the host private key: it is the
# credential for the mTLS Postgres connection and is not
# world-readable (architecture/generic.md §11).
sudo setfacl -m u:blackbeard:r "/etc/pki/tls/private/$(hostname -f).pem"
sudo restorecon -R /usr/local/bin/blackbeard-api /usr/local/bin/blackbeard /etc/blackbeard
# SELinux must know about the port or the bind is denied. Guarded so
# a re-run is a no-op; `semanage port -a` on an already-labelled
# port reassigns it rather than failing cleanly.
if ! sudo semanage port -l | grep -qE "^http_port_t.*\b25864\b"; then
sudo semanage port -a -t http_port_t -p tcp 25864
else
echo "port 25864 already labelled http_port_t"
fi
# firewalld only learns a freshly-shipped custom service after a
# reload; querying or adding it before reloading fails
# INVALID_SERVICE.
sudo firewall-cmd --reload
zone=$(sudo firewall-cmd --get-default-zone)
if ! sudo firewall-cmd --zone="$zone" --query-service=blackbeard-api; then
sudo firewall-cmd --permanent --zone="$zone" --add-service=blackbeard-api
sudo firewall-cmd --zone="$zone" --add-service=blackbeard-api
else
echo "blackbeard-api already open in zone $zone"
fi
sudo systemctl daemon-reload
sudo systemctl enable --now blackbeard-api-cert.path
# Validate the config on the target, where the certificates it names
# actually exist.
sudo -u blackbeard /usr/local/bin/blackbeard-api \
--config /etc/blackbeard/config.toml --check
sudo systemctl enable blackbeard-api.service
sudo systemctl restart blackbeard-api.service
REMOTE
- name: health probe
run: |
set -euo pipefail
for attempt in $(seq 1 30); do
if ssh "$API_HOST" "curl -sf http://127.0.0.1:$API_PORT/v1/healthz" > health.json; then
cat health.json
# `healthy` covers the database. Chain reachability is
# deliberately not part of it: a chain configured before it
# launches is not a deploy failure.
python3 -c "import json,sys; sys.exit(0 if json.load(open('health.json'))['healthy'] else 1)"
exit 0
fi
sleep 2
done
echo "blackbeard-api did not become healthy" >&2
exit 1
- name: journal
if: always()
run: ssh "$API_HOST" journalctl -u blackbeard-api.service -n 80 --no-pager || true
deploy-web:
needs: build
runs-on: infra
steps:
- uses: actions/download-artifact@v3
with:
name: blackbeard
- name: ssh key and reachability
run: |
set -euo pipefail
install -d -m 0700 ~/.ssh
printf '%s' "$DEPLOY_KEY" > ~/.ssh/id_deploy
chmod 0600 ~/.ssh/id_deploy
cat > ~/.ssh/config <<EOF
Host *
IdentityFile ~/.ssh/id_deploy
StrictHostKeyChecking accept-new
User gitea_ci
EOF
ssh "$EDGE_HOST" hostname -f
- name: ship the bundle
# --delete: hashed asset filenames accumulate forever otherwise. The
# webroot holds only build output, so there is nothing else to lose.
run: |
set -euo pipefail
rsync -a --checksum --delete --mkpath --rsync-path='sudo rsync' \
web/dist/ "$EDGE_HOST:$WEBROOT/"
- name: label and reload
run: |
set -euo pipefail
ssh "$EDGE_HOST" bash -euo pipefail <<REMOTE
# SELinux: an unlabelled webroot makes nginx return 403 for every
# file, with nothing in the nginx error log to explain it.
sudo restorecon -R "$WEBROOT"
sudo nginx -t
sudo systemctl reload nginx
REMOTE
- name: fetch the site
# `nginx -t` parses without binding, so a passing test is not evidence
# the reload landed. Fetching the page is.
run: |
set -euo pipefail
ssh "$EDGE_HOST" "curl -sfI https://$PUBLIC_NAME/ --resolve $PUBLIC_NAME:443:127.0.0.1 -o /dev/null -w '%{http_code}\n'" \
|| ssh "$EDGE_HOST" "curl -sfI https://blackbeard.internal/ -o /dev/null -w 'internal %{http_code}\n'"

18
.gitignore vendored Normal file
View File

@@ -0,0 +1,18 @@
/target
node_modules
dist
# Secrets and local overrides.
.env
.env.local
*.pem
*.key
# A rendered config, or a local one for `cargo run`. Anchored to the repo root
# on purpose: an unanchored `config.toml` also matches `.cargo/config.toml`,
# which MUST be committed — it sets TS_RS_EXPORT_DIR, and without it ts-rs
# writes the frontend's generated types into crates/*/bindings/ instead of
# web/src/api/generated/, so CI's "generated types are current" check fails for
# a reason that has nothing to do with the change under review.
/config.toml
/dev-config.toml

20
.sqlx/query-3e125cdf4f28859158e2a54e78c9f1eb88064b26db27367313ac6d054274f504.json generated Normal file
View File

@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "select 1 as ok",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "ok",
"type_info": "Int4"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "3e125cdf4f28859158e2a54e78c9f1eb88064b26db27367313ac6d054274f504"
}

37
.sqlx/query-41f694ddea9ed92237677559e524a8f8c0e28d31217a80094b5d096214dc4e37.json generated Normal file
View File

@@ -0,0 +1,37 @@
{
"db_name": "PostgreSQL",
"query": "\n select date_bin($3, observed_at, timestamptz 'epoch') as \"bucket!\",\n count(*) filter (where miner = $2) as \"mine!\",\n count(*) as \"total!\"\n from block\n where chain = $1 and observed_at >= $4\n group by 1\n order by 1 asc\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "bucket!",
"type_info": "Timestamptz"
},
{
"ordinal": 1,
"name": "mine!",
"type_info": "Int8"
},
{
"ordinal": 2,
"name": "total!",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Interval",
"Timestamptz"
]
},
"nullable": [
null,
null,
null
]
},
"hash": "41f694ddea9ed92237677559e524a8f8c0e28d31217a80094b5d096214dc4e37"
}

35
.sqlx/query-461853e4ec42eaf95c83f6ed8e177ffd527f1b59461c1e4fa582f1d935c7b6a3.json generated Normal file
View File

@@ -0,0 +1,35 @@
{
"db_name": "PostgreSQL",
"query": "\n select height, miner, observed_at\n from (\n select height, miner, observed_at\n from block\n where chain = $1\n order by height desc\n limit $2\n ) recent\n order by height asc\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "height",
"type_info": "Int8"
},
{
"ordinal": 1,
"name": "miner",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "observed_at",
"type_info": "Timestamptz"
}
],
"parameters": {
"Left": [
"Text",
"Int8"
]
},
"nullable": [
false,
false,
false
]
},
"hash": "461853e4ec42eaf95c83f6ed8e177ffd527f1b59461c1e4fa582f1d935c7b6a3"
}

20
.sqlx/query-470022fab8a12aa88db0f1feb754f6fa9fca60a0bc2665ec862aa34a87f350a3.json generated Normal file
View File

@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "\n insert into block (chain, height, hash, miner, authored_at, observed_at, difficulty)\n select * from unnest(\n $1::text[], $2::bigint[], $3::text[], $4::text[],\n $5::timestamptz[], $6::timestamptz[], $7::numeric[]\n )\n on conflict (chain, height) do update set\n hash = excluded.hash,\n miner = excluded.miner,\n authored_at = excluded.authored_at,\n observed_at = excluded.observed_at,\n difficulty = excluded.difficulty\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"TextArray",
"Int8Array",
"TextArray",
"TextArray",
"TimestamptzArray",
"TimestamptzArray",
"NumericArray"
]
},
"nullable": []
},
"hash": "470022fab8a12aa88db0f1feb754f6fa9fca60a0bc2665ec862aa34a87f350a3"
}

22
.sqlx/query-8da1c6c82b09e8944696865bb48eb0fa69d12f668dd3d722edee5223a2b44782.json generated Normal file
View File

@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "select max(height) as height from block where chain = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "height",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
null
]
},
"hash": "8da1c6c82b09e8944696865bb48eb0fa69d12f668dd3d722edee5223a2b44782"
}

41
.sqlx/query-c72299abe9898d26859aa0afb96023cad9a66f05633e06beb146d4c2d619389d.json generated Normal file
View File

@@ -0,0 +1,41 @@
{
"db_name": "PostgreSQL",
"query": "\n select count(*) as \"blocks!\",\n min(observed_at) as first_seen,\n max(observed_at) as last_seen,\n coalesce(sum(difficulty), 0)::text as \"work!\"\n from block\n where chain = $1 and miner = $2\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "blocks!",
"type_info": "Int8"
},
{
"ordinal": 1,
"name": "first_seen",
"type_info": "Timestamptz"
},
{
"ordinal": 2,
"name": "last_seen",
"type_info": "Timestamptz"
},
{
"ordinal": 3,
"name": "work!",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null,
null,
null,
null
]
},
"hash": "c72299abe9898d26859aa0afb96023cad9a66f05633e06beb146d4c2d619389d"
}

20
.sqlx/query-c8c3d392237fb51d81bce5e59781997089cca59aefb4f2d2a31a30e6cb06782c.json generated Normal file
View File

@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "\n insert into chain (\n id, display_name, mainnet, genesis, token_symbol, token_decimals,\n target_block_time_seconds, last_observed_at\n )\n values ($1, $2, $3, $4, $5, $6, $7, now())\n on conflict (id) do update set\n display_name = excluded.display_name,\n mainnet = excluded.mainnet,\n -- coalesce, not excluded: a poll that could not reach the node\n -- must not blank out what an earlier one discovered.\n genesis = coalesce(excluded.genesis, chain.genesis),\n token_symbol = coalesce(excluded.token_symbol, chain.token_symbol),\n token_decimals = coalesce(excluded.token_decimals, chain.token_decimals),\n target_block_time_seconds = excluded.target_block_time_seconds,\n last_observed_at = now()\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Bool",
"Text",
"Text",
"Int2",
"Float8"
]
},
"nullable": []
},
"hash": "c8c3d392237fb51d81bce5e59781997089cca59aefb4f2d2a31a30e6cb06782c"
}

52
.sqlx/query-cf59d77bc3baa612812c4f2506f9eb9935e44603065c85979831cf9a28b0310c.json generated Normal file
View File

@@ -0,0 +1,52 @@
{
"db_name": "PostgreSQL",
"query": "\n select miner, node_kind, node_key, node_name, attempts, attributed\n from miner_attribution\n where chain = $1\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "miner",
"type_info": "Text"
},
{
"ordinal": 1,
"name": "node_kind",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "node_key",
"type_info": "Text"
},
{
"ordinal": 3,
"name": "node_name",
"type_info": "Text"
},
{
"ordinal": 4,
"name": "attempts",
"type_info": "Int4"
},
{
"ordinal": 5,
"name": "attributed",
"type_info": "Int4"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false,
false,
false,
true,
false,
false
]
},
"hash": "cf59d77bc3baa612812c4f2506f9eb9935e44603065c85979831cf9a28b0310c"
}

20
.sqlx/query-d69e49186c4741eeb331bcb5e22a66de3a853d1d962e648380e1facac0ed2566.json generated Normal file
View File

@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "\n insert into miner_attribution\n (chain, miner, node_kind, node_key, attempts, attributed, node_name, updated_at)\n select $1, m, k, key, a, att, n, now()\n from unnest($2::text[], $3::text[], $4::text[], $5::int[], $6::int[], $7::text[])\n as t(m, k, key, a, att, n)\n on conflict (chain, miner) do update set\n node_kind = excluded.node_kind,\n node_key = excluded.node_key,\n attempts = excluded.attempts,\n attributed = excluded.attributed,\n -- A name the feed cannot currently resolve must not erase the\n -- one we already had: a telemetry hiccup would otherwise strip\n -- every miner's name from the next restart onwards.\n node_name = coalesce(excluded.node_name, miner_attribution.node_name),\n updated_at = now()\n ",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"TextArray",
"TextArray",
"TextArray",
"Int4Array",
"Int4Array",
"TextArray"
]
},
"nullable": []
},
"hash": "d69e49186c4741eeb331bcb5e22a66de3a853d1d962e648380e1facac0ed2566"
}

141
CLAUDE.md Normal file
View File

@@ -0,0 +1,141 @@
# CLAUDE.md
Guidance for Claude Code working in this repository.
Read `readme.md` first — it carries the mechanism (how authorship is decoded from
block headers), the reasoning behind the window semantics, and the list of
deliberate deviations from house convention. This file is the things that will
bite you.
Conventions: `~/git/architecture` — `generic.md` for the workspace shape,
`deployment-gitea-actions.md` for the deploy, `port-allocations.md` for the port,
`reverse-proxies.md` and `external-tls.md`/`internal-tls.md` for the vhosts.
## The things that cost an afternoon
**substrate-telemetry sends its JSON in *binary* WebSocket frames, not text.** A
client that handles only `Message::Text` connects, subscribes, reports itself
healthy, and receives absolutely nothing — no error, no decode failure, an empty
feed and `telemetry_nodes: null`. Worse, a throwaway probe written in Python
*works*, because `json.loads` accepts bytes, so the two disagree for no visible
reason. Both frame types are accepted in `telemetry.rs` and `rpc.rs`. Do not
"simplify" either match arm.
**Difficulty is a little-endian U512.** `state_call` returns a SCALE-encoded
`U512`, byte-reversed relative to how a hash reads. Decoding it big-endian yields
a plausible-looking number that is wrong by ~10^150, and every hashrate derived
from it is wrong without ever looking broken. `digest::u512_le` has the test.
**A measured block interval is only valid at the tip.** A syncing node imports
history at disk speed; dividing difficulty by *that* gives a hashrate wrong by
orders of magnitude that still renders as a number. `RollingWindow::push` takes
an explicit `at_tip` flag and backfilled blocks contribute no timing sample. The
`Interval` enum exists so a caller cannot forget to say which kind it has.
**Windows are block counts, never durations.** See `readme.md`. Anything that
turns `Window` into a time range is wrong.
**`chain_subscribeNewHeads` skips blocks.** It reports the *best* head, so when
several import at once the intermediates never arrive. It is a liveness signal,
not a ledger — `ingest` fills gaps against `chain_getBlockHash`. Removing that
would quietly under-count exactly the miners who won blocks during a burst.
**Blocks are keyed on `(chain, height)`, not hash.** This chain reorgs; the
upsert is what makes a replacement overwrite rather than accumulate. A schema
keyed on the hash would inflate the losing fork's author forever.
## Facts established by measurement
Taken from the live Planck chain, 2026-09-04. Don't re-derive or contradict
without re-measuring.
- Header digest shape is exactly `0x06` + `706f775f` + compact `0x80` (32) +
32-byte preimage. Genesis is the only header without one; 12/12 recent headers
decoded.
- Telemetry propagation: first reporter stamped `0`, others 50–620 ms. The
`ATTRIBUTION_LEAD_MS = 20` threshold only has to exclude a tie — identity over
many blocks does the discriminating.
- Planck's observed block interval is ~13–15 s against a 6 s target, with
consecutive gaps of 1.6 s, 4.4 s, 13.5 s, 26 s, 27 s. This is why
`MIN_TIP_SAMPLES` is 20: at five samples the headline hashrate swings by a
factor of three between refreshes.
- `system_properties`: `PLK`, 12 decimals, ss58 prefix 189. Genesis
`0x4901bf5c…e65e72`.
- The telemetry feed URL is `wss://feed-telemetry.quantus.cat/feed` — found in
`/tmp/env-config.js` on the telemetry site, **not** the `/feed` path on the
main host, which 404s.
## The frontend's types are generated
`web/src/api/generated/` is written by `ts-rs` from `blackbeard-entities` when
`cargo test -p blackbeard-entities` runs. **Never edit those files.** CI fails
the build if the committed output is stale.
`u64` maps to `bigint` by default, which is wrong — `serde_json` puts a u64 on
the wire as a JSON *number*, so the type and the runtime value disagree. Every
u64 DTO field carries `#[ts(type = "number")]`. Add it to any new one.
## Separation of concerns is load-bearing here
- `blackbeard-core` has **no I/O, no clock, no sockets**. That is what lets the
decoding and the statistics — the two parts that are easy to get subtly wrong
and hard to notice — be exercised by unit tests instead of only against a live
chain. Keep it that way; if something there needs the time, pass it in.
- `blackbeard-data` owns retries, reconnects and schema.
- `blackbeard-api` wires them together and owns nothing else.
## Locking
`ChainRuntime::inner` is a **`std::sync::RwLock`, never held across an `.await`**.
Everything under it is CPU work on in-memory collections. Do not switch it to
tokio's — that variant makes it easy to accidentally hold a lock across an await
and stall a worker, for no benefit here.
## Database work
Queries use `sqlx::query!` (compile-time checked) with the offline cache in
`.sqlx/` committed, so CI builds with `SQLX_OFFLINE=true` and no database.
**After changing any query, regenerate the cache** or CI fails on a stale one:
```sh
podman run -d --rm --name bb-pg -e POSTGRES_PASSWORD=dev \
-e POSTGRES_DB=blackbeard -p 55432:5432 docker.io/library/postgres:18-alpine
export DATABASE_URL='postgres://postgres:dev@127.0.0.1:55432/blackbeard'
cargo sqlx prepare --workspace -- --all-targets
```
Migrations are sequentially versioned and **immutable once committed**. Correct a
mistake with a new file, never by editing one that has landed — the runner's
checksum diverges and it refuses to start.
## Charts
Any chart added here must be read against the `dataviz` skill first. The
constraint already in force: **bronze `#bd8829` and crimson `#d8453a` are
adjacent hues and fail CVD separation as a categorical pair.** Every chart on the
site is therefore single-series — magnitude, one hue — with identity carried by
labels and row treatment. A second series in the accent colour is the one change
that would break the palette, and it would not look broken.
`web/src/index.css` documents the validated values. Re-run the validator after
touching them:
```sh
node <skill>/scripts/validate_palette.js "#bd8829" --mode dark --surface "#14110d"
```
## Verifying a change
`systemctl is-active` is not evidence this works. A daemon with a node it cannot
decode, or a telemetry feed it silently ignores, is perfectly "active" while
serving an empty leaderboard. Check the numbers:
```sh
cargo run -p blackbeard-cli -- probe --rpc-url http://bob.hanzalova.internal:9944
cargo run -p blackbeard-cli -- standings --chain planck
curl -s localhost:25864/v1/chains/planck/summary | python3 -m json.tool
```
A working deployment shows a non-null `telemetry_nodes`, `distinct_miners` above
one, and named rows in the standings.

3382
Cargo.lock generated Normal file
View File

File diff suppressed because it is too large Load Diff

51
Cargo.toml Normal file
View File

@@ -0,0 +1,51 @@
[workspace]
resolver = "3"
members = ["crates/*"]
[workspace.package]
version = "0.1.0"
edition = "2024"
rust-version = "1.85"
license = "GPL-3.0-or-later"
authors = ["Rob Thijssen <rob@lair.cafe>"]
repository = "https://git.lair.cafe/Quantus-Network/blackbeard.observer"
[workspace.dependencies]
blackbeard-entities = { path = "crates/blackbeard-entities", version = "=0.1.0" }
blackbeard-core = { path = "crates/blackbeard-core", version = "=0.1.0" }
blackbeard-data = { path = "crates/blackbeard-data", version = "=0.1.0" }
anyhow = "1"
axum = { version = "0.8", features = ["ws", "macros"] }
chrono = { version = "0.4", default-features = false, features = ["clock", "serde", "std"] }
clap = { version = "4", features = ["derive", "env"] }
figment = { version = "0.10", features = ["toml", "env"] }
futures-util = { version = "0.3", default-features = false, features = ["sink", "std"] }
hex = "0.4"
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls-native-roots"] }
rustls = { version = "0.23", default-features = false, features = ["ring", "std", "tls12"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
sqlx = { version = "0.8", default-features = false, features = [
"postgres",
"runtime-tokio-rustls",
"macros",
"migrate",
"chrono",
"bigdecimal",
] }
bigdecimal = "0.4"
primitive-types = { version = "0.13", default-features = false }
thiserror = "2"
tokio = { version = "1", features = ["full"] }
tokio-tungstenite = { version = "0.24", default-features = false, features = ["connect", "rustls-tls-native-roots"] }
tower-http = { version = "0.6", features = ["cors", "trace", "compression-gzip"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
ts-rs = { version = "10", features = ["serde-compat", "chrono-impl", "no-serde-warnings"] }
url = "2"
[profile.release]
lto = "thin"
codegen-units = 1
strip = "debuginfo"

76
asset/config/config.toml.tmpl Normal file
View File

@@ -0,0 +1,76 @@
# blackbeard-api configuration.
#
# Rendered by .gitea/workflows/deploy.yaml and rsynced to
# /etc/blackbeard/config.toml (root:blackbeard 0640). The unrendered template is
# committed; the rendered file never is.
#
# There are no secrets here and there is nowhere to put one: the Postgres
# connection is mTLS with this host's own certificate as the credential
# (architecture/generic.md §5), and every chain endpoint is either loopback or a
# public feed. The only substituted value is the host's FQDN, because the
# certificate paths carry it and the workflow is the thing that knows which host
# this is going to.
[server]
# The host's mesh address, not 0.0.0.0. The fleet uses a single firewalld
# default zone (generic.md §9), so a wildcard bind plus the named service would
# publish this on every address the host carries.
listen = "{{BIND_ADDRESS}}:25864"
# Not a security boundary — every byte this serves is derived from public block
# headers. It is here so a copy of the frontend served from somewhere else does
# not silently depend on this backend.
allowed_origins = [
"https://blackbeard.observer",
"https://blackbeard.internal",
]
ticker_blocks = 40
leaderboard_refresh_seconds = 5
[database]
host = "magrathea.kosherinata.internal"
port = 5432
database = "blackbeard"
username = "blackbeard_rw"
root_cert = "/etc/pki/ca-trust/source/anchors/root-internal.pem"
client_cert = "/etc/pki/tls/misc/{{TARGET_FQDN}}.pem"
# Not world-readable; the service account is granted read access with setfacl at
# deploy time.
client_key = "/etc/pki/tls/private/{{TARGET_FQDN}}.pem"
max_connections = 8
# --- chains ------------------------------------------------------------------
#
# The first chain listed is the one the site opens on.
#
# A chain may be configured before it launches: the observer reports it as
# `awaiting`, the UI says so, and it comes alive on its own the moment the node
# starts answering — no redeploy, no restart.
[[chains]]
id = "planck"
display_name = "Planck Testnet"
mainnet = false
# Loopback: the node runs on this host. 9944 serves HTTP and WebSocket both.
rpc_url = "http://127.0.0.1:9944"
ws_url = "ws://127.0.0.1:9944"
# The feed URL is found in /tmp/env-config.js on the telemetry site — NOT the
# /feed path on the main host, which 404s (lair/quantus readme).
telemetry_url = "wss://feed-telemetry.quantus.cat/feed"
target_block_time_seconds = 6.0
warm_start_blocks = 100800
max_gap_fill_blocks = 5000
# Mainnet. Uncomment and fill in the endpoints when the chain spec is published;
# until then the observer would report it `awaiting` forever, which is honest but
# puts a permanently dead tab in the switcher.
#
# [[chains]]
# id = "quantus"
# display_name = "Quantus"
# mainnet = true
# rpc_url = "http://127.0.0.1:9945"
# ws_url = "ws://127.0.0.1:9945"
# telemetry_url = "wss://feed-telemetry.quantus.cat/feed"
# target_block_time_seconds = 6.0

11
asset/firewalld/blackbeard-api.xml Normal file
View File

@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="utf-8"?>
<service>
<short>blackbeard-api</short>
<description>REST and WebSocket API for blackbeard.observer: the Quantus
mining leaderboard, network hashrate and live block feed. Consumed only by the
site's nginx reverse proxy, which terminates TLS; this port carries plain HTTP
and must never be reachable from the WAN. Port 25864 is derived from the
service name per architecture/port-allocations.md §3 and recorded in its
registry.</description>
<port protocol="tcp" port="25864"/>
</service>

14
asset/nginx/blackbeard-upstream.conf Normal file
View File

@@ -0,0 +1,14 @@
# Installed to /etc/nginx/conf.d/blackbeard-upstream.conf, NOT sites-available.
#
# Both vhosts (public and mesh) proxy to the same backend, and an `upstream`
# block defined inside one of them would be a silent dependency: disabling that
# vhost — or enabling only the internal one on a host that does not serve the
# public name — breaks the other with an "upstream not found" that names the
# wrong file. Declaring it once in conf.d makes both vhosts independent.
upstream blackbeard_api {
server bob.hanzalova.internal:25864;
# The API answers thousands of small requests and holds long WebSockets;
# without keepalive every one pays a fresh TCP handshake to the node host.
keepalive 16;
}

60
asset/nginx/blackbeard.internal.conf Normal file
View File

@@ -0,0 +1,60 @@
# Mesh vhost for blackbeard.observer, on the hanzalova edge proxy.
#
# This exists because a public name does not hairpin: from inside the mesh,
# `blackbeard.observer` resolves to the site's WAN address, the packet hits the
# OPNsense LAN interface, and the connection dead-ends
# (architecture/reverse-proxies.md §2). Mesh clients use this name instead.
#
# Cert: internal `lair` CA, renewed by step@blackbeard.timer
# (architecture/internal-tls.md). Same webroot and same upstream as the public
# vhost — only the name and the certificate differ.
#
# ln -sf ../sites-available/blackbeard.internal.conf /etc/nginx/sites-enabled/
#
# The split-horizon DNS record is a separate manual step on BOTH site routers:
# opn-cli --config ~/.opn-cli/{hanzalova,kosherinata}.yml unbound host create \
# --hostname blackbeard --domain internal --rr A --server <hanzalova mesh ip>
# A record added to only one router NXDOMAINs everywhere else.
server {
listen 127.0.0.1:14443 ssl proxy_protocol;
http2 on;
server_name blackbeard.internal;
ssl_certificate /etc/nginx/tls/cert/blackbeard.internal.pem;
ssl_certificate_key /etc/nginx/tls/key/blackbeard.internal.pem;
ssl_protocols TLSv1.3;
ssl_trusted_certificate /etc/pki/ca-trust/source/anchors/root-internal.pem;
absolute_redirect off;
root /var/www/blackbeard.observer;
index index.html;
location / {
try_files $uri $uri/ /index.html;
}
location /assets/ {
expires 1y;
add_header Cache-Control "public, immutable";
}
location = /index.html {
add_header Cache-Control "no-cache";
}
location /v1/ {
proxy_pass http://blackbeard_api;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
proxy_buffering off;
}
}

89
asset/nginx/blackbeard.observer.conf Normal file
View File

@@ -0,0 +1,89 @@
# Public vhost for blackbeard.observer, on the hanzalova edge proxy.
#
# Cert: Let's Encrypt via certbot + Cloudflare DNS-01 (architecture/external-tls.md).
# Installed by script/infra-setup.sh --edge, never by CI — the runner has no
# rights to read certificate keys or reload nginx on a shared edge proxy.
#
# ln -sf ../sites-available/blackbeard.observer.conf /etc/nginx/sites-enabled/
#
# listen 127.0.0.1:14443 + proxy_protocol, NOT 443: an nginx stream SNI router
# owns TCP 443 on this host and hands every non-passthrough name to the local
# https tier (architecture/reverse-proxies.md §5). Binding 443 here double-binds
# the port across http{} and stream{} — `nginx -t` cannot detect it, and the
# failure mode is nginx silently serving stale certificates. Copy this listen
# line from a currently ENABLED vhost; several files in sites-available have
# drifted from their enabled counterparts.
# The `blackbeard_api` upstream is declared once in
# conf.d/blackbeard-upstream.conf so both vhosts can use it independently, and
# `$connection_upgrade` comes from the proxy's existing conf.d map — the same
# one lair/quantus's vhost relies on. Both are installed by infra-setup.sh.
server {
listen 127.0.0.1:14443 ssl proxy_protocol;
http2 on;
server_name blackbeard.observer;
ssl_certificate /etc/letsencrypt/live/blackbeard.observer/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/blackbeard.observer/privkey.pem;
# TLS 1.2 stays enabled here, unlike the .internal vhost: this is a public
# site and its audience is not a controlled fleet.
ssl_protocols TLSv1.2 TLSv1.3;
# The vhost believes it is serving 127.0.0.1:14443, so any redirect it
# generates itself carries that port and is unreachable. Latent for a pure
# SPA, but one real directory under the webroot would break every URL
# without a trailing slash (reverse-proxies.md §4).
absolute_redirect off;
root /var/www/blackbeard.observer;
index index.html;
# The built SPA. Hashed asset filenames are immutable; index.html must not
# be cached or a deploy would leave browsers loading a bundle that no longer
# exists.
location / {
try_files $uri $uri/ /index.html;
}
location /assets/ {
expires 1y;
add_header Cache-Control "public, immutable";
}
location = /index.html {
add_header Cache-Control "no-cache";
}
location /v1/ {
proxy_pass http://blackbeard_api;
proxy_http_version 1.1;
# The WebSocket upgrade. Without these three lines the socket silently
# falls back to a plain request that hangs, and the page shows
# "reconnecting" forever with nothing in any log.
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# A chain can be quiet for a minute at a time and a browser tab can sit
# open all day. The 60s default would cut every socket mid-session and
# surface as an unexplained disconnect.
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
# Buffering an endless stream is exactly wrong: it would hold each
# message until a buffer filled, which on a quiet chain is minutes.
proxy_buffering off;
}
# Public, read-only, entirely derived from public block headers. No auth,
# nothing to leak; documented here so the absence is deliberate rather than
# an oversight.
add_header X-Content-Type-Options nosniff always;
add_header Referrer-Policy same-origin always;
}

39
asset/sql/bootstrap.sql Normal file
View File

@@ -0,0 +1,39 @@
-- Idempotent role and database creation for blackbeard.observer.
--
-- Run once, by an operator, on the Postgres PRIMARY only (magrathea) —
-- replication carries roles and databases to the standby. Applied by
-- script/infra-setup.sh --database.
--
-- No password is set on the role, deliberately and permanently. Authentication
-- is mTLS: the app host's certificate CN maps to this role through a
-- pg_ident.conf drop-in, which infra-setup.sh installs on BOTH servers. A
-- standby missing that mapping locks the app out on failover
-- (architecture/generic.md §5).
do $$
begin
if not exists (select from pg_roles where rolname = 'blackbeard_rw') then
create role blackbeard_rw with login;
end if;
-- A read-only role for ad-hoc queries and any future reporting, so nothing
-- has to borrow the writer's credentials to look at the data.
if not exists (select from pg_roles where rolname = 'blackbeard_ro') then
create role blackbeard_ro with login;
end if;
end
$$;
-- `create database` cannot run inside a transaction block or a DO block, so the
-- caller guards it: infra-setup.sh checks first and skips if present.
\connect blackbeard
-- The writer owns the schema so sqlx migrations can create tables.
alter schema public owner to blackbeard_rw;
grant usage on schema public to blackbeard_ro;
-- Applies to tables the migrations have not created yet, so a new migration
-- does not need this file re-run.
alter default privileges for role blackbeard_rw in schema public
grant select on tables to blackbeard_ro;
grant select on all tables in schema public to blackbeard_ro;

10
asset/systemd/blackbeard-api-cert-reload.service Normal file
View File

@@ -0,0 +1,10 @@
# Restart, not reload: the daemon holds no reloadable cert state of its own —
# sqlx reads the certificate files when it opens a connection. A restart is a
# few seconds of dropped WebSockets, which the browser reconnects from on its
# own, and it is unambiguous.
[Unit]
Description=Restart blackbeard-api after a host certificate rotation
[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl restart blackbeard-api.service

14
asset/systemd/blackbeard-api-cert.path Normal file
View File

@@ -0,0 +1,14 @@
# The host certificate is the credential for the mTLS Postgres connection and is
# reissued every 24 hours (architecture/generic.md §11). The daemon reads it
# when it opens a connection, so a rotation is picked up by the pool's own
# recycling within the hour — but a restart on change makes it immediate and
# removes the one window where a stale in-memory client cert could be presented.
[Unit]
Description=Watch the host certificate for blackbeard-api
[Path]
PathChanged=/etc/pki/tls/misc/%H.pem
Unit=blackbeard-api-cert-reload.service
[Install]
WantedBy=multi-user.target

60
asset/systemd/blackbeard-api.service Normal file
View File

@@ -0,0 +1,60 @@
# The observer daemon: watches the configured Quantus chains, keeps the
# standings, and serves REST + WebSocket to the site's nginx.
#
# Runs beside quantus-node on the same host and reads its loopback JSON-RPC, so
# it needs no chain data of its own, no credentials for the node, and no route
# to anything but Postgres and the telemetry feed.
[Unit]
Description=blackbeard.observer API (Quantus mining leaderboard)
Documentation=https://git.lair.cafe/Quantus-Network/blackbeard.observer
After=network-online.target
Wants=network-online.target
# Not Requires=: the node restarting must not take the site down. The daemon
# reconnects on its own and keeps serving what it already knows.
After=quantus-node.service
[Service]
# The binary sends sd_notify(READY=1) after it binds, so systemctl restart
# returns when the port is actually accepting rather than when the process
# started.
Type=notify
User=blackbeard
Group=blackbeard
ExecStart=/usr/local/bin/blackbeard-api --config /etc/blackbeard/config.toml
# SIGTERM drains in-flight requests and exits 0. Long-lived WebSockets are what
# this bound is for: without it, a deploy would wait for the last browser tab to
# close.
KillSignal=SIGTERM
TimeoutStopSec=20s
Restart=always
RestartSec=5s
# Hardening per architecture/generic.md §8. Nothing here is relaxed: the daemon
# reads two sockets and one config file and writes nothing to disk.
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true
SystemCallArchitectures=native
RestrictNamespaces=true
ProtectClock=true
ProtectHostname=true
ProtectProc=invisible
# ProtectSystem=strict makes everything read-only; the daemon keeps no state on
# disk, so it needs no ReadWritePaths at all. If one is ever added here, ask
# first what state left the database.
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
[Install]
WantedBy=multi-user.target

2
asset/systemd/blackbeard.sysusers.conf Normal file
View File

@@ -0,0 +1,2 @@
#Type Name ID GECOS Home directory Shell
u blackbeard - "blackbeard.observer service account" /var/lib/blackbeard /usr/sbin/nologin

33
crates/blackbeard-api/Cargo.toml Normal file
View File

@@ -0,0 +1,33 @@
[package]
name = "blackbeard-api"
description = "REST and WebSocket daemon for blackbeard.observer."
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
authors.workspace = true
repository.workspace = true
[[bin]]
name = "blackbeard-api"
path = "src/main.rs"
[dependencies]
blackbeard-core.workspace = true
blackbeard-data.workspace = true
blackbeard-entities.workspace = true
anyhow.workspace = true
axum.workspace = true
chrono.workspace = true
clap.workspace = true
figment.workspace = true
futures-util.workspace = true
primitive-types.workspace = true
serde.workspace = true
sqlx.workspace = true
serde_json.workspace = true
tokio.workspace = true
tower-http.workspace = true
tracing.workspace = true
tracing-subscriber.workspace = true

374
crates/blackbeard-api/src/config.rs Normal file
View File

@@ -0,0 +1,374 @@
//! Configuration: file, then environment, then CLI.
//!
//! Layered with figment in that order, so a deployed host reads
//! `/etc/blackbeard/config.toml` and an operator debugging on that host can
//! override one value with an environment variable without editing it.
//!
//! **No secret belongs in this file.** The database connection is mTLS with the
//! host's own certificate as the credential (`architecture/generic.md` §5), so
//! there is no password to configure — and there is deliberately nowhere to put
//! one. Everything here is infrastructure truth: hostnames, ports and paths.
use std::net::SocketAddr;
use std::path::PathBuf;
use blackbeard_entities::ChainId;
use serde::{Deserialize, Serialize};
/// The whole configuration.
#[derive(Debug, Clone, Deserialize, Serialize)]
#[serde(deny_unknown_fields)]
pub struct Config {
/// HTTP listener.
pub server: ServerConfig,
/// Postgres connection.
pub database: DatabaseConfig,
/// Every chain to watch. A chain may be listed before it launches.
#[serde(default)]
pub chains: Vec<ChainConfig>,
}
/// The HTTP listener and what it will accept.
#[derive(Debug, Clone, Deserialize, Serialize)]
#[serde(deny_unknown_fields)]
pub struct ServerConfig {
/// Address to bind.
///
/// Bind the host's mesh address or loopback, never `0.0.0.0`, on a host
/// that carries a public address: the fleet uses a single firewalld default
/// zone (`generic.md` §9), so a wildcard bind plus the named service would
/// publish this on every address the host has. TLS terminates at the site's
/// nginx (`reverse-proxies.md`), which is the only thing that should reach
/// this port.
pub listen: SocketAddr,
/// Origins allowed to call the API from a browser.
///
/// The data here is public — it is derived from public block headers — so
/// this is not a security boundary. It exists so that a copy of the
/// frontend served from somewhere else does not silently depend on this
/// backend, and so the origins the site actually uses are written down.
#[serde(default)]
pub allowed_origins: Vec<String>,
/// Blocks kept in the live ticker and replayed to a newly connected
/// browser.
#[serde(default = "default_ticker_blocks")]
pub ticker_blocks: usize,
/// Seconds between leaderboard recomputes.
///
/// Standings are recomputed on a timer rather than on every block: at a 6 s
/// target most blocks change nothing above the tenth row, and a week window
/// is a hundred thousand entries to re-tally. Only windows with at least
/// one subscriber are computed at all.
#[serde(default = "default_leaderboard_refresh")]
pub leaderboard_refresh_seconds: u64,
}
/// Postgres, over mTLS.
#[derive(Debug, Clone, Deserialize, Serialize)]
#[serde(deny_unknown_fields)]
pub struct DatabaseConfig {
/// Server hostname, as it appears in the server's certificate.
pub host: String,
/// Server port.
#[serde(default = "default_pg_port")]
pub port: u16,
/// Database name.
pub database: String,
/// Role to connect as; resolved from the client certificate CN by
/// `pg_ident.conf` on the server.
pub username: String,
/// Internal root CA bundle.
#[serde(default = "default_root_cert")]
pub root_cert: PathBuf,
/// This host's certificate — the credential.
pub client_cert: PathBuf,
/// This host's private key. The service account is granted read access via
/// `setfacl` at deploy time; it is not world-readable.
pub client_key: PathBuf,
/// Pool size.
#[serde(default = "default_max_connections")]
pub max_connections: u32,
}
/// One chain to watch.
#[derive(Debug, Clone, Deserialize, Serialize)]
#[serde(deny_unknown_fields)]
pub struct ChainConfig {
/// Operator name, used in routes and the WebSocket protocol.
pub id: ChainId,
/// Human-facing name.
pub display_name: String,
/// True for the production network.
#[serde(default)]
pub mainnet: bool,
/// HTTP JSON-RPC endpoint, e.g. `http://127.0.0.1:9944`.
pub rpc_url: String,
/// WebSocket JSON-RPC endpoint for the head subscription. Usually the same
/// host and port as `rpc_url` — `9944` serves both.
pub ws_url: String,
/// substrate-telemetry feed. Empty disables telemetry for this chain, which
/// costs node counts and miner names and nothing else.
#[serde(default)]
pub telemetry_url: String,
/// The chain's target seconds per block. Used as the hashrate denominator
/// whenever no measured interval is trustworthy.
#[serde(default = "default_target_block_time")]
pub target_block_time_seconds: f64,
/// Blocks to load from the database at startup, seeding the rolling window.
///
/// Defaults to the longest selectable window, so every window is populated
/// on the first request rather than filling in over the following week.
#[serde(default = "default_warm_start_blocks")]
pub warm_start_blocks: usize,
/// Cap on how many blocks a single gap-fill will fetch from the node.
///
/// A long outage must not have the observer hammer the node for hours
/// trying to catch up on history it can also simply not have. The gap is
/// closed from the newest end, so the tip is correct immediately and only
/// the oldest part of the gap is forgone.
#[serde(default = "default_max_gap_fill")]
pub max_gap_fill_blocks: u64,
}
fn default_pg_port() -> u16 {
5432
}
fn default_root_cert() -> PathBuf {
PathBuf::from("/etc/pki/ca-trust/source/anchors/root-internal.pem")
}
fn default_max_connections() -> u32 {
8
}
fn default_target_block_time() -> f64 {
6.0
}
fn default_ticker_blocks() -> usize {
40
}
fn default_leaderboard_refresh() -> u64 {
5
}
fn default_warm_start_blocks() -> usize {
blackbeard_entities::Window::Week.blocks() as usize
}
fn default_max_gap_fill() -> u64 {
5_000
}
impl Config {
/// Load from `path`, then overlay `BLACKBEARD_*` environment variables.
///
/// Nested keys use `__` as the separator, so
/// `BLACKBEARD_SERVER__LISTEN=127.0.0.1:25864` overrides `server.listen`.
pub fn load(path: &std::path::Path) -> anyhow::Result<Self> {
use figment::providers::{Env, Format, Toml};
let config: Config = figment::Figment::new()
.merge(Toml::file(path))
.merge(
Env::prefixed("BLACKBEARD_")
.split("__")
// Two variables share the prefix without being config keys:
// `BLACKBEARD_CONFIG` names this very file, and
// `BLACKBEARD_DEV_DATABASE_URL` is the development-only
// database escape hatch. Without this they are read as
// unknown keys and — because the structs deny unknown
// fields — refuse to start.
.ignore(&["CONFIG", "DEV_DATABASE_URL"]),
)
.extract()?;
config.validate()?;
Ok(config)
}
/// Reject a configuration that would start but never work.
///
/// Failing at startup with a clear message beats a daemon that comes up,
/// reports healthy, and serves an empty site — which is what a duplicate
/// chain id or a missing certificate would otherwise produce.
fn validate(&self) -> anyhow::Result<()> {
if self.chains.is_empty() {
anyhow::bail!("no chains configured — the observer would have nothing to observe");
}
let mut seen = std::collections::HashSet::new();
for chain in &self.chains {
if !seen.insert(&chain.id) {
anyhow::bail!(
"chain id `{}` is configured more than once; ids are the routing key and must be unique",
chain.id
);
}
if chain.target_block_time_seconds <= 0.0 {
anyhow::bail!(
"chain `{}` has a target block time of {}; hashrate is difficulty divided by it",
chain.id,
chain.target_block_time_seconds
);
}
}
for (label, path) in [
("database.root_cert", &self.database.root_cert),
("database.client_cert", &self.database.client_cert),
("database.client_key", &self.database.client_key),
] {
if !path.exists() {
anyhow::bail!(
"{label} points at {} which does not exist; the database connection is mTLS \
and this certificate is the credential",
path.display()
);
}
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
fn write(dir: &std::path::Path, name: &str, body: &str) -> PathBuf {
let p = dir.join(name);
std::fs::write(&p, body).unwrap();
p
}
/// A config whose certificate paths point at real (empty) files, so
/// validation exercises everything except the existence check.
fn config_toml(dir: &std::path::Path, chains: &str) -> String {
for f in ["root.pem", "cert.pem", "key.pem"] {
std::fs::write(dir.join(f), "").unwrap();
}
format!(
r#"
[server]
listen = "127.0.0.1:25864"
[database]
host = "magrathea.kosherinata.internal"
database = "blackbeard"
username = "blackbeard_rw"
root_cert = "{d}/root.pem"
client_cert = "{d}/cert.pem"
client_key = "{d}/key.pem"
{chains}
"#,
d = dir.display()
)
}
const PLANCK: &str = r#"
[[chains]]
id = "planck"
display_name = "Planck Testnet"
rpc_url = "http://127.0.0.1:9944"
ws_url = "ws://127.0.0.1:9944"
"#;
#[test]
fn a_minimal_config_fills_in_the_documented_defaults() {
let dir = tempdir();
let p = write(&dir, "config.toml", &config_toml(&dir, PLANCK));
let c = Config::load(&p).unwrap();
assert_eq!(c.database.port, 5432);
assert_eq!(c.chains[0].target_block_time_seconds, 6.0);
assert_eq!(c.chains[0].warm_start_blocks, 100_800);
assert!(!c.chains[0].mainnet);
assert_eq!(c.server.ticker_blocks, 40);
}
#[test]
fn a_duplicate_chain_id_is_refused_at_startup() {
// Both would answer at /v1/chains/planck; whichever won would be an
// accident of ordering, and the site would look fine while showing one
// chain's numbers under the other's name.
let dir = tempdir();
let body = config_toml(&dir, &format!("{PLANCK}{PLANCK}"));
let p = write(&dir, "config.toml", &body);
let err = Config::load(&p).unwrap_err().to_string();
assert!(err.contains("more than once"), "{err}");
}
#[test]
fn a_config_with_no_chains_is_refused() {
let dir = tempdir();
let p = write(&dir, "config.toml", &config_toml(&dir, ""));
assert!(
Config::load(&p)
.unwrap_err()
.to_string()
.contains("no chains")
);
}
#[test]
fn a_missing_certificate_is_refused_rather_than_discovered_later() {
let dir = tempdir();
let body = config_toml(&dir, PLANCK).replace("/cert.pem", "/absent.pem");
let p = write(&dir, "config.toml", &body);
let err = Config::load(&p).unwrap_err().to_string();
assert!(err.contains("client_cert"), "{err}");
}
#[test]
fn a_zero_target_block_time_is_refused() {
// It is the hashrate denominator; zero would make every figure on the
// site either zero or infinite.
let dir = tempdir();
let body = config_toml(&dir, &format!("{PLANCK}target_block_time_seconds = 0.0\n"));
let p = write(&dir, "config.toml", &body);
assert!(
Config::load(&p)
.unwrap_err()
.to_string()
.contains("target block time")
);
}
#[test]
fn the_daemons_own_environment_variables_are_not_read_as_config_keys() {
// BLACKBEARD_CONFIG names the config file and
// BLACKBEARD_DEV_DATABASE_URL is the dev database hatch. Both share the
// prefix; neither is a config key, and with `deny_unknown_fields` either
// one would otherwise stop the daemon from starting at all.
let dir = tempdir();
let p = write(&dir, "config.toml", &config_toml(&dir, PLANCK));
// SAFETY: single-threaded test process section; no other thread reads
// the environment concurrently.
unsafe {
std::env::set_var("BLACKBEARD_CONFIG", p.to_str().unwrap());
std::env::set_var("BLACKBEARD_DEV_DATABASE_URL", "postgres://x/y");
}
let loaded = Config::load(&p);
unsafe {
std::env::remove_var("BLACKBEARD_CONFIG");
std::env::remove_var("BLACKBEARD_DEV_DATABASE_URL");
}
assert!(loaded.is_ok(), "{:?}", loaded.err());
}
#[test]
fn an_unknown_key_is_refused_rather_than_ignored() {
// A typo in a deployed config must fail loudly. Silently ignoring
// `listen_address` would leave the daemon bound somewhere unexpected
// with nothing in the journal to say why.
let dir = tempdir();
let body = config_toml(&dir, PLANCK).replace("[server]", "[server]\nlisten_addr = \"x\"");
let p = write(&dir, "config.toml", &body);
assert!(Config::load(&p).is_err());
}
/// A unique scratch directory. Avoids a `tempfile` dependency for the one
/// thing these tests need.
fn tempdir() -> PathBuf {
let base = std::env::temp_dir().join(format!(
"blackbeard-config-test-{}-{:?}",
std::process::id(),
std::thread::current().id()
));
std::fs::create_dir_all(&base).unwrap();
base
}
}

522
crates/blackbeard-api/src/ingest.rs Normal file
View File

@@ -0,0 +1,522 @@
//! Watching a chain.
//!
//! Three tasks per chain, deliberately separated because they fail
//! independently and at different rates:
//!
//! 1. **[`ingest`]** — consumes the head stream, fills gaps, decodes authors,
//! writes blocks, pushes the ticker.
//! 2. **[`poll`]** — difficulty, sync state and chain properties on a timer.
//! Difficulty changes once per block but is cheap to over-read and expensive
//! to miss; sync state is what decides whether the hashrate figure is real.
//! 3. **[`housekeeping`]** — resolves telemetry attributions once the feed has
//! had time to report, recomputes watched leaderboards, and persists held
//! names.
//!
//! A node that is down affects all three; a telemetry feed that is down affects
//! only the third, and only its naming.
use std::sync::Arc;
use std::time::Duration;
use blackbeard_core::digest;
use blackbeard_core::window::Observed;
use blackbeard_data::rpc::{self, Header};
use blackbeard_data::store::{BlockRecord, ChainRecord, Store};
use blackbeard_data::telemetry::TelemetryFeed;
use blackbeard_entities::{ChainStatus, RecentBlock, ServerMessage};
use chrono::{DateTime, TimeZone, Utc};
use tokio::sync::mpsc;
use crate::state::{ChainRuntime, PendingAttribution};
/// Head notifications buffered before the subscription blocks.
///
/// Small on purpose: falling behind the tip by more than this means the ingest
/// loop is not keeping up, and the gap-fill path recovers correctly from a
/// dropped head while an ever-growing queue would only hide the problem.
const HEAD_BUFFER: usize = 64;
/// How long to wait for the telemetry feed to report a block before deciding
/// who reported it first.
///
/// Reports arrive from nodes all over the network over a couple of seconds;
/// judging at the moment the block is seen would count only the fastest-peered
/// nodes and systematically favour them.
const ATTRIBUTION_SETTLE: Duration = Duration::from_secs(8);
/// Interval between difficulty and sync-state polls.
const POLL_INTERVAL: Duration = Duration::from_secs(4);
/// Interval between attribution persists.
const PERSIST_INTERVAL: Duration = Duration::from_secs(30);
/// Start every task for one chain.
pub fn spawn(chain: Arc<ChainRuntime>, store: Store, ticker_blocks: usize, refresh: Duration) {
let (tx, rx) = mpsc::channel(HEAD_BUFFER);
tokio::spawn(rpc::subscribe_new_heads(chain.config.ws_url.clone(), tx));
tokio::spawn(ingest(Arc::clone(&chain), store.clone(), rx, ticker_blocks));
tokio::spawn(poll(Arc::clone(&chain), store.clone()));
tokio::spawn(housekeeping(chain, store, refresh));
}
/// Load what the database already knows, so a restart does not serve an empty
/// site while it re-watches a week of blocks.
pub async fn warm_start(chain: &Arc<ChainRuntime>, store: &Store) {
let id = chain.id().clone();
match store
.recent_blocks(&id, chain.config.warm_start_blocks as i64)
.await
{
Ok(blocks) => {
let count = blocks.len();
let mut inner = chain.write();
for block in blocks {
inner.height = Some(inner.height.unwrap_or(0).max(block.height));
// `at_tip: false` — these were observed by a previous process,
// so replaying them now must not contribute timing samples. The
// elapsed time between two database rows is not a block
// interval.
inner.window.push(block, false);
}
tracing::info!(chain = %id, blocks = count, "warm start: window restored");
}
Err(e) => tracing::warn!(chain = %id, error = %e, "warm start: no window restored"),
}
match store.load_attributions(&id).await {
Ok(held) => {
let count = held.len();
let mut inner = chain.write();
for a in held {
if let (blackbeard_core::attribution::NodeKey::Peer(peer), Some(name)) =
(&a.key, &a.name)
{
chain.telemetry.seed_name(peer.clone(), name.clone());
}
inner
.attributor
.restore_held(a.miner, a.key, a.attempts, a.attributed);
}
tracing::info!(chain = %id, names = count, "warm start: attributions restored");
}
Err(e) => tracing::warn!(chain = %id, error = %e, "warm start: no attributions restored"),
}
}
/// Consume the head stream.
async fn ingest(
chain: Arc<ChainRuntime>,
store: Store,
mut heads: mpsc::Receiver<Header>,
ticker_blocks: usize,
) {
while let Some(header) = heads.recv().await {
let Some(height) = header.height() else {
tracing::warn!(chain = %chain.id(), number = %header.number, "header carried an undecodable block number");
continue;
};
// `chain_subscribeNewHeads` reports the best head and skips
// intermediates when several import at once, so the stream is a
// liveness signal rather than a complete record. Anything missed is
// fetched here — otherwise the leaderboard would quietly under-count
// exactly the miners who won blocks during a burst.
let last = chain.read().height;
if let Some(last) = last
&& height > last + 1
{
let gap_start = (last + 1).max(height.saturating_sub(chain.config.max_gap_fill_blocks));
fill_gap(&chain, &store, gap_start, height).await;
}
record(&chain, &store, &header, height, ticker_blocks).await;
}
tracing::warn!(chain = %chain.id(), "head stream ended; ingest stopping");
}
/// Fetch and record the blocks between two heads.
///
/// Best effort: a node that has pruned, or that fails midway, costs the window
/// those blocks and nothing more. Ingest must not stall on history.
async fn fill_gap(chain: &Arc<ChainRuntime>, store: &Store, from: u64, to: u64) {
tracing::info!(chain = %chain.id(), from, to, "filling a gap in the head stream");
let mut batch = Vec::new();
let mut observed = Vec::new();
for height in from..to {
let Ok(Some(hash)) = chain.rpc.block_hash(height).await else {
continue;
};
let Ok(Some(header)) = chain.rpc.header(Some(&hash)).await else {
continue;
};
let Some(miner) = digest::author_preimage(&header.digest.logs) else {
continue;
};
let authored_at = chain
.rpc
.block_timestamp_ms(&hash)
.await
.ok()
.flatten()
.and_then(from_millis);
let now = Utc::now();
batch.push(BlockRecord {
chain: chain.id().clone(),
height,
hash,
miner: miner.clone(),
authored_at,
observed_at: now,
difficulty: chain
.read()
.difficulty
.map(|d| blackbeard_core::hashrate::u512_to_dec(d).0),
});
observed.push(Observed {
height,
miner,
observed_at: now,
});
}
if let Err(e) = store.record_blocks(&batch).await {
tracing::warn!(chain = %chain.id(), error = %e, "gap blocks were not persisted");
}
let mut inner = chain.write();
for o in observed {
// Not at the tip: these are being caught up on, and their observation
// times are all "now" rather than when they were authored.
inner.window.push(o, false);
}
}
/// Record one head.
async fn record(
chain: &Arc<ChainRuntime>,
store: &Store,
header: &Header,
height: u64,
ticker_blocks: usize,
) {
let Some(miner) = digest::author_preimage(&header.digest.logs) else {
// Genesis, or a header shape we do not decode. Not an error: the block
// simply contributes nothing to the standings.
tracing::debug!(chain = %chain.id(), height, "header carried no pow_ author digest");
return;
};
let hash = match chain.rpc.block_hash(height).await {
Ok(Some(h)) => h,
Ok(None) => return,
Err(e) => {
tracing::warn!(chain = %chain.id(), height, error = %e, "could not resolve the block hash");
return;
}
};
let authored_at = chain
.rpc
.block_timestamp_ms(&hash)
.await
.ok()
.flatten()
.and_then(from_millis);
let observed_at = Utc::now();
let difficulty = chain
.read()
.difficulty
.map(|d| blackbeard_core::hashrate::u512_to_dec(d).0);
if let Err(e) = store
.record_blocks(&[BlockRecord {
chain: chain.id().clone(),
height,
hash: hash.clone(),
miner: miner.clone(),
authored_at,
observed_at,
difficulty,
}])
.await
{
// The window is in memory and stays correct; only history is lost. A
// database blip must not stop the site from being live.
tracing::warn!(chain = %chain.id(), height, error = %e, "block not persisted");
}
let block = {
let mut inner = chain.write();
let gap_seconds = inner
.last_block_at
.map(|prev| (observed_at - prev).as_seconds_f64());
inner.window.push(
Observed {
height,
miner: miner.clone(),
observed_at,
},
true,
);
inner.height = Some(height);
inner.last_block_at = Some(observed_at);
let display = inner
.attributor
.attribute(&miner, |key| chain.telemetry.name_of(key))
.display;
let block = RecentBlock {
height,
miner: miner.clone(),
display,
observed_at,
gap_seconds,
};
inner.ticker.push_back(block.clone());
while inner.ticker.len() > ticker_blocks {
inner.ticker.pop_front();
}
block
};
chain.broadcast(
&ServerMessage::Block {
chain: chain.id().clone(),
block,
},
None,
);
// Queue the attribution join. The feed needs a few seconds to collect
// reports from around the network before "who was first" means anything.
if chain.telemetry.connected() {
let mut pending = chain
.pending_attributions
.lock()
.unwrap_or_else(|e| e.into_inner());
pending.push_back(PendingAttribution {
hash,
miner,
due: tokio::time::Instant::now() + ATTRIBUTION_SETTLE,
});
}
}
/// Poll the things that are not pushed: difficulty, sync state, properties.
async fn poll(chain: Arc<ChainRuntime>, store: Store) {
let mut ticker = tokio::time::interval(POLL_INTERVAL);
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
let mut telemetry_started = false;
loop {
ticker.tick().await;
let before = chain.read().status;
let health = match chain.rpc.health().await {
Ok(h) => h,
Err(e) => {
let status = if before == ChainStatus::Awaiting {
// Never answered. Expected for a chain configured before it
// launches, so it stays "awaiting" rather than becoming an
// alarm nobody can act on.
ChainStatus::Awaiting
} else {
ChainStatus::Unreachable
};
if status != before {
tracing::warn!(chain = %chain.id(), error = %e, "chain became unreachable");
chain.write().status = status;
announce_status(&chain);
}
continue;
}
};
{
let mut inner = chain.write();
inner.syncing = health.is_syncing;
inner.status = if health.is_syncing {
ChainStatus::Syncing
} else {
ChainStatus::Live
};
}
// Discovered once, then never again: genesis and token properties do
// not change for the life of a chain.
if chain.read().genesis.is_none()
&& let Ok(Some(genesis)) = chain.rpc.genesis().await
{
chain.write().genesis = Some(genesis.clone());
if !chain.config.telemetry_url.is_empty() && !telemetry_started {
telemetry_started = true;
tokio::spawn(TelemetryFeed::run(
chain.telemetry.clone(),
chain.config.telemetry_url.clone(),
genesis,
));
}
}
if chain.read().token_symbol.is_none()
&& let Ok(props) = chain.rpc.properties().await
{
let mut inner = chain.write();
inner.token_symbol = props.token_symbol;
inner.token_decimals = props.token_decimals;
}
if chain.read().max_difficulty.is_none()
&& let Ok(max) = chain.rpc.max_difficulty().await
{
chain.write().max_difficulty = Some(max);
}
match chain.rpc.difficulty().await {
Ok(d) => chain.write().difficulty = Some(d),
Err(e) => tracing::debug!(chain = %chain.id(), error = %e, "difficulty unavailable"),
}
if chain.read().status != before {
announce_status(&chain);
}
let record = {
let inner = chain.read();
ChainRecord {
chain: chain.id().clone(),
display_name: chain.config.display_name.clone(),
mainnet: chain.config.mainnet,
genesis: inner.genesis.clone(),
token_symbol: inner.token_symbol.clone(),
token_decimals: inner.token_decimals.map(i16::from),
target_block_time: chain.config.target_block_time_seconds,
}
};
if let Err(e) = store.upsert_chain(&record).await {
tracing::warn!(chain = %chain.id(), error = %e, "chain row not updated");
}
chain.broadcast(
&ServerMessage::Summary {
chain: chain.id().clone(),
summary: chain.summary(),
},
None,
);
}
}
fn announce_status(chain: &Arc<ChainRuntime>) {
chain.broadcast(
&ServerMessage::ChainStatus {
chain: chain.id().clone(),
info: chain.info(),
},
None,
);
}
/// Resolve attributions, recompute watched standings, persist held names.
async fn housekeeping(chain: Arc<ChainRuntime>, store: Store, refresh: Duration) {
let mut attribution = tokio::time::interval(Duration::from_secs(1));
let mut leaderboard = tokio::time::interval(refresh);
let mut persist = tokio::time::interval(PERSIST_INTERVAL);
for t in [&mut attribution, &mut leaderboard, &mut persist] {
t.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
}
loop {
tokio::select! {
_ = attribution.tick() => resolve_attributions(&chain),
_ = leaderboard.tick() => {
for window in chain.watched_windows() {
let (rows, changed) = chain.recompute_leaderboard(window);
if changed {
chain.broadcast(
&ServerMessage::Leaderboard {
chain: chain.id().clone(),
window,
rows,
},
Some(window),
);
}
}
}
_ = persist.tick() => {
let held: Vec<_> = {
let inner = chain.read();
inner
.attributor
.held_names()
.into_iter()
.map(|(miner, key, attempts, attributed)| {
let name = chain.telemetry.name_of(&key);
(miner, key, attempts, attributed, name)
})
.collect()
};
if let Err(e) = store.save_attributions(chain.id(), &held).await {
tracing::warn!(chain = %chain.id(), error = %e, "attributions not persisted");
}
}
}
}
}
/// Join settled blocks with the feed's first-import reports.
fn resolve_attributions(chain: &Arc<ChainRuntime>) {
let now = tokio::time::Instant::now();
let due: Vec<PendingAttribution> = {
let mut pending = chain
.pending_attributions
.lock()
.unwrap_or_else(|e| e.into_inner());
let mut due = Vec::new();
while pending.front().is_some_and(|p| p.due <= now) {
due.extend(pending.pop_front());
}
due
};
if due.is_empty() {
return;
}
let mut inner = chain.write();
for p in due {
inner
.attributor
.observe(&p.miner, chain.telemetry.first_import(&p.hash));
}
}
/// Milliseconds since the epoch as a UTC timestamp.
///
/// `None` for a value the calendar cannot represent — a block whose timestamp
/// inherent decoded to nonsense loses its authored time and nothing else.
fn from_millis(ms: u64) -> Option<DateTime<Utc>> {
// `as i64` would wrap rather than refuse: a misdecoded compact integer of
// u64::MAX becomes -1, and the block is silently dated to 1969 instead of
// being declined.
Utc.timestamp_millis_opt(i64::try_from(ms).ok()?).single()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_plausible_timestamp_converts() {
let t = from_millis(1_757_000_000_000).unwrap();
assert_eq!(t.timestamp(), 1_757_000_000);
}
#[test]
fn an_impossible_timestamp_is_declined_rather_than_panicking() {
// A misdecoded compact integer can produce an enormous value; it must
// cost the block its authored time, not silently become a 1969 date.
assert_eq!(from_millis(u64::MAX), None);
assert_eq!(from_millis(i64::MAX as u64 + 1), None);
// The epoch itself is representable and must not be swept up.
assert_eq!(from_millis(0).map(|t| t.timestamp()), Some(0));
}
}

237
crates/blackbeard-api/src/main.rs Normal file
View File

@@ -0,0 +1,237 @@
//! `blackbeard-api` — the observer daemon behind <https://blackbeard.observer>.
//!
//! Watches one or more Quantus chains, decodes the author of every block from
//! its header, keeps the standings, and pushes changes to connected browsers
//! over a WebSocket.
//!
//! Runs under systemd as a dedicated non-root user (`architecture/generic.md`
//! §8). It does not daemonise, logs to stdout for journald, and exits 0 on
//! `SIGTERM` after draining in-flight requests.
mod config;
mod ingest;
mod routes;
mod state;
mod ws;
use std::path::PathBuf;
use std::sync::Arc;
use std::time::Duration;
use anyhow::Context;
use blackbeard_data::rpc::RpcClient;
use blackbeard_data::store::{Store, StoreConfig};
use clap::Parser;
use crate::state::{AppState, ChainRuntime};
/// How long an RPC call may take before it is abandoned.
///
/// The node is on the same host over loopback; anything slower than this is a
/// node in trouble, and waiting longer only makes the ingest loop fall further
/// behind the tip.
const RPC_TIMEOUT: Duration = Duration::from_secs(10);
#[derive(Debug, Parser)]
#[command(name = "blackbeard-api", version, about)]
struct Args {
/// Configuration file.
#[arg(
long,
default_value = "/etc/blackbeard/config.toml",
env = "BLACKBEARD_CONFIG"
)]
config: PathBuf,
/// Check the configuration and exit without binding anything.
///
/// What the deploy runs before restarting the service, so a bad config
/// fails the deploy rather than the daemon.
#[arg(long)]
check: bool,
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let args = Args::parse();
init_tracing();
let config = config::Config::load(&args.config)
.with_context(|| format!("reading {}", args.config.display()))?;
if args.check {
println!(
"{} is valid: {} chain(s), listening on {}",
args.config.display(),
config.chains.len(),
config.server.listen
);
return Ok(());
}
tracing::info!(
version = env!("CARGO_PKG_VERSION"),
chains = config.chains.len(),
"blackbeard-api starting"
);
let store = connect_store(&config).await?;
tracing::info!(host = %config.database.host, "postgres connected, migrations applied");
let mut chains = Vec::new();
let mut by_id = std::collections::HashMap::new();
for chain_config in &config.chains {
let rpc = RpcClient::new(chain_config.rpc_url.clone(), RPC_TIMEOUT)?;
let runtime = Arc::new(ChainRuntime::new(chain_config.clone(), rpc));
// Before any task starts, so the first browser to connect sees a
// populated site rather than one filling in over the next week.
ingest::warm_start(&runtime, &store).await;
ingest::spawn(
Arc::clone(&runtime),
store.clone(),
config.server.ticker_blocks,
Duration::from_secs(config.server.leaderboard_refresh_seconds),
);
by_id.insert(runtime.id().clone(), Arc::clone(&runtime));
chains.push(runtime);
}
let state = AppState {
chains: Arc::new(chains),
by_id: Arc::new(by_id),
store,
started: chrono::Utc::now(),
leaderboard_max_age: Duration::from_secs(config.server.leaderboard_refresh_seconds),
};
let app = routes::router(state, &config.server.allowed_origins);
let listener = tokio::net::TcpListener::bind(config.server.listen)
.await
.with_context(|| {
format!(
"binding {} (SELinux needs this port labelled: semanage port -a -t http_port_t -p tcp {})",
config.server.listen,
config.server.listen.port()
)
})?;
tracing::info!(listen = %config.server.listen, "serving");
// systemd holds `systemctl restart` open until this arrives. A Type=notify
// unit whose daemon never sends it blocks until TimeoutStartSec and then
// reports a failure for a service that is running perfectly.
notify_systemd_ready();
axum::serve(listener, app)
.with_graceful_shutdown(shutdown_signal())
.await
.context("serving")?;
tracing::info!("drained, exiting");
Ok(())
}
/// Connect to Postgres.
///
/// Production is mTLS with the host's own certificate and no password, per
/// `architecture/generic.md` §5 — there is deliberately nowhere in the config
/// file to put a password.
///
/// `BLACKBEARD_DEV_DATABASE_URL` is the development escape hatch: a plain
/// connection string for a throwaway local Postgres, so `cargo run` works on a
/// workstation that has no `pg_ident` mapping on the fleet's cluster. It is an
/// environment variable rather than a config key on purpose — the systemd unit
/// never sets it, so it cannot be reached by editing a deployed config, and
/// taking it logs loudly enough that a production host using it is obvious in
/// the journal.
async fn connect_store(config: &config::Config) -> anyhow::Result<Store> {
if let Ok(url) = std::env::var("BLACKBEARD_DEV_DATABASE_URL") {
tracing::warn!(
"BLACKBEARD_DEV_DATABASE_URL is set: connecting WITHOUT mTLS. \
This is a development-only path and must never be used in production."
);
let pool = sqlx::postgres::PgPoolOptions::new()
.max_connections(config.database.max_connections)
.connect(&url)
.await
.context("connecting to the development database")?;
let store = Store::from_pool(pool);
store.migrate().await.context("applying migrations")?;
return Ok(store);
}
let store = Store::connect(&StoreConfig {
host: config.database.host.clone(),
port: config.database.port,
database: config.database.database.clone(),
username: config.database.username.clone(),
root_cert: config.database.root_cert.clone(),
client_cert: config.database.client_cert.clone(),
client_key: config.database.client_key.clone(),
max_connections: config.database.max_connections,
})
.await
.context("connecting to postgres (mTLS: check the host certificate and pg_ident mapping)")?;
Ok(store)
}
/// JSON to journald, human-readable to a terminal.
///
/// `JOURNAL_STREAM` is set by systemd for any unit whose stdout it captures, so
/// it is the honest test for "is a log aggregator reading this" — more so than
/// a TTY check, which is also false when output is piped to a file.
fn init_tracing() {
use tracing_subscriber::{EnvFilter, fmt};
let filter = EnvFilter::try_from_default_env()
.unwrap_or_else(|_| EnvFilter::new("info,blackbeard_api=info,blackbeard_data=info"));
if std::env::var_os("JOURNAL_STREAM").is_some() {
fmt().json().with_env_filter(filter).init();
} else {
fmt().with_env_filter(filter).init();
}
}
/// Tell systemd the service is up, if it asked to be told.
///
/// Implemented directly rather than through a crate: the protocol is one
/// datagram to the socket named in `NOTIFY_SOCKET`, and a dependency for that
/// is a dependency to keep patched for the life of the project.
///
/// A leading `@` means the Linux abstract namespace, whose addresses start with
/// a NUL byte — the one detail that makes a hand-rolled implementation fail
/// silently if missed.
fn notify_systemd_ready() {
let Some(path) = std::env::var_os("NOTIFY_SOCKET") else {
return;
};
let path = std::path::PathBuf::from(path);
let path = match path.to_str() {
Some(p) if p.starts_with('@') => PathBuf::from(format!("\0{}", &p[1..])),
_ => path,
};
match std::os::unix::net::UnixDatagram::unbound()
.and_then(|socket| socket.send_to(b"READY=1", &path))
{
Ok(_) => tracing::debug!("notified systemd"),
// Not fatal: a daemon run by hand has no notify socket, and one that
// cannot reach it is still serving.
Err(e) => tracing::warn!(error = %e, "could not notify systemd"),
}
}
/// Resolve on `SIGTERM` (systemd) or `SIGINT` (a terminal).
async fn shutdown_signal() {
let mut term = match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) {
Ok(s) => s,
Err(e) => {
tracing::error!(error = %e, "cannot listen for SIGTERM; shutdown will not be graceful");
std::future::pending::<()>().await;
return;
}
};
tokio::select! {
_ = term.recv() => tracing::info!("SIGTERM received, draining"),
_ = tokio::signal::ctrl_c() => tracing::info!("interrupted, draining"),
}
}

349
crates/blackbeard-api/src/routes.rs Normal file
View File

@@ -0,0 +1,349 @@
//! The REST surface.
//!
//! The WebSocket is the live path; these endpoints exist for the first paint,
//! for anything that is a query rather than a stream (one miner's history), and
//! so the whole thing is reachable with `curl`. Everything here is derived from
//! public block headers, so nothing is authenticated and nothing is secret.
//!
//! Versioned under `/v1/` from the first commit — retrofitting a version prefix
//! after clients exist is the expensive kind of change.
use std::time::Duration;
use axum::extract::{Path, Query, State};
use axum::http::{HeaderValue, Method, StatusCode};
use axum::response::{IntoResponse, Response};
use axum::routing::get;
use axum::{Json, Router};
use blackbeard_entities::{
ApiError, BigUintDec, ChainInfo, ChainSummary, LeaderboardRow, MinerDetail, MinerId,
MinerSeriesPoint, RecentBlock, Window,
};
use serde::{Deserialize, Serialize};
use tower_http::compression::CompressionLayer;
use tower_http::cors::CorsLayer;
use crate::state::AppState;
/// Build the router.
pub fn router(state: AppState, allowed_origins: &[String]) -> Router {
let cors = allowed_origins
.iter()
.filter_map(|o| HeaderValue::from_str(o).ok())
.fold(
CorsLayer::new().allow_methods([Method::GET]),
|layer, origin| layer.allow_origin(origin),
);
Router::new()
.route("/v1/healthz", get(healthz))
.route("/v1/chains", get(chains))
.route("/v1/chains/{chain}/summary", get(summary))
.route("/v1/chains/{chain}/leaderboard", get(leaderboard))
.route("/v1/chains/{chain}/blocks", get(blocks))
.route("/v1/chains/{chain}/miners/{miner}", get(miner))
.route("/v1/ws", get(crate::ws::handler))
// Leaderboards are repetitive JSON and compress to a fraction of their
// size; the snapshot a browser fetches on first paint is the largest
// single response this API serves.
.layer(CompressionLayer::new())
.layer(cors)
.with_state(state)
}
/// An error with the status code to render it under.
struct Failure(StatusCode, ApiError);
impl IntoResponse for Failure {
fn into_response(self) -> Response {
(self.0, Json(self.1)).into_response()
}
}
fn unknown_chain(id: &str) -> Failure {
Failure(
StatusCode::NOT_FOUND,
ApiError::new(
"unknown_chain",
format!("no chain named `{id}` is configured"),
),
)
}
/// `GET /v1/healthz`
#[derive(Serialize)]
struct HealthResponse {
/// True when the database answers. Chain reachability is deliberately *not*
/// part of this: a chain that has not launched, or a node that is
/// restarting, must not make systemd or a load balancer restart a daemon
/// that is working exactly as intended. Per-chain state is reported
/// alongside, for a human to read.
healthy: bool,
version: &'static str,
uptime_seconds: i64,
database: bool,
chains: Vec<ChainInfo>,
}
async fn healthz(State(state): State<AppState>) -> Response {
let database = state.store.ping().await.is_ok();
let body = HealthResponse {
healthy: database,
version: env!("CARGO_PKG_VERSION"),
uptime_seconds: (chrono::Utc::now() - state.started).num_seconds(),
database,
chains: state.chain_infos(),
};
let code = if database {
StatusCode::OK
} else {
StatusCode::SERVICE_UNAVAILABLE
};
(code, Json(body)).into_response()
}
/// `GET /v1/chains`
async fn chains(State(state): State<AppState>) -> Json<Vec<ChainInfo>> {
Json(state.chain_infos())
}
/// `GET /v1/chains/{chain}/summary`
async fn summary(
State(state): State<AppState>,
Path(chain): Path<String>,
) -> Result<Json<ChainSummary>, Failure> {
let runtime = state.chain(&chain).ok_or_else(|| unknown_chain(&chain))?;
Ok(Json(runtime.summary()))
}
/// Query string for the leaderboard and miner endpoints.
#[derive(Debug, Deserialize)]
struct WindowQuery {
#[serde(default)]
window: Option<String>,
#[serde(default)]
limit: Option<usize>,
}
impl WindowQuery {
fn window(&self) -> Result<Window, Failure> {
match &self.window {
None => Ok(Window::default()),
Some(w) => w.parse().map_err(|e: blackbeard_entities::EntityError| {
Failure(
StatusCode::BAD_REQUEST,
ApiError::new("unknown_window", e.to_string()),
)
}),
}
}
}
/// `GET /v1/chains/{chain}/leaderboard?window=six_hours&limit=100`
#[derive(Serialize)]
struct LeaderboardResponse {
window: Window,
/// The window's length in blocks. A window is a block count, not a
/// duration — see `Window` — and the client needs the number to say so.
window_blocks: u32,
/// Rows actually held in the window, which is fewer than `window_blocks`
/// until the observer has watched that many.
observed_blocks: u32,
rows: Vec<LeaderboardRow>,
}
async fn leaderboard(
State(state): State<AppState>,
Path(chain): Path<String>,
Query(query): Query<WindowQuery>,
) -> Result<Json<LeaderboardResponse>, Failure> {
let runtime = state.chain(&chain).ok_or_else(|| unknown_chain(&chain))?;
let window = query.window()?;
let mut rows = runtime.leaderboard(window, state.leaderboard_max_age);
let observed_blocks = rows.iter().map(|r| r.blocks).sum();
if let Some(limit) = query.limit {
rows.truncate(limit);
}
Ok(Json(LeaderboardResponse {
window,
window_blocks: window.blocks(),
observed_blocks,
rows,
}))
}
/// `GET /v1/chains/{chain}/blocks`
async fn blocks(
State(state): State<AppState>,
Path(chain): Path<String>,
) -> Result<Json<Vec<RecentBlock>>, Failure> {
let runtime = state.chain(&chain).ok_or_else(|| unknown_chain(&chain))?;
Ok(Json(runtime.ticker()))
}
/// How far back a miner's history chart reaches, and how finely it is bucketed.
///
/// Buckets are chosen so every range renders roughly the same number of points:
/// a chart with ten thousand points is slower to draw and no more informative
/// than one with a hundred and fifty.
fn bucketing(window: Window) -> (Duration, Duration) {
match window {
Window::Hour => (Duration::from_secs(3_600), Duration::from_secs(60)),
Window::SixHours => (Duration::from_secs(21_600), Duration::from_secs(300)),
Window::Day => (Duration::from_secs(86_400), Duration::from_secs(900)),
Window::Week => (Duration::from_secs(604_800), Duration::from_secs(7_200)),
}
}
/// `GET /v1/chains/{chain}/miners/{miner}?window=day`
///
/// The endpoint the site exists for: one miner, its standing, and how it has
/// been doing over time.
async fn miner(
State(state): State<AppState>,
Path((chain, miner)): Path<(String, String)>,
Query(query): Query<WindowQuery>,
) -> Result<Json<MinerDetail>, Failure> {
let runtime = state.chain(&chain).ok_or_else(|| unknown_chain(&chain))?;
let window = query.window()?;
let miner = parse_miner(&miner)?;
let current = runtime
.leaderboard(window, state.leaderboard_max_age)
.into_iter()
.find(|row| row.miner == miner);
let (blocks_observed, first_seen, last_seen, work) = state
.store
.miner_totals(runtime.id(), &miner)
.await
.map_err(database_unavailable)?;
let (span, bucket) = bucketing(window);
let since = chrono::Utc::now()
- chrono::Duration::from_std(span).unwrap_or_else(|_| chrono::Duration::days(7));
let network_hashrate = runtime.summary().network_hashrate;
let series = state
.store
.miner_series(runtime.id(), &miner, since, bucket)
.await
.map_err(database_unavailable)?
.into_iter()
.map(|(at, mine, total)| MinerSeriesPoint {
at,
blocks: mine,
share: if total == 0 {
0.0
} else {
f64::from(mine) / f64::from(total)
},
hashrate_estimate: network_hashrate
.and_then(|n| blackbeard_core::hashrate::miner_hashrate(mine, total, n)),
})
.collect();
Ok(Json(MinerDetail {
chain: runtime.id().clone(),
miner,
current,
blocks_observed,
first_seen,
last_seen,
series,
cumulative_work: BigUintDec(work),
}))
}
/// A reward preimage is 32 bytes of hex.
///
/// Validated here rather than passed through, so a typo answers "that is not a
/// preimage" instead of "no blocks found" — which reads identically to a real
/// miner who has not won anything yet.
fn parse_miner(raw: &str) -> Result<MinerId, Failure> {
let hex = raw.strip_prefix("0x").unwrap_or(raw);
if hex.len() != 64 || !hex.chars().all(|c| c.is_ascii_hexdigit()) {
return Err(Failure(
StatusCode::BAD_REQUEST,
ApiError::new(
"malformed_miner",
format!("`{raw}` is not a 32-byte reward preimage"),
),
));
}
Ok(MinerId(format!("0x{}", hex.to_ascii_lowercase())))
}
fn database_unavailable(e: blackbeard_data::DataError) -> Failure {
tracing::warn!(error = %e, "a miner query could not reach the database");
Failure(
StatusCode::SERVICE_UNAVAILABLE,
ApiError::new(
"database_unavailable",
"miner history is temporarily unavailable",
),
)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_preimage_is_accepted_with_or_without_the_prefix() {
let bare = "134e73f06fa9bdb1dbfa909e149c563f5860ceb71a0e7307918f7033970edf59";
let expected = MinerId(format!("0x{bare}"));
assert_eq!(parse_miner(bare).ok(), Some(expected.clone()));
assert_eq!(
parse_miner(&format!("0x{bare}")).ok(),
Some(expected.clone())
);
// Case-insensitive in, canonical lowercase out, so one miner is one row.
assert_eq!(parse_miner(&bare.to_uppercase()).ok(), Some(expected));
}
#[test]
fn anything_that_is_not_a_preimage_is_rejected() {
// "not found" for a typo is indistinguishable from "not found" for a
// real miner who has not won a block yet, which is why this is a 400.
for bad in [
"",
"0x",
"0xdeadbeef",
"zz",
&"0".repeat(63),
&"0".repeat(65),
] {
assert!(parse_miner(bad).is_err(), "{bad} should be rejected");
}
}
#[test]
fn every_window_gets_a_bucket_count_a_browser_can_draw() {
for w in Window::all() {
let (span, bucket) = bucketing(w);
let points = span.as_secs() / bucket.as_secs();
assert!(
(50..=400).contains(&points),
"{w:?} would render {points} points"
);
}
}
#[test]
fn an_absent_window_parameter_means_the_default() {
let q = WindowQuery {
window: None,
limit: None,
};
assert_eq!(q.window().ok(), Some(Window::SixHours));
}
#[test]
fn an_unknown_window_is_a_bad_request_not_a_silent_default() {
let q = WindowQuery {
window: Some("fortnight".into()),
limit: None,
};
assert!(q.window().is_err());
}
}

506
crates/blackbeard-api/src/state.rs Normal file
View File

@@ -0,0 +1,506 @@
//! Per-chain runtime state, and the fanout to connected browsers.
//!
//! ## Locking
//!
//! [`ChainRuntime::inner`] is a `std::sync::RwLock`, not tokio's, and it is
//! **never held across an `.await`**. Everything under it is CPU work on
//! in-memory collections; making it async would add a scheduling hop to every
//! read for no benefit, and the tokio variant is the one that makes it *easy*
//! to accidentally hold a lock across an await and stall a worker.
//!
//! ## Fanout
//!
//! Messages are serialised **once**, at broadcast time, and every connected
//! socket writes the same `Arc<str>`. With a few hundred watchers and a
//! leaderboard every few seconds, serialising per client would be the single
//! largest cost in the process.
use std::collections::HashMap;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, RwLock};
use blackbeard_core::attribution::Attributor;
use blackbeard_core::hashrate::{self, Interval};
use blackbeard_core::window::RollingWindow;
use blackbeard_data::rpc::RpcClient;
use blackbeard_data::store::Store;
use blackbeard_data::telemetry::TelemetryFeed;
use blackbeard_entities::{
ChainId, ChainInfo, ChainStatus, ChainSummary, LeaderboardRow, RecentBlock, ServerMessage,
Window,
};
use chrono::{DateTime, Utc};
use primitive_types::U512;
use tokio::sync::broadcast;
use crate::config::ChainConfig;
/// One serialised message on its way to every subscriber of a chain.
#[derive(Debug)]
pub struct Broadcast {
/// The window this message is about, when it is window-specific.
///
/// A client watching the six-hour standings has no use for the week's, so
/// the socket task drops anything tagged with a window that is not the
/// one it subscribed to. `None` means "everyone gets this".
pub window: Option<Window>,
/// The message, already JSON.
pub json: Arc<str>,
}
/// How many messages a slow browser may fall behind before it is disconnected.
///
/// A tab that has been backgrounded on a bad connection must not make the
/// server buffer without limit. Lagging out is the right outcome: the client
/// reconnects and gets a fresh snapshot, which is cheaper and more correct than
/// replaying a queue of stale deltas.
const BROADCAST_CAPACITY: usize = 256;
/// Everything mutable about one chain.
#[derive(Debug)]
pub struct ChainInner {
/// Rolling window, sized to the longest selectable window.
pub window: RollingWindow,
/// Telemetry attribution votes.
pub attributor: Attributor,
/// The live block ticker, newest last.
pub ticker: std::collections::VecDeque<RecentBlock>,
/// Cached standings per window, with when each was computed.
///
/// The timestamp is what stops a REST-only caller reading a table that was
/// cached once and never refreshed: the recompute timer only runs for
/// windows a socket is watching, so without an age check a `curl` of the
/// leaderboard would return the same rows forever.
pub leaderboards: HashMap<Window, (std::time::Instant, Vec<LeaderboardRow>)>,
/// Current reachability.
pub status: ChainStatus,
/// Genesis hash, once the node has answered.
pub genesis: Option<String>,
/// Token ticker.
pub token_symbol: Option<String>,
/// Token decimals.
pub token_decimals: Option<u8>,
/// Current difficulty.
pub difficulty: Option<U512>,
/// Ceiling difficulty.
pub max_difficulty: Option<U512>,
/// Whether the node is still importing history.
pub syncing: bool,
/// Best height seen.
pub height: Option<u64>,
/// When the last block was observed, for the ticker's gap figure.
pub last_block_at: Option<DateTime<Utc>>,
}
/// A block waiting for the telemetry feed to settle before its author can be
/// attributed.
///
/// Lives beside the runtime rather than inside `ChainInner` because it is
/// touched on a different cadence — appended by ingest on every block, drained
/// by housekeeping once a second — and pairing it with the leaderboard's lock
/// would put both behind the same contention.
#[derive(Debug)]
pub struct PendingAttribution {
/// Block hash, the key the telemetry feed reports imports under.
pub hash: String,
/// Its author.
pub miner: blackbeard_entities::MinerId,
/// When the feed has had long enough to report.
pub due: tokio::time::Instant,
}
/// One chain, live.
#[derive(Debug)]
pub struct ChainRuntime {
/// Static configuration.
pub config: ChainConfig,
/// HTTP JSON-RPC client for this chain's node.
pub rpc: RpcClient,
/// Telemetry feed. Unconnected and inert when no URL is configured.
pub telemetry: TelemetryFeed,
/// Mutable state.
pub inner: RwLock<ChainInner>,
/// Fanout to connected browsers.
pub events: broadcast::Sender<Arc<Broadcast>>,
/// Blocks awaiting their telemetry attribution.
pub pending_attributions: std::sync::Mutex<std::collections::VecDeque<PendingAttribution>>,
/// Live subscriber count per window, so the recompute timer can skip a
/// window nobody is looking at. A week window is a hundred thousand entries
/// to re-tally; computing all four every few seconds for an empty site
/// would be the daemon's largest cost while doing nothing for anyone.
subscribers: [AtomicUsize; 4],
}
impl ChainRuntime {
/// Build a runtime for a configured chain. Does no I/O.
pub fn new(config: ChainConfig, rpc: RpcClient) -> Self {
let (events, _) = broadcast::channel(BROADCAST_CAPACITY);
Self {
inner: RwLock::new(ChainInner {
// Sized to the longest window: every shorter one is then a tail
// of the same buffer rather than separate state to keep in sync.
window: RollingWindow::new(Window::Week.blocks() as usize),
attributor: Attributor::new(),
ticker: std::collections::VecDeque::new(),
leaderboards: HashMap::new(),
// Not `Unreachable`: nothing has failed yet, and a chain that
// has not launched must not read as broken.
status: ChainStatus::Awaiting,
genesis: None,
token_symbol: None,
token_decimals: None,
difficulty: None,
max_difficulty: None,
syncing: false,
height: None,
last_block_at: None,
}),
config,
rpc,
telemetry: TelemetryFeed::new(),
events,
pending_attributions: std::sync::Mutex::new(std::collections::VecDeque::new()),
subscribers: Default::default(),
}
}
/// The chain's id.
pub fn id(&self) -> &ChainId {
&self.config.id
}
fn slot(window: Window) -> usize {
match window {
Window::Hour => 0,
Window::SixHours => 1,
Window::Day => 2,
Window::Week => 3,
}
}
/// Register interest in a window. Returns a guard that deregisters on drop,
/// so a dropped socket cannot leak a permanent subscriber.
pub fn watch(self: &Arc<Self>, window: Window) -> WindowGuard {
self.subscribers[Self::slot(window)].fetch_add(1, Ordering::Relaxed);
WindowGuard {
chain: Arc::clone(self),
window,
}
}
/// Windows currently worth computing.
pub fn watched_windows(&self) -> Vec<Window> {
Window::all()
.into_iter()
.filter(|w| self.subscribers[Self::slot(*w)].load(Ordering::Relaxed) > 0)
.collect()
}
/// Serialise and fan out a message.
///
/// A send with no receivers is not an error — an idle site is the normal
/// case, and there is nothing to do about it.
pub fn broadcast(&self, message: &ServerMessage, window: Option<Window>) {
let json = match serde_json::to_string(message) {
Ok(j) => j,
Err(e) => {
tracing::error!(error = %e, "a server message failed to serialise");
return;
}
};
let _ = self.events.send(Arc::new(Broadcast {
window,
json: Arc::from(json.as_str()),
}));
}
/// The chain's current identity and reachability.
pub fn info(&self) -> ChainInfo {
let inner = self.read();
ChainInfo {
id: self.config.id.clone(),
display_name: self.config.display_name.clone(),
mainnet: self.config.mainnet,
genesis: inner.genesis.clone(),
token_symbol: inner.token_symbol.clone(),
token_decimals: inner.token_decimals,
target_block_time_seconds: self.config.target_block_time_seconds,
status: inner.status,
}
}
/// The interval to divide difficulty by, and whether it is nominal.
///
/// A measured interval is only used when the node is following the tip. A
/// syncing node imports history at disk speed, and dividing by *that*
/// yields a hashrate wrong by orders of magnitude while looking entirely
/// plausible on a chart.
fn interval(&self, inner: &ChainInner) -> Interval {
match inner.window.measured_interval() {
Some(measured) if !inner.syncing => Interval::Measured(measured),
_ => Interval::Target(self.config.target_block_time_seconds),
}
}
/// The headline numbers.
pub fn summary(&self) -> ChainSummary {
let inner = self.read();
let interval = self.interval(&inner);
let (tallies, total) = inner.window.tally(Window::SixHours.blocks() as usize);
ChainSummary {
chain: self.config.id.clone(),
height: inner.height,
difficulty: inner.difficulty.map(hashrate::u512_to_dec),
max_difficulty: inner.max_difficulty.map(hashrate::u512_to_dec),
network_hashrate: inner
.difficulty
.map(|d| hashrate::network_hashrate(d, interval)),
hashrate_from_target: interval.is_nominal(),
block_interval_seconds: inner.window.measured_interval(),
target_block_time_seconds: self.config.target_block_time_seconds,
window_blocks: total,
distinct_miners: tallies.len() as u32,
telemetry_nodes: self.telemetry.node_count(),
telemetry_connected: self.telemetry.connected(),
updated_at: Utc::now(),
}
}
/// Recompute the standings for `window` and cache them.
///
/// Returns the rows, and whether they differ from what was cached — the
/// caller broadcasts only on a change, so a quiet chain produces no traffic
/// rather than an identical table every few seconds.
pub fn recompute_leaderboard(&self, window: Window) -> (Vec<LeaderboardRow>, bool) {
let mut inner = self.write();
let interval = self.interval(&inner);
let network = inner
.difficulty
.map(|d| hashrate::network_hashrate(d, interval));
let (tallies, total) = inner.window.tally(window.blocks() as usize);
// Split the borrow: the tally is done, and `leaderboard` needs a mutable
// attributor while it walks the rows.
let ChainInner {
attributor,
leaderboards,
..
} = &mut *inner;
let telemetry = &self.telemetry;
let rows = blackbeard_core::window::leaderboard(&tallies, total, network, |miner| {
let a = attributor.attribute(miner, |key| telemetry.name_of(key));
(a.display, a.source, a.confidence)
});
let changed = leaderboards.get(&window).map(|(_, cached)| cached) != Some(&rows);
leaderboards.insert(window, (std::time::Instant::now(), rows.clone()));
(rows, changed)
}
/// The standings for a window, recomputing them if the cache has aged out.
pub fn leaderboard(&self, window: Window, max_age: std::time::Duration) -> Vec<LeaderboardRow> {
if let Some((computed_at, rows)) = self.read().leaderboards.get(&window)
&& computed_at.elapsed() < max_age
{
return rows.clone();
}
self.recompute_leaderboard(window).0
}
/// The live ticker, oldest first.
pub fn ticker(&self) -> Vec<RecentBlock> {
self.read().ticker.iter().cloned().collect()
}
/// Read the mutable state.
pub fn read(&self) -> std::sync::RwLockReadGuard<'_, ChainInner> {
self.inner.read().unwrap_or_else(|e| e.into_inner())
}
/// Write the mutable state.
pub fn write(&self) -> std::sync::RwLockWriteGuard<'_, ChainInner> {
self.inner.write().unwrap_or_else(|e| e.into_inner())
}
}
/// Keeps a window counted as watched for as long as it is held.
pub struct WindowGuard {
chain: Arc<ChainRuntime>,
window: Window,
}
impl Drop for WindowGuard {
fn drop(&mut self) {
self.chain.subscribers[ChainRuntime::slot(self.window)].fetch_sub(1, Ordering::Relaxed);
}
}
/// Everything the HTTP handlers need.
#[derive(Clone)]
pub struct AppState {
/// Chains in configured order, which is the order the UI shows them in.
pub chains: Arc<Vec<Arc<ChainRuntime>>>,
/// Lookup by id.
pub by_id: Arc<HashMap<ChainId, Arc<ChainRuntime>>>,
/// Postgres.
pub store: Store,
/// When the process started, for the health endpoint.
pub started: DateTime<Utc>,
/// How stale a cached leaderboard may be before a REST request rebuilds it.
pub leaderboard_max_age: std::time::Duration,
}
impl AppState {
/// Find a chain by id.
pub fn chain(&self, id: &str) -> Option<&Arc<ChainRuntime>> {
self.by_id.get(&ChainId(id.to_owned()))
}
/// Every chain's identity, in configured order.
pub fn chain_infos(&self) -> Vec<ChainInfo> {
self.chains.iter().map(|c| c.info()).collect()
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
fn runtime() -> Arc<ChainRuntime> {
Arc::new(ChainRuntime::new(
ChainConfig {
id: ChainId("planck".into()),
display_name: "Planck Testnet".into(),
mainnet: false,
rpc_url: "http://127.0.0.1:9944".into(),
ws_url: "ws://127.0.0.1:9944".into(),
telemetry_url: String::new(),
target_block_time_seconds: 6.0,
warm_start_blocks: 100,
max_gap_fill_blocks: 100,
},
RpcClient::new("http://127.0.0.1:9944", Duration::from_secs(5)).unwrap(),
))
}
#[test]
fn nothing_is_computed_for_a_window_nobody_is_watching() {
let c = runtime();
assert!(c.watched_windows().is_empty());
let guard = c.watch(Window::SixHours);
assert_eq!(c.watched_windows(), vec![Window::SixHours]);
drop(guard);
// A dropped socket must not leave its window pinned forever.
assert!(c.watched_windows().is_empty());
}
#[test]
fn several_watchers_of_one_window_all_have_to_leave() {
let c = runtime();
let a = c.watch(Window::Day);
let b = c.watch(Window::Day);
drop(a);
assert_eq!(c.watched_windows(), vec![Window::Day]);
drop(b);
assert!(c.watched_windows().is_empty());
}
#[test]
fn a_chain_that_has_never_answered_is_awaiting_not_unreachable() {
// Mainnet is configured before it launches. "Unreachable" would read as
// a broken deployment for weeks.
assert_eq!(runtime().info().status, ChainStatus::Awaiting);
}
#[test]
fn hashrate_is_nominal_until_the_node_is_at_the_tip() {
let c = runtime();
{
let mut inner = c.write();
inner.difficulty = Some(U512::from(1_200_000_000u64));
inner.syncing = true;
for h in 1..=20u64 {
inner.window.push(
blackbeard_core::window::Observed {
height: h,
miner: blackbeard_entities::MinerId(format!("0x{:064x}", 1)),
observed_at: Utc::now(),
},
true,
);
}
}
let summary = c.summary();
// Twenty blocks imported in well under a second would otherwise read as
// an astronomical hashrate.
assert!(summary.hashrate_from_target);
assert_eq!(summary.network_hashrate, Some(200_000_000.0));
}
#[test]
fn identical_standings_report_no_change() {
let c = runtime();
{
let mut inner = c.write();
for h in 1..=10u64 {
inner.window.push(
blackbeard_core::window::Observed {
height: h,
miner: blackbeard_entities::MinerId(format!("0x{:064x}", h % 3)),
observed_at: Utc::now(),
},
true,
);
}
}
assert!(
c.recompute_leaderboard(Window::Hour).1,
"first compute is a change"
);
assert!(
!c.recompute_leaderboard(Window::Hour).1,
"an unchanged table must not be rebroadcast every few seconds"
);
}
#[test]
fn a_stale_cache_is_recomputed_rather_than_served() {
// Only windows a socket is watching get refreshed by the timer, so
// without an age check a REST caller would read the same rows forever.
let c = runtime();
{
let mut inner = c.write();
inner.window.push(
blackbeard_core::window::Observed {
height: 1,
miner: blackbeard_entities::MinerId(format!("0x{:064x}", 1)),
observed_at: Utc::now(),
},
true,
);
}
assert_eq!(
c.leaderboard(Window::Hour, Duration::from_secs(60)).len(),
1
);
{
let mut inner = c.write();
inner.window.push(
blackbeard_core::window::Observed {
height: 2,
miner: blackbeard_entities::MinerId(format!("0x{:064x}", 2)),
observed_at: Utc::now(),
},
true,
);
}
// A cache still inside its age is served as-is...
assert_eq!(
c.leaderboard(Window::Hour, Duration::from_secs(60)).len(),
1
);
// ...and one past it is rebuilt.
assert_eq!(c.leaderboard(Window::Hour, Duration::ZERO).len(), 2);
}
}

194
crates/blackbeard-api/src/ws.rs Normal file
View File

@@ -0,0 +1,194 @@
//! The browser socket.
//!
//! One socket per tab, carrying every chain the tab is looking at. On
//! subscribe the client gets a full [`ServerMessage::Snapshot`] and thereafter
//! only deltas, so the page never refetches and never polls.
//!
//! Each subscription is a task holding a `broadcast::Receiver` for its chain.
//! The alternative — one task selecting over a growing set of receivers — needs
//! either a `FuturesUnordered` rebuilt on every subscription change or a
//! restructure into a state machine; a task per subscription is a few hundred
//! bytes and reads as what it is.
use std::collections::HashMap;
use std::sync::Arc;
use axum::extract::State;
use axum::extract::ws::{Message, WebSocket, WebSocketUpgrade};
use axum::response::IntoResponse;
use blackbeard_entities::{ChainId, ClientMessage, ServerMessage, Window};
use futures_util::{SinkExt, StreamExt};
use tokio::sync::mpsc;
use crate::state::AppState;
/// Outbound frames buffered per socket before the writer is considered stuck.
///
/// A snapshot plus a burst of blocks is a handful of frames; anything beyond
/// this is a client that has stopped reading, and dropping it is better than
/// growing a queue on its behalf.
const CLIENT_BUFFER: usize = 64;
/// How stale a cached leaderboard may be when it goes into a snapshot.
///
/// A subscriber's window is refreshed on the recompute timer, so the cache is
/// almost always fresh; this bounds the case where the *first* subscriber to a
/// window arrives before the timer has ever run for it.
const MAX_SNAPSHOT_AGE: std::time::Duration = std::time::Duration::from_secs(2);
/// Upgrade handler for `GET /v1/ws`.
pub async fn handler(ws: WebSocketUpgrade, State(state): State<AppState>) -> impl IntoResponse {
ws.on_upgrade(move |socket| serve(socket, state))
}
async fn serve(socket: WebSocket, state: AppState) {
let (mut sink, mut stream) = socket.split();
let (tx, mut rx) = mpsc::channel::<String>(CLIENT_BUFFER);
// One writer owns the sink. Every subscription task and the reader loop
// send through the channel, so nothing has to share the socket.
let writer = tokio::spawn(async move {
while let Some(text) = rx.recv().await {
if sink.send(Message::Text(text.into())).await.is_err() {
break;
}
}
let _ = sink.close().await;
});
// Unprompted, before anything is asked for: the chain switcher can render
// immediately instead of waiting for a round trip it does not need.
send(
&tx,
&ServerMessage::Chains {
chains: state.chain_infos(),
},
)
.await;
let mut subscriptions: HashMap<ChainId, tokio::task::JoinHandle<()>> = HashMap::new();
while let Some(Ok(message)) = stream.next().await {
let text = match message {
Message::Text(t) => t,
Message::Close(_) => break,
// Ping and Pong are answered by axum; binary frames are not part of
// this protocol.
_ => continue,
};
let request: ClientMessage = match serde_json::from_str(&text) {
Ok(m) => m,
Err(e) => {
send(
&tx,
&ServerMessage::Error {
message: format!("could not read that message: {e}"),
},
)
.await;
continue;
}
};
match request {
ClientMessage::Ping => send(&tx, &ServerMessage::Pong).await,
ClientMessage::Unsubscribe { chain } => {
if let Some(task) = subscriptions.remove(&chain) {
task.abort();
}
}
ClientMessage::Subscribe { chain, window } => {
let Some(runtime) = state.chain(chain.as_str()) else {
send(
&tx,
&ServerMessage::Error {
message: format!("no chain named `{chain}` is configured"),
},
)
.await;
continue;
};
// Re-subscribing replaces: a client changing window sends
// `subscribe` again rather than unsubscribing first, and two
// live subscriptions to one chain would double every delta.
if let Some(previous) = subscriptions.remove(&chain) {
previous.abort();
}
let runtime = Arc::clone(runtime);
let tx = tx.clone();
subscriptions.insert(chain, tokio::spawn(subscription(runtime, window, tx)));
}
}
}
for (_, task) in subscriptions {
task.abort();
}
// Dropping the sender ends the writer, which closes the socket.
drop(tx);
let _ = writer.await;
}
/// Serve one chain subscription: snapshot, then deltas until aborted.
async fn subscription(
chain: Arc<crate::state::ChainRuntime>,
window: Window,
tx: mpsc::Sender<String>,
) {
// Subscribe to the fanout *before* building the snapshot. The other order
// has a gap: a block arriving between the snapshot and the subscription
// would be in neither, and the ticker would silently skip it.
let mut events = chain.events.subscribe();
// Held for the life of the subscription, so the recompute timer knows this
// window is worth computing — and stops when the socket goes away.
let _watching = chain.watch(window);
let snapshot = ServerMessage::Snapshot {
chain: chain.id().clone(),
window,
summary: chain.summary(),
leaderboard: chain.leaderboard(window, MAX_SNAPSHOT_AGE),
recent_blocks: chain.ticker(),
};
send(&tx, &snapshot).await;
loop {
match events.recv().await {
Ok(item) => {
// Window-tagged messages are for one window's watchers only.
if item.window.is_some_and(|w| w != window) {
continue;
}
if tx.send(item.json.to_string()).await.is_err() {
return;
}
}
Err(tokio::sync::broadcast::error::RecvError::Lagged(missed)) => {
// This client stopped reading long enough to fall off the
// channel. Deltas are gone, so resend a snapshot rather than
// leaving the page showing a state that silently skipped
// blocks.
tracing::debug!(chain = %chain.id(), missed, "client lagged; resending snapshot");
let snapshot = ServerMessage::Snapshot {
chain: chain.id().clone(),
window,
summary: chain.summary(),
leaderboard: chain.leaderboard(window, MAX_SNAPSHOT_AGE),
recent_blocks: chain.ticker(),
};
send(&tx, &snapshot).await;
}
Err(tokio::sync::broadcast::error::RecvError::Closed) => return,
}
}
}
async fn send(tx: &mpsc::Sender<String>, message: &ServerMessage) {
match serde_json::to_string(message) {
Ok(json) => {
let _ = tx.send(json).await;
}
Err(e) => tracing::error!(error = %e, "a server message failed to serialise"),
}
}

27
crates/blackbeard-cli/Cargo.toml Normal file
View File

@@ -0,0 +1,27 @@
[package]
name = "blackbeard-cli"
description = "Operator CLI for blackbeard.observer."
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
authors.workspace = true
repository.workspace = true
[[bin]]
name = "blackbeard"
path = "src/main.rs"
[dependencies]
blackbeard-core.workspace = true
blackbeard-data.workspace = true
blackbeard-entities.workspace = true
anyhow.workspace = true
chrono.workspace = true
clap.workspace = true
reqwest.workspace = true
serde.workspace = true
serde_json.workspace = true
sqlx.workspace = true
tokio.workspace = true

361
crates/blackbeard-cli/src/main.rs Normal file
View File

@@ -0,0 +1,361 @@
//! `blackbeard` — the operator CLI.
//!
//! Three jobs, all of them things an operator needs *before* or *around* a
//! deploy rather than during normal running:
//!
//! - **`probe`** answers "will the observer work against this node?" without
//! deploying anything. It is the command to run when a chain is added to the
//! config, and the one to run on launch day for mainnet.
//! - **`backfill`** pulls history into the database, so a newly deployed
//! observer has a leaderboard immediately instead of one that fills in over
//! the following week.
//! - **`standings`** prints the board in a terminal, which is what an operator
//! actually wants when checking whether a deploy is serving real data.
use std::time::Duration;
use anyhow::{Context, bail};
use blackbeard_core::{digest, hashrate};
use blackbeard_data::rpc::RpcClient;
use blackbeard_data::store::{BlockRecord, ChainRecord, Store};
use blackbeard_entities::{ChainId, MinerId};
use chrono::{TimeZone, Utc};
use clap::{Parser, Subcommand};
const RPC_TIMEOUT: Duration = Duration::from_secs(15);
#[derive(Parser)]
#[command(
name = "blackbeard",
version,
about = "Operator tools for blackbeard.observer"
)]
struct Cli {
#[command(subcommand)]
command: Command,
}
#[derive(Subcommand)]
enum Command {
/// Check that a node answers everything the observer needs, and print what
/// it says.
///
/// Every failure here is a failure the deployed daemon would hit silently:
/// an RPC without the QPoW runtime API produces a site with no hashrate, and
/// headers without a `pow_` digest produce an empty leaderboard — both of
/// which look like a working deployment.
Probe {
/// The node's HTTP JSON-RPC endpoint.
#[arg(long, default_value = "http://127.0.0.1:9944")]
rpc_url: String,
/// Target seconds per block, for the hashrate estimate.
#[arg(long, default_value_t = 6.0)]
target_block_time: f64,
/// How many recent headers to decode. More is a better sample of
/// whether authorship decoding actually works across blocks.
#[arg(long, default_value_t = 20)]
blocks: u64,
},
/// Read history from a node into the database.
Backfill {
/// The node's HTTP JSON-RPC endpoint.
#[arg(long, default_value = "http://127.0.0.1:9944")]
rpc_url: String,
/// Postgres connection string. mTLS in production; this command is run
/// by an operator, so it takes a URL rather than the daemon's config.
#[arg(long, env = "DATABASE_URL")]
database_url: String,
/// Chain id, matching the daemon's config.
#[arg(long)]
chain: String,
/// Human-facing chain name, used if the chain row does not exist yet.
#[arg(long)]
display_name: Option<String>,
/// First block to read. Defaults to `to` minus `count`.
#[arg(long)]
from: Option<u64>,
/// Last block to read. Defaults to the current tip.
#[arg(long)]
to: Option<u64>,
/// Blocks to read when `from` is not given.
#[arg(long, default_value_t = 3600)]
count: u64,
},
/// Print the standings from a running observer.
Standings {
/// The observer's API base.
#[arg(long, default_value = "http://127.0.0.1:25864/v1")]
api: String,
/// Chain id.
#[arg(long, default_value = "planck")]
chain: String,
/// Window: hour, six_hours, day or week.
#[arg(long, default_value = "six_hours")]
window: String,
/// Rows to print.
#[arg(long, default_value_t = 20)]
limit: usize,
},
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
match Cli::parse().command {
Command::Probe {
rpc_url,
target_block_time,
blocks,
} => probe(&rpc_url, target_block_time, blocks).await,
Command::Backfill {
rpc_url,
database_url,
chain,
display_name,
from,
to,
count,
} => {
backfill(
&rpc_url,
&database_url,
&chain,
display_name.as_deref(),
from,
to,
count,
)
.await
}
Command::Standings {
api,
chain,
window,
limit,
} => standings(&api, &chain, &window, limit).await,
}
}
async fn probe(rpc_url: &str, target_block_time: f64, blocks: u64) -> anyhow::Result<()> {
let rpc = RpcClient::new(rpc_url, RPC_TIMEOUT)?;
let health = rpc
.health()
.await
.with_context(|| format!("{rpc_url} did not answer system_health"))?;
println!(
"node peers {}, syncing {}",
health.peers, health.is_syncing
);
let genesis = rpc.genesis().await?.unwrap_or_default();
println!("genesis {genesis}");
let props = rpc.properties().await.unwrap_or_default();
println!(
"token {} ({} decimals), ss58 prefix {}",
props.token_symbol.as_deref().unwrap_or("?"),
props.token_decimals.map_or("?".into(), |d| d.to_string()),
props.ss58_format.map_or("?".into(), |p| p.to_string()),
);
let head = rpc.header(None).await?.context("node returned no head")?;
let height = head
.height()
.context("head carried an undecodable number")?;
println!("height {height}");
let difficulty = rpc.difficulty().await.context(
"QPoWApi_get_difficulty failed — without it the site has no hashrate and no difficulty",
)?;
println!("difficulty {difficulty} expected hashes per block");
println!(
"hashrate {:.3} GH/s at the {target_block_time}s target",
hashrate::network_hashrate(difficulty, hashrate::Interval::Target(target_block_time)) / 1e9
);
// The load-bearing check. Everything above can pass on a node whose headers
// this observer cannot read, and that node produces an empty leaderboard
// with no error anywhere.
println!("\ndecoding the last {blocks} headers…");
let mut decoded = 0u64;
let mut authors = std::collections::BTreeMap::<MinerId, u32>::new();
let mut timestamps = 0u64;
for h in height.saturating_sub(blocks)..height {
let Some(hash) = rpc.block_hash(h).await? else {
continue;
};
let Some(header) = rpc.header(Some(&hash)).await? else {
continue;
};
if let Some(miner) = digest::author_preimage(&header.digest.logs) {
decoded += 1;
*authors.entry(miner).or_default() += 1;
}
if rpc.block_timestamp_ms(&hash).await?.is_some() {
timestamps += 1;
}
}
println!("authors {decoded}/{blocks} headers carried a pow_ reward preimage");
println!("timestamps {timestamps}/{blocks} blocks carried a decodable timestamp inherent");
println!("distinct {} miners in that sample", authors.len());
for (miner, n) in authors.iter().take(10) {
println!(" {n:>3} {miner}");
}
if decoded == 0 {
bail!(
"no header carried a pow_ PreRuntime digest. The observer would run, report healthy, \
and show an empty leaderboard — this chain's headers are not in a shape it can read."
);
}
if decoded < blocks {
println!(
"\nnote: {} of {blocks} headers had no author digest. Genesis and any non-PoW block \
are expected; a large fraction is not.",
blocks - decoded
);
}
println!("\nOK — this node is usable.");
Ok(())
}
#[allow(clippy::too_many_arguments)]
async fn backfill(
rpc_url: &str,
database_url: &str,
chain: &str,
display_name: Option<&str>,
from: Option<u64>,
to: Option<u64>,
count: u64,
) -> anyhow::Result<()> {
let rpc = RpcClient::new(rpc_url, RPC_TIMEOUT)?;
let pool = sqlx::postgres::PgPoolOptions::new()
.max_connections(4)
.connect(database_url)
.await
.context("connecting to postgres")?;
let store = Store::from_pool(pool);
store.migrate().await?;
let tip = rpc
.header(None)
.await?
.and_then(|h| h.height())
.context("node returned no usable head")?;
let to = to.unwrap_or(tip);
let from = from.unwrap_or_else(|| to.saturating_sub(count));
if from > to {
bail!("--from {from} is above --to {to}");
}
let chain_id = ChainId(chain.to_owned());
// The chain row has to exist before any block can reference it.
store
.upsert_chain(&ChainRecord {
chain: chain_id.clone(),
display_name: display_name.unwrap_or(chain).to_owned(),
mainnet: false,
genesis: rpc.genesis().await.ok().flatten(),
token_symbol: None,
token_decimals: None,
target_block_time: 6.0,
})
.await?;
let difficulty = rpc.difficulty().await.ok().map(hashrate::u512_to_dec);
println!("reading {from}..={to} from {rpc_url}");
// Written in batches rather than one statement: a week of blocks is a
// hundred thousand rows, and a single array parameter that large is a
// needlessly large allocation on both ends.
const BATCH: usize = 500;
let mut batch: Vec<BlockRecord> = Vec::with_capacity(BATCH);
let mut written = 0u64;
let mut skipped = 0u64;
for height in from..=to {
let Some(hash) = rpc.block_hash(height).await? else {
skipped += 1;
continue;
};
let Some(header) = rpc.header(Some(&hash)).await? else {
skipped += 1;
continue;
};
let Some(miner) = digest::author_preimage(&header.digest.logs) else {
skipped += 1;
continue;
};
let authored_at = rpc
.block_timestamp_ms(&hash)
.await?
.and_then(|ms| i64::try_from(ms).ok())
.and_then(|ms| Utc.timestamp_millis_opt(ms).single());
batch.push(BlockRecord {
chain: chain_id.clone(),
height,
hash,
miner,
authored_at,
// Backfilled blocks are stamped with the author's own timestamp
// where there is one. Stamping them "now" would make a week of
// history look like it all arrived in the last minute, and every
// per-miner chart would collapse into one bucket.
observed_at: authored_at.unwrap_or_else(Utc::now),
difficulty: difficulty.as_ref().map(|d| d.0.clone()),
});
if batch.len() >= BATCH {
written += store.record_blocks(&batch).await?;
batch.clear();
println!(" {written} written…");
}
}
written += store.record_blocks(&batch).await?;
println!(
"done: {written} blocks written, {skipped} skipped (no author digest or not on chain)"
);
Ok(())
}
async fn standings(api: &str, chain: &str, window: &str, limit: usize) -> anyhow::Result<()> {
let url = format!("{api}/chains/{chain}/leaderboard?window={window}&limit={limit}");
let body: serde_json::Value = reqwest::Client::new()
.get(&url)
.timeout(RPC_TIMEOUT)
.send()
.await
.with_context(|| format!("GET {url}"))?
.error_for_status()?
.json()
.await?;
println!(
"{} · window {} ({} blocks observed)",
chain,
body["window"].as_str().unwrap_or(window),
body["observed_blocks"].as_u64().unwrap_or(0)
);
println!(
"{:>3} {:<34} {:>6} {:>7} {:>12}",
"#", "MINER", "BLOCKS", "SHARE", "EST. RATE"
);
for row in body["rows"].as_array().into_iter().flatten() {
let rate = row["hashrate_estimate"].as_f64().unwrap_or(0.0);
println!(
"{:>3} {:<34} {:>6} {:>6.1}% {:>9.2} MH/s",
row["rank"].as_u64().unwrap_or(0),
row["display"].as_str().unwrap_or("?"),
row["blocks"].as_u64().unwrap_or(0),
row["share"].as_f64().unwrap_or(0.0) * 100.0,
rate / 1e6,
);
}
Ok(())
}

17
crates/blackbeard-core/Cargo.toml Normal file
View File

@@ -0,0 +1,17 @@
[package]
name = "blackbeard-core"
description = "Chain decoding and leaderboard state for blackbeard.observer. Pure — no I/O."
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
authors.workspace = true
repository.workspace = true
[dependencies]
blackbeard-entities.workspace = true
chrono.workspace = true
hex.workspace = true
primitive-types.workspace = true
thiserror.workspace = true
tracing.workspace = true

373
crates/blackbeard-core/src/attribution.rs Normal file
View File

@@ -0,0 +1,373 @@
//! Putting a name to a reward preimage.
//!
//! The chain tells us *which preimage* won each block and nothing else. A
//! leaderboard of 64-hex-character strings is honest but unreadable, and the
//! whole point of this site is that miners can find themselves on it.
//!
//! substrate-telemetry closes the gap, indirectly. The feed reports block
//! imports with a propagation time, stamping the **first** reporter of a hash
//! with `0` and everyone after with their delay. A node that authored a block
//! imports it before it announces it, so the author — if it is on telemetry at
//! all — is nearly always the first reporter of its own blocks.
//!
//! One block proves nothing: on a well-connected network the author's lead over
//! the second reporter is tens of milliseconds, easily produced by a
//! well-peered bystander. Identity does the work instead. An author that is on
//! the feed is first on nearly *every* one of its blocks; an author that is not
//! gets first-reported by a different peer each time. So this module votes over
//! a rolling window and only shows a name once enough blocks agree.
//!
//! **What this is not.** An attributed name is an inference, never a claim the
//! miner made and never an identity check. Two operators behind one NAT, a node
//! that relays unusually fast, or a miner deliberately running its node
//! elsewhere will all mislead it. That is why [`Attribution`] carries a
//! confidence and why the UI must not render an inferred name the same way it
//! renders a preimage.
use std::collections::{HashMap, VecDeque};
use blackbeard_entities::{AttributionSource, MinerId};
/// Minimum lead over the second reporter, in milliseconds, for a first import
/// to count as a vote.
///
/// Measured on Planck: authors on the feed lead by 40–100 ms, up to 400 ms when
/// poorly peered. The threshold only has to exclude a tie — identity over many
/// blocks does the discriminating, so setting this high would reject good votes
/// without rejecting any bad ones.
pub const LEAD_MS: i64 = 20;
/// Blocks that must have been checked before any name is shown.
pub const MIN_ATTEMPTS: u32 = 3;
/// Fraction of the vote window that must agree before a name is shown.
pub const MIN_CONFIDENCE: f32 = 0.6;
/// Votes kept per author.
///
/// Bounded so a node that restarts or is renamed — which gives it a new
/// telemetry node id — takes over its own attribution after a few blocks
/// instead of having to out-vote its entire history.
pub const VOTE_WINDOW: usize = 20;
/// Stable identity for a telemetry node across the feed's reconnections.
///
/// Telemetry node ids are per-connection: they change on every reconnect and on
/// every rename, which would split an author's votes across several identities
/// and drop its name. The peer id survives both, so it is preferred; the node
/// id is the fallback for a feed entry that did not carry one.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub enum NodeKey {
/// libp2p peer id — stable across restarts and renames.
Peer(String),
/// Telemetry node id — per-connection, used only when no peer id was given.
Node(u64),
}
/// The outcome of attributing one miner.
#[derive(Debug, Clone, PartialEq)]
pub struct Attribution {
/// What to render.
pub display: String,
/// Whether `display` is an inferred node name or the abbreviated preimage.
pub source: AttributionSource,
/// Fraction of the vote window agreeing with the held name, 0.0–1.0.
pub confidence: f32,
}
#[derive(Debug, Default, Clone)]
struct AuthorVotes {
attempts: u32,
attributed: u32,
votes: VecDeque<Option<NodeKey>>,
held: Option<NodeKey>,
}
/// Rolling per-author attribution state.
#[derive(Debug, Default)]
pub struct Attributor {
authors: HashMap<MinerId, AuthorVotes>,
}
impl Attributor {
/// Empty state.
pub fn new() -> Self {
Self::default()
}
/// Record one block's evidence.
///
/// `first_reporter` is the node that reported the block first together with
/// its lead over the second reporter, or `None` when the block never
/// appeared on the feed. A `None` — and a lead too small to count — is still
/// recorded, as a vote for nobody: an author that is *not* on telemetry must
/// accumulate abstentions, so its confidence stays low and it keeps showing
/// its preimage rather than drifting onto whichever bystander happened to
/// relay it fastest.
pub fn observe(&mut self, miner: &MinerId, first_reporter: Option<(NodeKey, Option<i64>)>) {
let entry = self.authors.entry(miner.clone()).or_default();
entry.attempts += 1;
let vote = match first_reporter {
Some((key, Some(lead))) if lead >= LEAD_MS => {
entry.attributed += 1;
Some(key)
}
_ => None,
};
if entry.votes.len() == VOTE_WINDOW {
entry.votes.pop_front();
}
entry.votes.push_back(vote);
}
/// Seed a held name from persisted state, so a restart does not strip every
/// author of its name for a few blocks.
///
/// That churn is not cosmetic: a row whose display flips from a name to a
/// preimage and back splits the miner's series on every restart, which is
/// exactly the continuity the site exists to provide.
pub fn restore_held(&mut self, miner: MinerId, key: NodeKey, attempts: u32, attributed: u32) {
let entry = self.authors.entry(miner).or_default();
entry.held = Some(key.clone());
entry.attempts = attempts;
entry.attributed = attributed;
// One synthetic vote so the held name survives its first evaluation:
// hysteresis drops a name whose node has no vote left in the window,
// and a restored author has no votes at all yet.
entry.votes.push_back(Some(key));
}
/// Every author currently holding a name, for persistence.
pub fn held_names(&self) -> Vec<(MinerId, NodeKey, u32, u32)> {
self.authors
.iter()
.filter_map(|(m, a)| {
a.held
.clone()
.map(|k| (m.clone(), k, a.attempts, a.attributed))
})
.collect()
}
/// Resolve a miner's display, updating the held name.
///
/// `name_of` maps a node key to its current telemetry name; it returns
/// `None` for a node the feed has never named, in which case the preimage
/// is shown even though the vote was decisive — a nameless attribution is
/// not worth showing.
///
/// Hysteresis is deliberate: once a name is held it stays until its node
/// has no vote left in the window, or another node out-votes it outright. A
/// dip in confidence alone never falls back to the preimage, because a
/// display that flickers between the two on every wobble is worse than a
/// slightly stale name.
pub fn attribute<F>(&mut self, miner: &MinerId, name_of: F) -> Attribution
where
F: Fn(&NodeKey) -> Option<String>,
{
let fallback = Attribution {
display: miner.abbreviated(),
source: AttributionSource::Preimage,
confidence: 0.0,
};
let Some(entry) = self.authors.get_mut(miner) else {
return fallback;
};
let mut counts: HashMap<&NodeKey, u32> = HashMap::new();
for vote in entry.votes.iter().flatten() {
*counts.entry(vote).or_default() += 1;
}
let total = entry.votes.len() as f32;
let mut held = entry.held.clone();
if let Some(h) = &held
&& counts.get(h).copied().unwrap_or(0) == 0
{
held = None;
}
// Deterministic winner: ties break by key, not by hash iteration order,
// so an author's name cannot flip back and forth between two equally
// voted nodes on successive renders.
let best = counts
.iter()
.max_by(|a, b| a.1.cmp(b.1).then_with(|| b.0.cmp_key().cmp(&a.0.cmp_key())))
.map(|(k, n)| ((*k).clone(), *n));
if let Some((best_key, best_n)) = best {
match &held {
None => {
if entry.attempts >= MIN_ATTEMPTS && best_n as f32 / total >= MIN_CONFIDENCE {
held = Some(best_key);
}
}
Some(h) if &best_key != h && best_n > counts.get(h).copied().unwrap_or(0) => {
held = Some(best_key);
}
_ => {}
}
}
entry.held = held.clone();
let Some(held) = held else {
return fallback;
};
let confidence = counts.get(&held).copied().unwrap_or(0) as f32 / total.max(1.0);
match name_of(&held) {
Some(name) if !name.is_empty() => Attribution {
display: name,
source: AttributionSource::Telemetry,
confidence,
},
_ => fallback,
}
}
}
impl NodeKey {
/// A total order over keys, so tie-breaks are deterministic.
fn cmp_key(&self) -> (u8, String) {
match self {
NodeKey::Peer(p) => (0, p.clone()),
NodeKey::Node(n) => (1, n.to_string()),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn miner() -> MinerId {
MinerId("0x134e73f06fa9bdb1dbfa909e149c563f5860ceb71a0e7307918f7033970edf59".into())
}
fn named(key: &NodeKey) -> Option<String> {
match key {
NodeKey::Peer(p) => Some(format!("node-{p}")),
NodeKey::Node(n) => Some(format!("node-{n}")),
}
}
#[test]
fn a_consistent_first_reporter_earns_the_name() {
let mut a = Attributor::new();
let key = NodeKey::Peer("12D3KooWabc".into());
for _ in 0..5 {
a.observe(&miner(), Some((key.clone(), Some(80))));
}
let got = a.attribute(&miner(), named);
assert_eq!(got.source, AttributionSource::Telemetry);
assert_eq!(got.display, "node-12D3KooWabc");
assert_eq!(got.confidence, 1.0);
}
#[test]
fn too_few_blocks_shows_the_preimage() {
let mut a = Attributor::new();
a.observe(&miner(), Some((NodeKey::Peer("p".into()), Some(80))));
let got = a.attribute(&miner(), named);
assert_eq!(got.source, AttributionSource::Preimage);
assert_eq!(got.display, miner().abbreviated());
}
#[test]
fn a_miner_not_on_telemetry_never_gets_a_name() {
// Different bystander first-reports each block — exactly the signature
// of an author that is not on the feed. Naming any of them would be a
// confident lie.
let mut a = Attributor::new();
for i in 0..20u64 {
a.observe(&miner(), Some((NodeKey::Node(i), Some(80))));
}
assert_eq!(
a.attribute(&miner(), named).source,
AttributionSource::Preimage
);
}
#[test]
fn a_lead_below_the_threshold_is_an_abstention() {
let mut a = Attributor::new();
let key = NodeKey::Peer("p".into());
for _ in 0..10 {
a.observe(&miner(), Some((key.clone(), Some(1))));
}
assert_eq!(
a.attribute(&miner(), named).source,
AttributionSource::Preimage
);
}
#[test]
fn blocks_absent_from_the_feed_are_abstentions() {
let mut a = Attributor::new();
for _ in 0..10 {
a.observe(&miner(), None);
}
assert_eq!(
a.attribute(&miner(), named).source,
AttributionSource::Preimage
);
}
#[test]
fn a_held_name_survives_a_confidence_dip() {
let mut a = Attributor::new();
let key = NodeKey::Peer("p".into());
for _ in 0..10 {
a.observe(&miner(), Some((key.clone(), Some(80))));
}
assert_eq!(
a.attribute(&miner(), named).source,
AttributionSource::Telemetry
);
// Five blocks the feed missed. Confidence falls to 10/15, still named.
for _ in 0..5 {
a.observe(&miner(), None);
}
let got = a.attribute(&miner(), named);
assert_eq!(got.source, AttributionSource::Telemetry);
assert!(got.confidence < 1.0);
}
#[test]
fn a_restarted_node_takes_over_its_own_attribution() {
let mut a = Attributor::new();
let old = NodeKey::Node(1);
for _ in 0..VOTE_WINDOW {
a.observe(&miner(), Some((old.clone(), Some(80))));
}
assert_eq!(a.attribute(&miner(), named).display, "node-1");
// Same operator, new telemetry node id after a restart. The bounded
// vote window means it wins in VOTE_WINDOW blocks rather than never.
let new = NodeKey::Node(2);
for _ in 0..VOTE_WINDOW {
a.observe(&miner(), Some((new.clone(), Some(80))));
}
assert_eq!(a.attribute(&miner(), named).display, "node-2");
}
#[test]
fn a_nameless_node_falls_back_to_the_preimage() {
let mut a = Attributor::new();
let key = NodeKey::Peer("p".into());
for _ in 0..10 {
a.observe(&miner(), Some((key.clone(), Some(80))));
}
assert_eq!(
a.attribute(&miner(), |_| None).source,
AttributionSource::Preimage
);
}
#[test]
fn restored_state_names_the_author_before_any_new_votes() {
let mut a = Attributor::new();
a.restore_held(miner(), NodeKey::Peer("p".into()), 40, 38);
assert_eq!(a.attribute(&miner(), named).display, "node-p");
assert_eq!(a.held_names().len(), 1);
}
#[test]
fn an_unknown_miner_is_its_preimage() {
let mut a = Attributor::new();
assert_eq!(a.attribute(&miner(), named).display, miner().abbreviated());
}
}

138
crates/blackbeard-core/src/digest.rs Normal file
View File

@@ -0,0 +1,138 @@
//! Reading a block's author out of its header.
//!
//! This is the whole reason the observer needs no indexer, no cooperation from
//! miners, and no permission from anyone: **every** block header carries its
//! author's 32-byte wormhole reward preimage in a `PreRuntime` digest log
//! stamped with the engine id `pow_`. Authorship for the entire network is
//! therefore derivable from headers alone, which is what makes a complete
//! leaderboard possible rather than a leaderboard of whoever opted in.
//!
//! The node writes it there so the `mining-rewards` pallet can read it back and
//! derive the payout address on-chain. That it is also a perfect public miner
//! identifier is a side effect of the design, not a leak: spending the rewards
//! requires a plonky2 proof of knowledge of the underlying secret, which the
//! preimage does not give up.
use blackbeard_entities::MinerId;
use crate::scale;
/// `DigestItem::PreRuntime` discriminant, as the first byte of an encoded log.
const DIGEST_PRE_RUNTIME: u8 = 0x06;
/// Consensus engine id `b"pow_"`.
const POW_ENGINE_ID: [u8; 4] = *b"pow_";
/// A wormhole reward preimage is a Poseidon2 digest: 32 bytes, always.
const PREIMAGE_LEN: usize = 32;
/// Pull the author's reward preimage out of a header's digest logs.
///
/// Returns `None` for a header carrying no `pow_` PreRuntime log — genesis, or
/// a header shape this decoder does not recognise. Neither is worth failing an
/// ingest loop over: the block simply does not contribute to the leaderboard.
///
/// `logs` are the hex strings exactly as `chain_getHeader` returns them.
pub fn author_preimage<S: AsRef<str>>(logs: &[S]) -> Option<MinerId> {
for log in logs {
let raw = log.as_ref();
let bytes = match hex::decode(raw.strip_prefix("0x").unwrap_or(raw)) {
Ok(b) => b,
Err(_) => continue,
};
// discriminant + 4-byte engine id, then a SCALE-encoded byte vector.
if bytes.first() != Some(&DIGEST_PRE_RUNTIME) || bytes.get(1..5) != Some(&POW_ENGINE_ID) {
continue;
}
let (len, body_start) = scale::compact(&bytes, 5).ok()?;
let payload = bytes.get(body_start..body_start + len as usize)?;
if payload.len() != PREIMAGE_LEN {
// A `pow_` log that is not 32 bytes is not a preimage. Refusing it
// is right: a truncated or padded id would silently split or merge
// leaderboard rows.
return None;
}
return Some(MinerId(format!("0x{}", hex::encode(payload))));
}
None
}
/// Parse a `0x`-prefixed **little-endian** U512 hex string, as the QPoW runtime
/// API returns difficulty.
///
/// Little-endian is the trap here: `state_call` hands back a SCALE-encoded
/// `U512`, which is byte-reversed relative to how a hash or an address reads.
/// Interpreting it big-endian yields a plausible-looking astronomically wrong
/// number, and every hashrate derived from it is wrong by orders of magnitude
/// without ever looking obviously broken.
pub fn u512_le(hex_str: &str) -> Option<primitive_types::U512> {
let bytes = hex::decode(hex_str.strip_prefix("0x").unwrap_or(hex_str)).ok()?;
if bytes.is_empty() || bytes.len() > 64 {
return None;
}
Some(primitive_types::U512::from_little_endian(&{
let mut buf = [0u8; 64];
buf[..bytes.len()].copy_from_slice(&bytes);
buf
}))
}
#[cfg(test)]
mod tests {
use super::*;
/// A real Planck header digest log: discriminant 06, engine id 706f775f
/// ("pow_"), compact length 0x80 (= 32), then the 32-byte preimage.
fn sample_log(preimage_hex: &str) -> String {
format!("0x06706f775f80{preimage_hex}")
}
#[test]
fn extracts_the_preimage_from_a_pow_digest() {
let pre = "134e73f06fa9bdb1dbfa909e149c563f5860ceb71a0e7307918f7033970edf59";
let logs = vec![sample_log(pre)];
assert_eq!(author_preimage(&logs), Some(MinerId(format!("0x{pre}"))));
}
#[test]
fn skips_logs_from_other_engines() {
let pre = "134e73f06fa9bdb1dbfa909e149c563f5860ceb71a0e7307918f7033970edf59";
// A Seal log and a different engine id, then the one that matters.
let logs = vec![
"0x05616c6c2f80deadbeef".to_string(),
"0x06424142453401000000".to_string(),
sample_log(pre),
];
assert_eq!(author_preimage(&logs), Some(MinerId(format!("0x{pre}"))));
}
#[test]
fn genesis_has_no_author() {
let empty: Vec<String> = vec![];
assert_eq!(author_preimage(&empty), None);
}
#[test]
fn rejects_a_pow_log_that_is_not_32_bytes() {
// Length says 4, so this is not a preimage — better no author than a
// truncated one that would open a bogus leaderboard row.
assert_eq!(
author_preimage(&["0x06706f775f10deadbeef".to_string()]),
None
);
}
#[test]
fn difficulty_is_little_endian() {
// 0x0100…00 little-endian is 1, not 2^504. Getting this backwards
// produces a wrong-by-10^150 hashrate that still renders as a number.
let mut le = String::from("0x01");
le.push_str(&"00".repeat(63));
assert_eq!(u512_le(&le), Some(primitive_types::U512::one()));
}
#[test]
fn difficulty_tolerates_a_short_encoding() {
assert_eq!(u512_le("0x0a"), Some(primitive_types::U512::from(10u8)));
assert_eq!(u512_le("0x"), None);
assert_eq!(u512_le("0xzz"), None);
}
}

139
crates/blackbeard-core/src/hashrate.rs Normal file
View File

@@ -0,0 +1,139 @@
//! Turning difficulty and block timing into hashrate.
//!
//! The arithmetic is one division, but two things about it are easy to get
//! wrong and expensive to notice, so they live here with the reasoning attached.
use blackbeard_entities::BigUintDec;
use primitive_types::U512;
/// A U512 as an `f64`, for arithmetic that only needs magnitude.
///
/// Difficulty is routinely past `1e30`, so it has long since exceeded `f64`'s
/// 53-bit exact-integer range — but hashrate is an estimate over a finite
/// window whose error is dominated by block-timing variance, not by the last
/// significant bit of the divisor. Precision is kept exactly where it matters
/// instead: the difficulty *displayed* and the difficulty *stored* both go
/// through [`BigUintDec`], never through this.
pub fn u512_to_f64(v: U512) -> f64 {
// Limbs are little-endian; fold from the top so the accumulator never has
// to represent more than one limb's worth of new magnitude at a time.
v.0.iter().rev().fold(0.0f64, |acc, &limb| {
acc * 18_446_744_073_709_551_616.0 + limb as f64
})
}
/// Render a U512 as the decimal string that crosses every boundary.
pub fn u512_to_dec(v: U512) -> BigUintDec {
BigUintDec(v.to_string())
}
/// Estimated network hashrate in hashes per second.
///
/// Difficulty **is** the expected number of hashes to win a block, so hashrate
/// is simply difficulty divided by the seconds between blocks. The subtlety is
/// which interval to divide by, which is what [`Interval`] exists to make
/// explicit at the call site.
pub fn network_hashrate(difficulty: U512, interval: Interval) -> f64 {
let seconds = interval.seconds();
if seconds <= 0.0 {
return 0.0;
}
u512_to_f64(difficulty) / seconds
}
/// Which block interval a hashrate figure was derived from.
///
/// A measured interval is only meaningful **at the tip**. While a node is
/// catching up it imports historical blocks at disk speed, so the elapsed time
/// between observing consecutive blocks is a measure of the node's disk, not of
/// the network's hashrate — and dividing by it yields a hashrate that is wrong
/// by several orders of magnitude while looking entirely plausible. Making the
/// two cases separate variants means a caller cannot forget to say which it has,
/// and the UI can label a nominal figure as nominal.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum Interval {
/// Mean seconds per block measured from tip observations.
Measured(f64),
/// The chain's configured target, used when no measurement is trustworthy.
Target(f64),
}
impl Interval {
/// The interval in seconds.
pub fn seconds(self) -> f64 {
match self {
Interval::Measured(s) | Interval::Target(s) => s,
}
}
/// True when this is the configured target rather than an observation.
pub fn is_nominal(self) -> bool {
matches!(self, Interval::Target(_))
}
}
/// A miner's implied hashrate: its share of blocks times the network's.
///
/// This is a statistical estimate over a finite window, not a measurement of
/// anyone's hardware. Winning blocks is a Poisson process, so a miner holding
/// 1% of the network hashrate will, over a 600-block window, quite ordinarily
/// show anywhere from 2 to 11 blocks. Short windows flatter the lucky and libel
/// the unlucky; the UI should show the window length beside the figure.
pub fn miner_hashrate(blocks_in_window: u32, window_total: u32, network: f64) -> Option<f64> {
if window_total == 0 {
return None;
}
Some(network * f64::from(blocks_in_window) / f64::from(window_total))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn u512_to_f64_survives_astronomical_difficulty() {
// 2^128, well past f64's exact-integer range but comfortably inside its
// exponent range — magnitude is what matters here.
let v = U512::from(1u8) << 128;
let f = u512_to_f64(v);
assert!((f / 2f64.powi(128) - 1.0).abs() < 1e-12);
}
#[test]
fn u512_to_f64_is_exact_for_small_values() {
assert_eq!(u512_to_f64(U512::from(6u8)), 6.0);
assert_eq!(u512_to_f64(U512::zero()), 0.0);
}
#[test]
fn hashrate_is_difficulty_over_interval() {
let d = U512::from(1_200_000_000u64);
assert_eq!(network_hashrate(d, Interval::Measured(6.0)), 200_000_000.0);
assert_eq!(network_hashrate(d, Interval::Target(6.0)), 200_000_000.0);
}
#[test]
fn a_zero_interval_yields_zero_not_infinity() {
// Two blocks in the same second is ordinary; an infinite hashrate on
// the dashboard is not.
assert_eq!(
network_hashrate(U512::from(10u8), Interval::Measured(0.0)),
0.0
);
}
#[test]
fn nominal_is_distinguishable_from_measured() {
assert!(Interval::Target(6.0).is_nominal());
assert!(!Interval::Measured(6.0).is_nominal());
}
#[test]
fn miner_share_scales_the_network_figure() {
assert_eq!(miner_hashrate(25, 100, 400.0), Some(100.0));
assert_eq!(miner_hashrate(0, 100, 400.0), Some(0.0));
// An empty window has no share to compute, and 0/0 must not become NaN
// on the dashboard.
assert_eq!(miner_hashrate(5, 0, 400.0), None);
}
}

32
crates/blackbeard-core/src/lib.rs Normal file
View File

@@ -0,0 +1,32 @@
//! Chain decoding and leaderboard state for blackbeard.observer.
//!
//! Everything here is pure: given the same headers and the same telemetry
//! reports it produces the same standings, with no clock of its own, no socket
//! and no database. That is deliberate — the interesting parts of this project
//! are the decoding (which is easy to get subtly wrong and hard to notice) and
//! the statistics (which are easy to overstate), and both are far easier to
//! trust when they can be exercised by a unit test rather than only against a
//! live chain.
//!
//! The I/O that feeds it lives in `blackbeard-data`; the orchestration that
//! drives it lives in `blackbeard-api`.
#![deny(missing_docs)]
pub mod attribution;
pub mod digest;
pub mod hashrate;
pub mod scale;
pub mod window;
/// Failures decoding what a node handed us.
#[derive(Debug, thiserror::Error, PartialEq)]
pub enum CoreError {
/// A SCALE value ran off the end of its buffer.
#[error("truncated SCALE encoding")]
TruncatedScale,
/// A compact integer wider than anything this decoder reads.
#[error("compact integer of {0} bytes is wider than this decoder reads")]
CompactTooWide(usize),
}

151
crates/blackbeard-core/src/scale.rs Normal file
View File

@@ -0,0 +1,151 @@
//! The two pieces of SCALE this project needs, and nothing else.
//!
//! Pulling in a full SCALE codec plus chain metadata would buy generality we
//! never use: the observer reads exactly two things out of a block — the
//! author's reward preimage from a digest log and the author's timestamp from
//! the first extrinsic — and both are fixed-shape. Hand-decoding them keeps the
//! crate dependency-light and, more usefully, keeps the decoding auditable
//! against the runtime source.
use crate::CoreError;
/// Decode a SCALE compact integer at `offset`.
///
/// Returns the value and the offset just past it. The four modes are a 6-bit
/// single byte, a 14-bit u16, a 30-bit u32, and a big-integer form whose first
/// byte carries the length.
pub fn compact(bytes: &[u8], offset: usize) -> Result<(u64, usize), CoreError> {
let first = *bytes.get(offset).ok_or(CoreError::TruncatedScale)?;
match first & 0b11 {
0b00 => Ok((u64::from(first >> 2), offset + 1)),
0b01 => {
let raw = bytes
.get(offset..offset + 2)
.ok_or(CoreError::TruncatedScale)?;
Ok((
u64::from(u16::from_le_bytes([raw[0], raw[1]]) >> 2),
offset + 2,
))
}
0b10 => {
let raw = bytes
.get(offset..offset + 4)
.ok_or(CoreError::TruncatedScale)?;
let v = u32::from_le_bytes([raw[0], raw[1], raw[2], raw[3]]) >> 2;
Ok((u64::from(v), offset + 4))
}
_ => {
let len = usize::from(first >> 2) + 4;
if len > 8 {
// Nothing this decoder reads is wider than a u64: a compact
// length prefix or a millisecond timestamp. A wider value means
// we are decoding something we did not expect, and guessing at
// it would be worse than refusing.
return Err(CoreError::CompactTooWide(len));
}
let raw = bytes
.get(offset + 1..offset + 1 + len)
.ok_or(CoreError::TruncatedScale)?;
let mut buf = [0u8; 8];
buf[..len].copy_from_slice(raw);
Ok((u64::from_le_bytes(buf), offset + 1 + len))
}
}
}
/// The `Timestamp` pallet's index in the runtime (`pallet_index(1)`), call `0`
/// = `set`. The timestamp inherent is always a block's first extrinsic.
const TIMESTAMP_PALLET_INDEX: u8 = 1;
const TIMESTAMP_CALL_SET: u8 = 0;
/// Extract the author's timestamp, in milliseconds, from a block's first
/// extrinsic.
///
/// Returns `None` — rather than an error — when the extrinsic is not the
/// timestamp inherent. A block whose first extrinsic is something else is a
/// runtime shape we do not recognise, and the honest response is "no author
/// timestamp for this block", not a crash in the ingest loop.
///
/// This timestamp is set when the author **builds** the proposal, not when the
/// block reaches the network. That is what makes it useful: compared against
/// when the observer saw the block, it separates proposal timing from release
/// timing, and a block released long after it was built is a withheld one.
pub fn timestamp_inherent_ms(extrinsic_hex: &str) -> Option<u64> {
let bytes = hex::decode(extrinsic_hex.strip_prefix("0x").unwrap_or(extrinsic_hex)).ok()?;
// Extrinsics are length-prefixed inside the block body.
let (_, i) = compact(&bytes, 0).ok()?;
// 0x04 is a v4 unsigned extrinsic, 0x05 a v5 bare one. Inherents are
// unsigned; anything else here is not the timestamp call.
let version = *bytes.get(i)?;
if version != 0x04 && version != 0x05 {
return None;
}
if *bytes.get(i + 1)? != TIMESTAMP_PALLET_INDEX || *bytes.get(i + 2)? != TIMESTAMP_CALL_SET {
return None;
}
compact(&bytes, i + 3).ok().map(|(ms, _)| ms)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn compact_covers_all_four_modes() {
assert_eq!(compact(&[0b0000_0100], 0).unwrap(), (1, 1));
// 0b01: 14-bit. 0x15 0x01 => 0x0115 >> 2 = 69.
assert_eq!(compact(&[0x15, 0x01], 0).unwrap(), (69, 2));
// 0b10: 30-bit. 0xfeff0300 >> 2 = 65535.
assert_eq!(compact(&[0xfe, 0xff, 0x03, 0x00], 0).unwrap(), (65535, 4));
// big-integer form, 4 bytes of payload.
assert_eq!(
compact(&[0x03, 0x00, 0x00, 0x00, 0x40], 0).unwrap(),
(0x4000_0000, 5)
);
}
#[test]
fn compact_refuses_to_run_off_the_end() {
assert!(matches!(compact(&[], 0), Err(CoreError::TruncatedScale)));
assert!(matches!(
compact(&[0x15], 0),
Err(CoreError::TruncatedScale)
));
}
#[test]
fn compact_refuses_values_wider_than_u64() {
// 0b11 with (first >> 2) = 12 means 16 payload bytes: a u128. Nothing
// this decoder reads is that wide, so it must refuse rather than
// truncate silently.
assert!(matches!(
compact(
&[0b0011_0011, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
0
),
Err(CoreError::CompactTooWide(16))
));
}
#[test]
fn timestamp_inherent_decodes_a_v4_set_call() {
// length prefix (0x2c = 11 bytes), v4 unsigned, pallet 1, call 0,
// compact u64 timestamp 1_757_000_000_000 ms.
let mut body = vec![0x04u8, TIMESTAMP_PALLET_INDEX, TIMESTAMP_CALL_SET];
// 1_757_000_000_000 needs 6 bytes: big-integer compact, (6-4)<<2 | 0b11.
body.push(((6 - 4) << 2) | 0b11);
body.extend_from_slice(&1_757_000_000_000u64.to_le_bytes()[..6]);
let mut framed = vec![(body.len() as u8) << 2];
framed.extend_from_slice(&body);
let hex = format!("0x{}", hex::encode(framed));
assert_eq!(timestamp_inherent_ms(&hex), Some(1_757_000_000_000));
}
#[test]
fn timestamp_inherent_declines_anything_else() {
// A signed extrinsic (version byte 0x84) is not the inherent.
assert_eq!(timestamp_inherent_ms("0x0c840100"), None);
assert_eq!(timestamp_inherent_ms("not hex"), None);
assert_eq!(timestamp_inherent_ms(""), None);
}
}

448
crates/blackbeard-core/src/window.rs Normal file
View File

@@ -0,0 +1,448 @@
//! The rolling window and the standings computed from it.
//!
//! One window per chain, sized to the **longest** selectable window. Every
//! shorter window is then the tail of the same buffer, so switching from a week
//! to an hour costs a slice rather than a second set of state to keep in sync.
use std::collections::HashMap;
use blackbeard_entities::{AttributionSource, LeaderboardRow, MinerId};
use chrono::{DateTime, Utc};
/// One observed block, reduced to what the standings need.
#[derive(Debug, Clone, PartialEq)]
pub struct Observed {
/// Block number.
pub height: u64,
/// Author's reward preimage.
pub miner: MinerId,
/// When this observer first saw it.
pub observed_at: DateTime<Utc>,
}
/// Per-miner totals over a window.
#[derive(Debug, Clone, PartialEq)]
pub struct MinerTally {
/// Blocks authored in the window.
pub blocks: u32,
/// Highest block this miner authored in the window.
pub last_block_height: u64,
/// When that block was observed.
pub last_seen: DateTime<Utc>,
/// Longest run of back-to-back blocks by this miner inside the window.
pub best_streak: u32,
}
/// A bounded ring of observed blocks, plus the tip samples that give the
/// measured block interval.
#[derive(Debug)]
pub struct RollingWindow {
capacity: usize,
blocks: std::collections::VecDeque<Observed>,
/// Tip observations only: `(when observed, height)`. Blocks backfilled from
/// history must never land here — the elapsed time between importing two
/// historical blocks measures the node's disk, not the network's hashrate.
tip_samples: std::collections::VecDeque<(DateTime<Utc>, u64)>,
}
/// Tip samples needed before a measured interval is reported at all.
///
/// Winning a block is a Poisson process, so consecutive gaps on a live chain
/// vary wildly — measured on Planck: 1.6 s, 4.4 s, 13.5 s, 26 s, 27 s in a row
/// against a 6 s target. Averaging five of those produces a headline hashrate
/// that swings by a factor of three between refreshes and is simply wrong. At
/// twenty samples the mean is stable enough to publish, and until then the
/// configured target is used and flagged as nominal.
const MIN_TIP_SAMPLES: usize = 20;
/// Tip samples retained. At a 6 s target this is roughly the last 20 minutes —
/// long enough to smooth Poisson noise, short enough to follow a real change.
const MAX_TIP_SAMPLES: usize = 200;
impl RollingWindow {
/// A window holding at most `capacity` blocks.
pub fn new(capacity: usize) -> Self {
Self {
capacity,
blocks: std::collections::VecDeque::with_capacity(capacity.min(4096)),
tip_samples: std::collections::VecDeque::with_capacity(MAX_TIP_SAMPLES),
}
}
/// Record a block.
///
/// `at_tip` is false for blocks read out of history — a startup backfill or
/// a gap fill after an outage. Those still count towards the standings (the
/// leaderboard would otherwise restart empty on every deploy) but
/// contribute no timing sample.
pub fn push(&mut self, observed: Observed, at_tip: bool) {
if at_tip {
if self.tip_samples.len() == MAX_TIP_SAMPLES {
self.tip_samples.pop_front();
}
self.tip_samples
.push_back((observed.observed_at, observed.height));
}
if self.blocks.len() == self.capacity {
self.blocks.pop_front();
}
self.blocks.push_back(observed);
}
/// Blocks currently held.
pub fn len(&self) -> usize {
self.blocks.len()
}
/// Whether the window has seen anything yet.
pub fn is_empty(&self) -> bool {
self.blocks.is_empty()
}
/// The most recently observed block.
pub fn latest(&self) -> Option<&Observed> {
self.blocks.back()
}
/// Mean seconds per block, measured at the tip.
///
/// `None` until enough tip samples exist. Derived from height difference
/// rather than sample count so a missed poll — which observes two blocks at
/// once — does not read as the chain having sped up.
pub fn measured_interval(&self) -> Option<f64> {
if self.tip_samples.len() < MIN_TIP_SAMPLES {
return None;
}
let (first_at, first_h) = *self.tip_samples.front()?;
let (last_at, last_h) = *self.tip_samples.back()?;
let span = (last_at - first_at).as_seconds_f64();
let heights = last_h.checked_sub(first_h)?;
if span <= 0.0 || heights == 0 {
return None;
}
Some(span / heights as f64)
}
/// The last `n` blocks, newest last.
pub fn tail(&self, n: usize) -> impl DoubleEndedIterator<Item = &Observed> {
self.blocks.iter().skip(self.blocks.len().saturating_sub(n))
}
/// Per-miner totals over the last `n` blocks, plus how many blocks that
/// actually was (fewer than `n` before the window has filled).
pub fn tally(&self, n: usize) -> (HashMap<MinerId, MinerTally>, u32) {
let mut tallies: HashMap<MinerId, MinerTally> = HashMap::new();
let mut total = 0u32;
// Streaks are runs of *consecutive observations*, which is what the
// selfish-mining signature actually looks like. A gap in the observed
// heights breaks the run rather than joining across it, so a restart
// cannot manufacture a streak that never happened.
let mut run: Option<(MinerId, u64, u32)> = None;
for b in self.tail(n) {
total += 1;
let entry = tallies.entry(b.miner.clone()).or_insert(MinerTally {
blocks: 0,
last_block_height: b.height,
last_seen: b.observed_at,
best_streak: 0,
});
entry.blocks += 1;
if b.height >= entry.last_block_height {
entry.last_block_height = b.height;
entry.last_seen = b.observed_at;
}
run = match run.take() {
Some((miner, prev_height, len))
if miner == b.miner && b.height == prev_height + 1 =>
{
Some((miner, b.height, len + 1))
}
_ => Some((b.miner.clone(), b.height, 1)),
};
if let Some((miner, _, len)) = &run
&& let Some(t) = tallies.get_mut(miner)
&& *len > t.best_streak
{
t.best_streak = *len;
}
}
(tallies, total)
}
}
/// Build the standings for a window.
///
/// Ranking is by blocks won, ties broken by the most recent block — so between
/// two miners on equal blocks, the one still winning them ranks higher. Rows
/// are not truncated here: capping belongs to the caller, which knows whether
/// it is filling a top-ten panel or answering a query for a specific miner
/// that may sit at rank four hundred.
pub fn leaderboard<F>(
tallies: &HashMap<MinerId, MinerTally>,
window_total: u32,
network_hashrate: Option<f64>,
mut attribute: F,
) -> Vec<LeaderboardRow>
where
F: FnMut(&MinerId) -> (String, AttributionSource, f32),
{
let mut ranked: Vec<(&MinerId, &MinerTally)> = tallies.iter().collect();
ranked.sort_by(|a, b| {
b.1.blocks
.cmp(&a.1.blocks)
.then_with(|| b.1.last_block_height.cmp(&a.1.last_block_height))
// Final tie-break on the id itself: without it, two miners with
// identical blocks and heights would swap places between renders
// purely on hash-map iteration order, and the table would jitter.
.then_with(|| a.0.cmp(b.0))
});
let mut rows: Vec<LeaderboardRow> = ranked
.into_iter()
.enumerate()
.map(|(i, (miner, tally))| {
let (display, attribution, confidence) = attribute(miner);
LeaderboardRow {
rank: i as u32 + 1,
miner: miner.clone(),
display,
attribution,
confidence,
blocks: tally.blocks,
share: if window_total == 0 {
0.0
} else {
f64::from(tally.blocks) / f64::from(window_total)
},
hashrate_estimate: network_hashrate
.and_then(|n| crate::hashrate::miner_hashrate(tally.blocks, window_total, n)),
last_block_height: tally.last_block_height,
last_seen: tally.last_seen,
best_streak: tally.best_streak,
}
})
.collect();
disambiguate(&mut rows);
rows
}
/// Make every display name in a set of standings unique.
///
/// Two miners can legitimately arrive with the same name: one operator running
/// several reward preimages from identically-named nodes, or two people who
/// never renamed theirs off the default. Either way, two rows reading
/// `QuantusMinerGUI` side by side are indistinguishable — a miner looking for
/// their own row cannot tell which is theirs, which is exactly the question the
/// board exists to answer. The preimage is appended to *every* member of a
/// colliding group, so no row is silently privileged as "the real one".
///
/// The suffix is the preimage's last four hex characters, not the full
/// abbreviated form: it only has to separate the two or three members of one
/// colliding group, and a longer suffix pushed the standings wide enough to
/// clip the hashrate column. The full preimage stays one hover away.
///
/// Only the standings are disambiguated. The block ticker is a chronological
/// feed rather than a comparison, and each of its rows links to one miner, so a
/// repeated name there costs nothing.
fn disambiguate(rows: &mut [LeaderboardRow]) {
let mut counts: HashMap<&str, u32> = HashMap::new();
for row in rows.iter() {
*counts.entry(row.display.as_str()).or_default() += 1;
}
let ambiguous: std::collections::HashSet<String> = counts
.into_iter()
.filter(|(_, n)| *n > 1)
.map(|(name, _)| name.to_owned())
.collect();
if ambiguous.is_empty() {
return;
}
for row in rows.iter_mut() {
if ambiguous.contains(&row.display) {
let id = row.miner.as_str();
row.display = format!("{} · {}", row.display, &id[id.len().saturating_sub(4)..]);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use chrono::TimeZone;
fn miner(n: u8) -> MinerId {
MinerId(format!("0x{:064x}", n))
}
fn at(secs: i64) -> DateTime<Utc> {
Utc.timestamp_opt(1_757_000_000 + secs, 0).unwrap()
}
fn obs(height: u64, m: u8, secs: i64) -> Observed {
Observed {
height,
miner: miner(m),
observed_at: at(secs),
}
}
#[test]
fn the_window_drops_the_oldest_block_when_full() {
let mut w = RollingWindow::new(3);
for h in 1..=5 {
w.push(obs(h, 1, h as i64), true);
}
assert_eq!(w.len(), 3);
assert_eq!(w.latest().unwrap().height, 5);
assert_eq!(w.tail(10).next().unwrap().height, 3);
}
#[test]
fn a_short_window_is_the_tail_of_a_long_one() {
let mut w = RollingWindow::new(100);
for h in 1..=50 {
w.push(obs(h, if h % 2 == 0 { 1 } else { 2 }, h as i64), true);
}
let (all, total_all) = w.tally(100);
assert_eq!(total_all, 50);
assert_eq!(all[&miner(1)].blocks, 25);
let (recent, total_recent) = w.tally(10);
assert_eq!(total_recent, 10);
assert_eq!(recent[&miner(1)].blocks, 5);
}
#[test]
fn interval_needs_enough_tip_samples() {
let mut w = RollingWindow::new(200);
for h in 1..MIN_TIP_SAMPLES as u64 {
w.push(obs(h, 1, h as i64 * 6), true);
}
assert_eq!(
w.measured_interval(),
None,
"too few samples to publish a mean"
);
w.push(
obs(MIN_TIP_SAMPLES as u64, 1, MIN_TIP_SAMPLES as i64 * 6),
true,
);
assert_eq!(w.measured_interval(), Some(6.0));
}
#[test]
fn a_missed_poll_does_not_read_as_a_faster_chain() {
// Blocks arrive between polls; the interval must come out of the height
// difference, not the number of samples taken. Here every third height
// is missed, so a count-based mean would report 9 s instead of 6 s.
let mut w = RollingWindow::new(200);
let mut height = 1u64;
for i in 0..MIN_TIP_SAMPLES + 5 {
w.push(obs(height, 1, (height as i64 - 1) * 6), true);
height += if i % 3 == 0 { 2 } else { 1 };
}
assert_eq!(w.measured_interval(), Some(6.0));
}
#[test]
fn backfilled_blocks_count_but_do_not_time() {
// The whole point: a node importing history at disk speed would
// otherwise produce an interval of milliseconds and a hashrate wrong by
// several orders of magnitude.
let mut w = RollingWindow::new(100);
for h in 1..=20 {
w.push(obs(h, 1, 0), false);
}
assert_eq!(w.len(), 20);
assert_eq!(w.measured_interval(), None);
}
#[test]
fn streaks_count_only_consecutive_heights() {
let mut w = RollingWindow::new(100);
for (h, m) in [(1u64, 1u8), (2, 1), (3, 1), (4, 2), (5, 1), (7, 1)] {
w.push(obs(h, m, h as i64), true);
}
let (t, _) = w.tally(100);
// 1,2,3 is a run of three; 5 and 7 are not adjacent, so no run of two.
assert_eq!(t[&miner(1)].best_streak, 3);
assert_eq!(t[&miner(1)].blocks, 5);
assert_eq!(t[&miner(2)].best_streak, 1);
}
#[test]
fn standings_rank_by_blocks_then_recency() {
let mut w = RollingWindow::new(100);
for (h, m) in [(1u64, 1u8), (2, 2), (3, 1), (4, 3), (5, 2)] {
w.push(obs(h, m, h as i64), true);
}
let (t, total) = w.tally(100);
let rows = leaderboard(&t, total, Some(1000.0), |m| {
(m.abbreviated(), AttributionSource::Preimage, 0.0)
});
assert_eq!(rows.len(), 3);
// 1 and 2 both have two blocks; 2's most recent is higher, so it leads.
assert_eq!(rows[0].miner, miner(2));
assert_eq!(rows[1].miner, miner(1));
assert_eq!(rows[2].miner, miner(3));
assert_eq!(rows[0].rank, 1);
assert!((rows[0].share - 0.4).abs() < 1e-9);
assert_eq!(rows[0].hashrate_estimate, Some(400.0));
}
#[test]
fn miners_sharing_a_node_name_are_told_apart() {
// One operator running two reward preimages from identically-named
// nodes is ordinary, and two rows both reading "rig-01" would leave a
// miner unable to find their own.
let mut w = RollingWindow::new(100);
for (h, m) in [(1u64, 1u8), (2, 2), (3, 1), (4, 3)] {
w.push(obs(h, m, h as i64), true);
}
let (t, total) = w.tally(100);
let rows = leaderboard(&t, total, None, |m| {
// Miners 1 and 2 share a name; miner 3 has its own.
let name = if m == &miner(3) { "solo" } else { "rig-01" };
(name.to_owned(), AttributionSource::Telemetry, 1.0)
});
let displays: Vec<&str> = rows.iter().map(|r| r.display.as_str()).collect();
assert_eq!(
displays
.iter()
.filter(|d| d.starts_with("rig-01 · "))
.count(),
2
);
// Short enough not to widen the table past its columns.
assert!(displays.iter().all(|d| d.len() <= "rig-01 · ".len() + 4));
// The unaffected row keeps its clean name — disambiguation is applied
// to the colliding group only.
assert!(displays.contains(&"solo"));
// Every display is unique.
let unique: std::collections::HashSet<_> = displays.iter().collect();
assert_eq!(unique.len(), displays.len());
}
#[test]
fn unique_names_are_left_alone() {
let mut w = RollingWindow::new(100);
w.push(obs(1, 1, 1), true);
let (t, total) = w.tally(100);
let rows = leaderboard(&t, total, None, |_| {
("alice".to_owned(), AttributionSource::Telemetry, 1.0)
});
assert_eq!(rows[0].display, "alice");
}
#[test]
fn an_empty_window_yields_no_rows_and_no_division_by_zero() {
let w = RollingWindow::new(100);
let (t, total) = w.tally(100);
assert!(w.is_empty());
let rows = leaderboard(&t, total, Some(1000.0), |m| {
(m.abbreviated(), AttributionSource::Preimage, 0.0)
});
assert!(rows.is_empty());
}
}

32
crates/blackbeard-data/Cargo.toml Normal file
View File

@@ -0,0 +1,32 @@
[package]
name = "blackbeard-data"
description = "Chain RPC, telemetry feed and Postgres access for blackbeard.observer."
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
authors.workspace = true
repository.workspace = true
[dependencies]
blackbeard-core.workspace = true
blackbeard-entities.workspace = true
bigdecimal.workspace = true
chrono.workspace = true
futures-util.workspace = true
primitive-types.workspace = true
reqwest.workspace = true
serde.workspace = true
serde_json.workspace = true
sqlx.workspace = true
thiserror.workspace = true
tokio.workspace = true
tokio-tungstenite.workspace = true
tracing.workspace = true
url.workspace = true
[dev-dependencies]
tokio = { workspace = true }
sqlx = { workspace = true }
chrono = { workspace = true }

73
crates/blackbeard-data/migrations/0001_init.sql Normal file
View File

@@ -0,0 +1,73 @@
-- Initial schema for blackbeard.observer.
--
-- Migrations are sequentially versioned and immutable once committed
-- (architecture/generic.md §5): correct a mistake with a new file, never by
-- editing this one, or the runner's checksum diverges and it refuses to start.
-- One row per chain the observer watches. Config in `config.toml` is the source
-- of truth for which chains exist; this table holds what the *node* told us
-- about them (genesis, token) plus the observer's own bookkeeping, so a restart
-- does not have to re-discover everything before it can serve a page.
create table chain (
id text primary key,
display_name text not null,
mainnet boolean not null,
genesis text,
token_symbol text,
token_decimals smallint,
target_block_time_seconds double precision not null,
first_observed_at timestamptz not null default now(),
last_observed_at timestamptz
);
-- Every block the observer has seen, with its author.
--
-- Keyed on (chain, height) rather than on the block hash, deliberately: this is
-- a proof-of-work chain and it reorgs. An upsert on this key means a block that
-- replaces another at the same height overwrites it, which is exactly the
-- semantics wanted — the leaderboard should reflect the canonical chain, not
-- the union of every fork the observer happened to witness.
create table block (
chain text not null references chain (id) on delete cascade,
height bigint not null,
hash text not null,
miner text not null,
-- The author's own timestamp, from the block's Timestamp::set inherent. Set
-- when the author BUILDS the proposal, so it measures proposal timing.
authored_at timestamptz,
-- When this observer first saw the block. The difference between the two is
-- propagation plus poll lag when negative, a future-dated block when
-- positive, and a withheld block when large and positive-going.
observed_at timestamptz not null,
-- Expected hashes to win this block. numeric, not bigint: difficulty is a
-- U512 and routinely past 1e30, so every integer type Postgres has would
-- overflow. numeric is arbitrary-precision and orders and sums correctly.
difficulty numeric,
primary key (chain, height)
);
-- The leaderboard's hot query: the last N blocks of a chain, newest first.
create index block_chain_height_desc_idx on block (chain, height desc);
-- The per-miner history query: one miner's blocks over a time range.
create index block_chain_miner_observed_idx on block (chain, miner, observed_at desc);
-- Held telemetry attributions, so a restart does not strip every author of its
-- name for a few blocks.
--
-- That churn is not cosmetic: a row whose display flips from a node name to a
-- raw preimage and back splits the miner's series on every deploy, which is
-- precisely the continuity this site exists to provide.
create table miner_attribution (
chain text not null references chain (id) on delete cascade,
miner text not null,
-- 'peer' for a libp2p peer id (stable across restarts and renames) or
-- 'node' for a telemetry node id (per-connection, the fallback).
node_kind text not null check (node_kind in ('peer', 'node')),
node_key text not null,
node_name text,
attempts integer not null default 0,
attributed integer not null default 0,
updated_at timestamptz not null default now(),
primary key (chain, miner)
);

56
crates/blackbeard-data/src/lib.rs Normal file
View File

@@ -0,0 +1,56 @@
//! Everything that talks to something outside this process: the chain's
//! JSON-RPC, the substrate-telemetry feed, and Postgres.
//!
//! The split is deliberate. `blackbeard-core` decodes and tallies with no I/O
//! at all and is exercised entirely by unit tests; this crate does the talking
//! and holds the retry, reconnect and schema concerns. `blackbeard-api` wires
//! the two together.
#![deny(missing_docs)]
pub mod rpc;
pub mod store;
pub mod telemetry;
/// Failures reaching a node, the telemetry feed, or the database.
#[derive(Debug, thiserror::Error)]
pub enum DataError {
/// The node answered with a JSON-RPC error, or an unusable result.
#[error("rpc {method}: {message}")]
Rpc {
/// Which method.
method: String,
/// What the node said.
message: String,
},
/// The HTTP request itself failed — connection refused, timeout, TLS.
#[error("http transport: {0}")]
Http(#[from] reqwest::Error),
/// A WebSocket connection failed or dropped.
///
/// Boxed: tungstenite's error is by far the largest variant here, and an
/// unboxed copy would be paid for on the stack by every `Result` in this
/// crate — including the RPC calls made several times a second.
#[error("websocket: {0}")]
WebSocket(#[from] Box<tokio_tungstenite::tungstenite::Error>),
/// A response did not have the shape we expected.
#[error("decoding a response: {0}")]
Decode(#[from] serde_json::Error),
/// A database query failed.
#[error("database: {0}")]
Database(#[from] sqlx::Error),
/// A migration failed to apply.
#[error("migration: {0}")]
Migration(#[from] sqlx::migrate::MigrateError),
}
impl From<tokio_tungstenite::tungstenite::Error> for DataError {
fn from(e: tokio_tungstenite::tungstenite::Error) -> Self {
DataError::WebSocket(Box::new(e))
}
}

321
crates/blackbeard-data/src/rpc.rs Normal file
View File

@@ -0,0 +1,321 @@
//! Talking to a Quantus node's JSON-RPC.
//!
//! Two transports on the **same** port, because `9944` serves HTTP and
//! WebSocket both:
//!
//! - [`RpcClient`] over HTTP for request/response — block hashes, bodies,
//! runtime calls. Stateless, so a node restart costs one failed request
//! rather than a reconnect dance.
//! - [`subscribe_new_heads`] over WebSocket for the head stream. The node
//! pushes, so the observer learns about a block the moment the node imports
//! it instead of up to a poll interval later — which is what lets the
//! browser's socket be genuinely live rather than merely frequent.
//!
//! No `subxt`. It would bring runtime metadata, a codec and a type registry to
//! read two fixed-shape fields the observer already decodes by hand
//! (`blackbeard-core::digest`), and it refuses plain `ws://` to anything but
//! localhost — which would mean either TLS or an ssh tunnel between the
//! observer and a node sitting on the same host.
use std::time::Duration;
use futures_util::{SinkExt, StreamExt};
use serde::Deserialize;
use serde_json::{Value, json};
use tokio::sync::mpsc;
use crate::DataError;
/// A Substrate block header, reduced to the fields the observer reads.
///
/// `#[serde(rename_all = "camelCase")]` matches the RPC's JSON. Unknown fields
/// are ignored rather than rejected — Quantus headers carry a `zkTreeRoot` the
/// upstream Substrate shape does not, and future fields must not break ingest.
#[derive(Debug, Clone, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct Header {
/// Block number as `0x`-prefixed hex.
pub number: String,
/// Parent block hash.
pub parent_hash: String,
/// Consensus digest logs. The `pow_` PreRuntime log in here carries the
/// author's reward preimage.
pub digest: Digest,
}
/// A header's digest logs.
#[derive(Debug, Clone, Deserialize)]
pub struct Digest {
/// Hex-encoded logs, in order.
pub logs: Vec<String>,
}
impl Header {
/// The block number, decoded from its hex form.
pub fn height(&self) -> Option<u64> {
u64::from_str_radix(self.number.strip_prefix("0x").unwrap_or(&self.number), 16).ok()
}
}
/// What `system_health` reports.
#[derive(Debug, Clone, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct Health {
/// True while the node is importing history rather than following the tip.
pub is_syncing: bool,
/// Connected peers.
pub peers: u32,
}
/// What `system_properties` reports.
#[derive(Debug, Clone, Default, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ChainProperties {
/// Token ticker, e.g. `PLK`.
pub token_symbol: Option<String>,
/// Decimal places in the smallest unit.
pub token_decimals: Option<u8>,
/// SS58 address prefix.
pub ss58_format: Option<u16>,
}
/// An HTTP JSON-RPC client for one node.
#[derive(Debug, Clone)]
pub struct RpcClient {
http: reqwest::Client,
url: String,
}
impl RpcClient {
/// Build a client for `url` (`http://host:9944`).
pub fn new(url: impl Into<String>, timeout: Duration) -> Result<Self, DataError> {
Ok(Self {
http: reqwest::Client::builder()
.timeout(timeout)
// The node is one host away and answers thousands of these per
// minute; without pooling every call would pay a fresh TCP
// handshake.
.pool_idle_timeout(Duration::from_secs(90))
.build()?,
url: url.into(),
})
}
/// Issue one JSON-RPC call.
pub async fn call(&self, method: &str, params: Value) -> Result<Value, DataError> {
let body = json!({"jsonrpc": "2.0", "id": 1, "method": method, "params": params});
let resp: Value = self
.http
.post(&self.url)
.json(&body)
.send()
.await?
.error_for_status()?
.json()
.await?;
if let Some(err) = resp.get("error") {
return Err(DataError::Rpc {
method: method.to_owned(),
message: err.to_string(),
});
}
resp.get("result").cloned().ok_or_else(|| DataError::Rpc {
method: method.to_owned(),
message: "response carried neither result nor error".into(),
})
}
/// `system_health`.
pub async fn health(&self) -> Result<Health, DataError> {
Ok(serde_json::from_value(
self.call("system_health", json!([])).await?,
)?)
}
/// `system_properties` — token symbol, decimals, SS58 prefix.
pub async fn properties(&self) -> Result<ChainProperties, DataError> {
Ok(serde_json::from_value(
self.call("system_properties", json!([])).await?,
)?)
}
/// The current best header, or the header at `hash`.
pub async fn header(&self, hash: Option<&str>) -> Result<Option<Header>, DataError> {
let params = match hash {
Some(h) => json!([h]),
None => json!([]),
};
let v = self.call("chain_getHeader", params).await?;
if v.is_null() {
return Ok(None);
}
Ok(Some(serde_json::from_value(v)?))
}
/// The block hash at `height`, if the node has that block.
pub async fn block_hash(&self, height: u64) -> Result<Option<String>, DataError> {
let v = self.call("chain_getBlockHash", json!([height])).await?;
Ok(v.as_str().map(str::to_owned))
}
/// The genesis hash. Discovered rather than configured — one fewer value to
/// get wrong when a chain launches or is respecced.
pub async fn genesis(&self) -> Result<Option<String>, DataError> {
self.block_hash(0).await
}
/// The author's timestamp for a block, in milliseconds.
///
/// `None` when the block's first extrinsic is not the timestamp inherent —
/// a shape we do not recognise, which costs this block its timing data and
/// nothing else.
pub async fn block_timestamp_ms(&self, hash: &str) -> Result<Option<u64>, DataError> {
let v = self.call("chain_getBlock", json!([hash])).await?;
let first = v
.pointer("/block/extrinsics/0")
.and_then(Value::as_str)
.map(str::to_owned);
Ok(first
.as_deref()
.and_then(blackbeard_core::scale::timestamp_inherent_ms))
}
/// Current mining difficulty: expected hashes to win a block.
pub async fn difficulty(&self) -> Result<primitive_types::U512, DataError> {
self.u512_runtime_call("QPoWApi_get_difficulty").await
}
/// The ceiling difficulty can reach.
pub async fn max_difficulty(&self) -> Result<primitive_types::U512, DataError> {
self.u512_runtime_call("QPoWApi_get_max_difficulty").await
}
async fn u512_runtime_call(&self, api: &str) -> Result<primitive_types::U512, DataError> {
let v = self.call("state_call", json!([api, "0x"])).await?;
let hex = v.as_str().ok_or_else(|| DataError::Rpc {
method: api.to_owned(),
message: "runtime call did not return a hex string".into(),
})?;
blackbeard_core::digest::u512_le(hex).ok_or_else(|| DataError::Rpc {
method: api.to_owned(),
message: format!("`{hex}` is not a little-endian U512"),
})
}
}
/// How long to wait between reconnection attempts on the head subscription.
const RECONNECT_DELAY: Duration = Duration::from_secs(5);
/// Follow a node's new heads over WebSocket, forwarding each to `sink`.
///
/// Runs until the channel closes. Reconnects on its own: a node restart, a
/// dropped mesh link or a proxy timeout are ordinary operating conditions here,
/// not reasons to stop watching a chain — and a chain configured before it
/// launches will spend its first weeks failing to connect on every attempt,
/// which must stay quiet rather than filling the journal.
///
/// **Heads are not a complete block record.** `chain_subscribeNewHeads` reports
/// the *best* head and skips intermediate blocks when several import at once,
/// so the caller must fill gaps against `chain_getBlockHash`. It is a liveness
/// signal, not a ledger.
pub async fn subscribe_new_heads(
ws_url: String,
sink: mpsc::Sender<Header>,
) -> Result<(), DataError> {
let mut backoff_logged = false;
loop {
if sink.is_closed() {
return Ok(());
}
match follow_once(&ws_url, &sink).await {
Ok(()) => {
tracing::info!(url = %ws_url, "head subscription closed cleanly, reconnecting");
backoff_logged = false;
}
Err(e) => {
// First failure at warn, the rest at debug. A chain that has not
// launched fails every five seconds forever; logging each at warn
// would bury everything else in the journal.
if backoff_logged {
tracing::debug!(url = %ws_url, error = %e, "head subscription still down");
} else {
tracing::warn!(url = %ws_url, error = %e, "head subscription lost");
backoff_logged = true;
}
}
}
tokio::time::sleep(RECONNECT_DELAY).await;
}
}
async fn follow_once(ws_url: &str, sink: &mpsc::Sender<Header>) -> Result<(), DataError> {
let (mut socket, _) = tokio_tungstenite::connect_async(ws_url).await?;
socket
.send(tokio_tungstenite::tungstenite::Message::Text(
json!({"jsonrpc": "2.0", "id": 1, "method": "chain_subscribeNewHeads", "params": []})
.to_string(),
))
.await?;
while let Some(msg) = socket.next().await {
let msg = msg?;
// Text and binary both accepted. Substrate sends text, but the
// telemetry feed on this same fleet sends binary (see
// `telemetry::follow_once`), and a client that silently drops the wrong
// frame type looks perfectly healthy while receiving nothing.
let payload = match msg {
tokio_tungstenite::tungstenite::Message::Text(t) => t.as_bytes().to_vec(),
tokio_tungstenite::tungstenite::Message::Binary(b) => b.to_vec(),
// The library answers pings itself; close ends the loop and the
// caller reconnects.
tokio_tungstenite::tungstenite::Message::Close(_) => break,
_ => continue,
};
let v: Value = match serde_json::from_slice(&payload) {
Ok(v) => v,
Err(e) => {
tracing::debug!(error = %e, "unparsable frame on the head subscription");
continue;
}
};
// The subscription confirmation carries `result` as a bare id; only
// notifications carry `params.result`.
let Some(result) = v.pointer("/params/result") else {
continue;
};
match serde_json::from_value::<Header>(result.clone()) {
Ok(header) => {
if sink.send(header).await.is_err() {
return Ok(());
}
}
Err(e) => tracing::warn!(error = %e, "head notification did not decode as a header"),
}
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn header_height_decodes_hex() {
let h: Header = serde_json::from_value(json!({
"number": "0xfe2a8",
"parentHash": "0x87f4",
"stateRoot": "0x3398",
"zkTreeRoot": "0xd382",
"digest": {"logs": ["0x06706f775f80aa"]}
}))
.expect("a Quantus header carries fields upstream Substrate does not; ignore them");
assert_eq!(h.height(), Some(1_041_064));
assert_eq!(h.digest.logs.len(), 1);
}
#[test]
fn a_header_missing_its_digest_is_a_decode_error_not_a_panic() {
let r = serde_json::from_value::<Header>(json!({"number": "0x1", "parentHash": "0x0"}));
assert!(r.is_err());
}
}

473
crates/blackbeard-data/src/store.rs Normal file
View File

@@ -0,0 +1,473 @@
//! Postgres access.
//!
//! ## Why there is a database at all
//!
//! The rolling window in `blackbeard-core` is enough to render a leaderboard,
//! and the arena exporter this project descends from needs nothing more. But
//! this site promises miners they can *track* their effort, and a window that
//! empties on every restart cannot answer "how did I do last week" or draw a
//! hashrate line older than the process. Blocks therefore land in Postgres as
//! they are observed, and the window is rebuilt from it at startup.
//!
//! ## Connection
//!
//! mTLS, passwordless, per `architecture/generic.md` §5 and §11: the host's own
//! certificate is the credential and `pg_ident.conf` maps its CN to the role.
//! **No password is accepted anywhere in this module** — if a connection string
//! ever needs one, something is wrong upstream of here.
//!
//! Host certificates are issued with a **24-hour** expiry and renewed
//! continuously, which is why the pool caps connection lifetime well below that:
//! Postgres does not re-validate a certificate mid-session, so a long-lived
//! connection would keep working on an expired cert until it dropped — and then
//! reconnect, at an arbitrary moment, with whatever is on disk. Recycling on a
//! schedule makes that transition routine instead of a surprise.
use std::time::Duration;
use bigdecimal::BigDecimal;
use blackbeard_core::attribution::NodeKey;
use blackbeard_core::window::Observed;
use blackbeard_entities::{ChainId, MinerId};
use chrono::{DateTime, Utc};
use sqlx::PgPool;
use sqlx::postgres::{PgConnectOptions, PgPoolOptions, PgSslMode};
use crate::DataError;
/// How the store reaches Postgres.
#[derive(Debug, Clone)]
pub struct StoreConfig {
/// Server hostname. Verified against the server certificate, so it must be
/// the name in that certificate, not an address.
pub host: String,
/// Server port.
pub port: u16,
/// Database name.
pub database: String,
/// Role to connect as. Resolved from the client certificate's CN by
/// `pg_ident.conf` on the server; it is named here so a host mapped to more
/// than one role can say which it wants.
pub username: String,
/// Internal root CA bundle, for verifying the server.
pub root_cert: std::path::PathBuf,
/// This host's certificate — the credential.
pub client_cert: std::path::PathBuf,
/// This host's private key.
pub client_key: std::path::PathBuf,
/// Maximum pooled connections.
pub max_connections: u32,
}
/// A connected Postgres store.
#[derive(Debug, Clone)]
pub struct Store {
pool: PgPool,
}
/// A block on its way into the database.
#[derive(Debug, Clone)]
pub struct BlockRecord {
/// Which chain.
pub chain: ChainId,
/// Block number.
pub height: u64,
/// Block hash.
pub hash: String,
/// Author's reward preimage.
pub miner: MinerId,
/// The author's own timestamp.
pub authored_at: Option<DateTime<Utc>>,
/// When this observer first saw it.
pub observed_at: DateTime<Utc>,
/// Difficulty in force, as a decimal string.
pub difficulty: Option<String>,
}
/// What the observer knows about a chain, for the `chain` table.
#[derive(Debug, Clone)]
pub struct ChainRecord {
/// Operator name for the chain.
pub chain: ChainId,
/// Human-facing name.
pub display_name: String,
/// True for the production network.
pub mainnet: bool,
/// Genesis hash, once the node has told us.
pub genesis: Option<String>,
/// Token ticker.
pub token_symbol: Option<String>,
/// Token decimals.
pub token_decimals: Option<i16>,
/// Configured target seconds per block.
pub target_block_time: f64,
}
/// A held attribution loaded back from the database.
#[derive(Debug, Clone)]
pub struct StoredAttribution {
/// Which miner.
pub miner: MinerId,
/// The node it is attributed to.
pub key: NodeKey,
/// That node's name when it was stored, so a name resolves before the feed
/// has re-announced the node.
pub name: Option<String>,
/// Blocks checked.
pub attempts: u32,
/// Blocks that produced a usable vote.
pub attributed: u32,
}
/// Connections are recycled well inside the 24-hour host-certificate lifetime,
/// so certificate rotation is a routine reconnect rather than a surprise
/// mid-session failure.
const MAX_CONNECTION_LIFETIME: Duration = Duration::from_secs(60 * 60);
impl Store {
/// Connect and run migrations.
pub async fn connect(config: &StoreConfig) -> Result<Self, DataError> {
let options = PgConnectOptions::new()
.host(&config.host)
.port(config.port)
.database(&config.database)
.username(&config.username)
// verify-full, not verify-ca: verify-ca proves the server holds a
// certificate from our CA but not that it is the server we asked
// for, which any other host on the mesh also satisfies.
.ssl_mode(PgSslMode::VerifyFull)
.ssl_root_cert(&config.root_cert)
.ssl_client_cert(&config.client_cert)
.ssl_client_key(&config.client_key);
let pool = PgPoolOptions::new()
.max_connections(config.max_connections)
.acquire_timeout(Duration::from_secs(10))
.max_lifetime(MAX_CONNECTION_LIFETIME)
.connect_with(options)
.await?;
sqlx::migrate!("./migrations").run(&pool).await?;
Ok(Self { pool })
}
/// Wrap an already-built pool. Used by tests and by the CLI, which may be
/// pointed at a throwaway database.
pub fn from_pool(pool: PgPool) -> Self {
Self { pool }
}
/// Run migrations against an existing pool.
pub async fn migrate(&self) -> Result<(), DataError> {
sqlx::migrate!("./migrations").run(&self.pool).await?;
Ok(())
}
/// Record what the node told us about a chain.
///
/// Called on every successful poll, so it must be an upsert: the row exists
/// from the first poll onward, and genesis and token properties are
/// discovered rather than configured.
pub async fn upsert_chain(&self, record: &ChainRecord) -> Result<(), DataError> {
let ChainRecord {
chain,
display_name,
mainnet,
genesis,
token_symbol,
token_decimals,
target_block_time,
} = record;
sqlx::query!(
r#"
insert into chain (
id, display_name, mainnet, genesis, token_symbol, token_decimals,
target_block_time_seconds, last_observed_at
)
values ($1, $2, $3, $4, $5, $6, $7, now())
on conflict (id) do update set
display_name = excluded.display_name,
mainnet = excluded.mainnet,
-- coalesce, not excluded: a poll that could not reach the node
-- must not blank out what an earlier one discovered.
genesis = coalesce(excluded.genesis, chain.genesis),
token_symbol = coalesce(excluded.token_symbol, chain.token_symbol),
token_decimals = coalesce(excluded.token_decimals, chain.token_decimals),
target_block_time_seconds = excluded.target_block_time_seconds,
last_observed_at = now()
"#,
chain.as_str(),
display_name,
mainnet,
genesis.as_deref(),
token_symbol.as_deref(),
*token_decimals,
*target_block_time,
)
.execute(&self.pool)
.await?;
Ok(())
}
/// Insert or replace a batch of blocks.
///
/// One statement with array parameters rather than a loop: a startup
/// backfill inserts tens of thousands of rows, and a round trip each would
/// take minutes. The `on conflict` is what makes a reorg correct — a block
/// replacing another at the same height overwrites it.
pub async fn record_blocks(&self, blocks: &[BlockRecord]) -> Result<u64, DataError> {
if blocks.is_empty() {
return Ok(0);
}
let chains: Vec<String> = blocks.iter().map(|b| b.chain.0.clone()).collect();
let heights: Vec<i64> = blocks.iter().map(|b| b.height as i64).collect();
let hashes: Vec<String> = blocks.iter().map(|b| b.hash.clone()).collect();
let miners: Vec<String> = blocks.iter().map(|b| b.miner.0.clone()).collect();
let authored: Vec<Option<DateTime<Utc>>> = blocks.iter().map(|b| b.authored_at).collect();
let observed: Vec<DateTime<Utc>> = blocks.iter().map(|b| b.observed_at).collect();
let difficulties: Vec<Option<BigDecimal>> = blocks
.iter()
.map(|b| b.difficulty.as_deref().and_then(|d| d.parse().ok()))
.collect();
let result = sqlx::query!(
r#"
insert into block (chain, height, hash, miner, authored_at, observed_at, difficulty)
select * from unnest(
$1::text[], $2::bigint[], $3::text[], $4::text[],
$5::timestamptz[], $6::timestamptz[], $7::numeric[]
)
on conflict (chain, height) do update set
hash = excluded.hash,
miner = excluded.miner,
authored_at = excluded.authored_at,
observed_at = excluded.observed_at,
difficulty = excluded.difficulty
"#,
&chains,
&heights,
&hashes,
&miners,
&authored as &[Option<DateTime<Utc>>],
&observed,
&difficulties as &[Option<BigDecimal>],
)
.execute(&self.pool)
.await?;
Ok(result.rows_affected())
}
/// The highest block recorded for a chain, if any.
pub async fn max_height(&self, chain: &ChainId) -> Result<Option<u64>, DataError> {
let row = sqlx::query!(
"select max(height) as height from block where chain = $1",
chain.as_str()
)
.fetch_one(&self.pool)
.await?;
Ok(row.height.map(|h| h as u64))
}
/// The last `limit` blocks of a chain, **oldest first** — the order
/// `RollingWindow::push` expects.
pub async fn recent_blocks(
&self,
chain: &ChainId,
limit: i64,
) -> Result<Vec<Observed>, DataError> {
let rows = sqlx::query!(
r#"
select height, miner, observed_at
from (
select height, miner, observed_at
from block
where chain = $1
order by height desc
limit $2
) recent
order by height asc
"#,
chain.as_str(),
limit,
)
.fetch_all(&self.pool)
.await?;
Ok(rows
.into_iter()
.map(|r| Observed {
height: r.height as u64,
miner: MinerId(r.miner),
observed_at: r.observed_at,
})
.collect())
}
/// Totals for one miner across everything the observer has recorded.
///
/// `cumulative_work` is the sum of the difficulty of every block the miner
/// won: the expected number of hashes it took. That is the honest
/// cumulative measure — a block count flatters miners who were around when
/// difficulty was low, and this does not.
pub async fn miner_totals(
&self,
chain: &ChainId,
miner: &MinerId,
) -> Result<(u64, Option<DateTime<Utc>>, Option<DateTime<Utc>>, String), DataError> {
let row = sqlx::query!(
r#"
select count(*) as "blocks!",
min(observed_at) as first_seen,
max(observed_at) as last_seen,
coalesce(sum(difficulty), 0)::text as "work!"
from block
where chain = $1 and miner = $2
"#,
chain.as_str(),
miner.as_str(),
)
.fetch_one(&self.pool)
.await?;
Ok((row.blocks as u64, row.first_seen, row.last_seen, row.work))
}
/// A miner's blocks bucketed over time, alongside the total blocks in each
/// bucket so a share can be computed.
///
/// Buckets come from `date_bin`, which anchors them to a fixed epoch rather
/// than to the range's start — so the same bucket boundaries fall in the
/// same places on every request, and a chart does not shift under the
/// reader as time passes.
pub async fn miner_series(
&self,
chain: &ChainId,
miner: &MinerId,
since: DateTime<Utc>,
bucket: Duration,
) -> Result<Vec<(DateTime<Utc>, u32, u32)>, DataError> {
let interval = sqlx::postgres::types::PgInterval {
months: 0,
days: 0,
microseconds: bucket.as_micros() as i64,
};
let rows = sqlx::query!(
r#"
select date_bin($3, observed_at, timestamptz 'epoch') as "bucket!",
count(*) filter (where miner = $2) as "mine!",
count(*) as "total!"
from block
where chain = $1 and observed_at >= $4
group by 1
order by 1 asc
"#,
chain.as_str(),
miner.as_str(),
interval,
since,
)
.fetch_all(&self.pool)
.await?;
Ok(rows
.into_iter()
.map(|r| (r.bucket, r.mine as u32, r.total as u32))
.collect())
}
/// Persist the attributions currently held.
pub async fn save_attributions(
&self,
chain: &ChainId,
held: &[(MinerId, NodeKey, u32, u32, Option<String>)],
) -> Result<(), DataError> {
if held.is_empty() {
return Ok(());
}
let miners: Vec<String> = held.iter().map(|h| h.0.0.clone()).collect();
let (kinds, keys): (Vec<String>, Vec<String>) = held
.iter()
.map(|h| match &h.1 {
NodeKey::Peer(p) => ("peer".to_owned(), p.clone()),
NodeKey::Node(n) => ("node".to_owned(), n.to_string()),
})
.unzip();
let attempts: Vec<i32> = held.iter().map(|h| h.2 as i32).collect();
let attributed: Vec<i32> = held.iter().map(|h| h.3 as i32).collect();
let names: Vec<Option<String>> = held.iter().map(|h| h.4.clone()).collect();
sqlx::query!(
r#"
insert into miner_attribution
(chain, miner, node_kind, node_key, attempts, attributed, node_name, updated_at)
select $1, m, k, key, a, att, n, now()
from unnest($2::text[], $3::text[], $4::text[], $5::int[], $6::int[], $7::text[])
as t(m, k, key, a, att, n)
on conflict (chain, miner) do update set
node_kind = excluded.node_kind,
node_key = excluded.node_key,
attempts = excluded.attempts,
attributed = excluded.attributed,
-- A name the feed cannot currently resolve must not erase the
-- one we already had: a telemetry hiccup would otherwise strip
-- every miner's name from the next restart onwards.
node_name = coalesce(excluded.node_name, miner_attribution.node_name),
updated_at = now()
"#,
chain.as_str(),
&miners,
&kinds,
&keys,
&attempts,
&attributed,
&names as &[Option<String>],
)
.execute(&self.pool)
.await?;
Ok(())
}
/// Load held attributions for a chain.
pub async fn load_attributions(
&self,
chain: &ChainId,
) -> Result<Vec<StoredAttribution>, DataError> {
let rows = sqlx::query!(
r#"
select miner, node_kind, node_key, node_name, attempts, attributed
from miner_attribution
where chain = $1
"#,
chain.as_str(),
)
.fetch_all(&self.pool)
.await?;
Ok(rows
.into_iter()
.filter_map(|r| {
let key = match r.node_kind.as_str() {
"peer" => NodeKey::Peer(r.node_key),
// A node id that no longer parses is a row written by an
// older shape; dropping it costs one miner its name until
// the next few blocks re-earn it.
"node" => NodeKey::Node(r.node_key.parse().ok()?),
_ => return None,
};
Some(StoredAttribution {
miner: MinerId(r.miner),
key,
name: r.node_name,
attempts: r.attempts.max(0) as u32,
attributed: r.attributed.max(0) as u32,
})
})
.collect())
}
/// A cheap liveness probe for the health endpoint.
pub async fn ping(&self) -> Result<(), DataError> {
sqlx::query!("select 1 as ok").fetch_one(&self.pool).await?;
Ok(())
}
}

465
crates/blackbeard-data/src/telemetry.rs Normal file
View File

@@ -0,0 +1,465 @@
//! The substrate-telemetry feed.
//!
//! Two things the chain itself cannot tell us come from here.
//!
//! **The node population.** A miner too small to win a block in a given window
//! is invisible in the authorship leaderboard but still present on the feed.
//! Node count holding flat while distinct authors falls is the signature of
//! small miners still running and simply being out-hashed — the difference
//! between "people are giving up" and "people are losing", which a leaderboard
//! alone cannot distinguish.
//!
//! **Names.** The feed reports block imports with a propagation time, stamping
//! the first reporter of a hash with `0`. A node imports its own block before
//! announcing it, so the first reporter is usually the author — enough, over
//! many blocks, to put a name to a preimage. The voting that turns "usually"
//! into something safe to display lives in `blackbeard-core::attribution`; this
//! module only supplies the evidence.
//!
//! Everything here is **optional**. A chain with no telemetry URL configured, or
//! a feed that is down, costs the site node counts and miner names and nothing
//! else — the leaderboard, hashrate and block ticker all come from the chain.
//!
//! ## Protocol
//!
//! Connect, send `subscribe:<genesis hash>`, then receive JSON arrays of
//! alternating `[action_code, payload, action_code, payload, …]`. The codes are
//! substrate-telemetry's `FeedMessage` discriminants; the four that matter are
//! below. Payload shapes are positional arrays with no field names, which is
//! why every access here is bounds-checked and a malformed payload increments a
//! counter instead of taking the feed down.
use std::collections::{HashMap, VecDeque};
use std::sync::{Arc, RwLock};
use std::time::Duration;
use blackbeard_core::attribution::NodeKey;
use futures_util::{SinkExt, StreamExt};
use serde_json::Value;
use crate::DataError;
/// `AddedNode`: `[node_id, [name, implementation, version, …, network_id, os, …, hardware]]`
const ACTION_ADDED_NODE: u64 = 3;
/// `RemovedNode`: `node_id`
const ACTION_REMOVED_NODE: u64 = 4;
/// `ImportedBlock`: `[node_id, [height, hash, block_time, timestamp, propagation_ms]]`
const ACTION_IMPORTED_BLOCK: u64 = 6;
/// `AddedChain`: `[name, genesis, node_count]`
const ACTION_ADDED_CHAIN: u64 = 11;
/// Block hashes retained for attribution.
///
/// At a 6 s target this is roughly the last six hours — far longer than the few
/// seconds the attribution join actually needs, but cheap, and it means a stalled
/// ingest loop catching up does not find every block already evicted.
const MAX_TRACKED_IMPORTS: usize = 4_000;
/// Delay between reconnection attempts.
const RECONNECT_DELAY: Duration = Duration::from_secs(15);
#[derive(Debug, Default)]
struct ImportRecord {
first: Option<u64>,
/// Smallest positive propagation time seen from any *later* reporter: the
/// first reporter's lead. `None` until a second node reports the block.
second_ms: Option<i64>,
}
#[derive(Debug, Default)]
struct State {
connected: bool,
errors: u64,
/// Authoritative count for the chain, from `AddedChain`. Nodes we have
/// individually seen is a different (and smaller) number — we only learn
/// about nodes added while connected.
chain_node_count: Option<u32>,
/// Node id to `(name, peer id)`. Node ids are per-connection; peer ids are
/// not, which is why both are kept.
names: HashMap<u64, (String, String)>,
/// Peer id to its most recent name. Survives reconnects and renames, so an
/// attribution made before a feed hiccup still renders a name after it.
peer_names: HashMap<String, String>,
imports: HashMap<String, ImportRecord>,
import_order: VecDeque<String>,
}
/// A live view of one chain's telemetry feed.
///
/// Cheap to clone — every clone shares the same state — so the ingest loop and
/// the HTTP handlers can each hold one.
#[derive(Debug, Clone, Default)]
pub struct TelemetryFeed {
state: Arc<RwLock<State>>,
}
impl TelemetryFeed {
/// An unconnected feed. Reports nothing; every accessor degrades to `None`
/// or `false`. This is what a chain with no telemetry URL gets.
pub fn new() -> Self {
Self::default()
}
/// Whether the feed is currently connected.
pub fn connected(&self) -> bool {
self.read().connected
}
/// Nodes on this chain, per the feed's own count.
pub fn node_count(&self) -> Option<u32> {
self.read().chain_node_count
}
/// Malformed payloads seen since start. A feed that is connected but whose
/// error count climbs means substrate-telemetry changed a payload shape.
pub fn decode_errors(&self) -> u64 {
self.read().errors
}
/// Who reported a block first, and by how much they led the second
/// reporter.
///
/// `None` when the block never appeared on the feed. `Some((key, None))`
/// when exactly one node has reported it so far — a lead cannot be computed
/// from a single report, and treating "nobody else has spoken yet" as an
/// infinite lead would attribute a block to whichever node happened to be
/// polled first.
pub fn first_import(&self, block_hash: &str) -> Option<(NodeKey, Option<i64>)> {
let state = self.read();
let record = state.imports.get(block_hash)?;
let node_id = record.first?;
Some((state.node_key(node_id), record.second_ms))
}
/// The current name for a node key, if the feed has ever named it.
pub fn name_of(&self, key: &NodeKey) -> Option<String> {
let state = self.read();
let name = match key {
NodeKey::Peer(peer) => state.peer_names.get(peer).cloned(),
NodeKey::Node(id) => state.names.get(id).map(|(n, _)| n.clone()),
}?;
(!name.is_empty()).then_some(name)
}
/// Seed the peer-id-to-name map from persisted state, so names restored
/// from the database resolve before the feed has re-announced their nodes.
pub fn seed_name(&self, peer_id: String, name: String) {
if peer_id.is_empty() || name.is_empty() {
return;
}
self.write().peer_names.entry(peer_id).or_insert(name);
}
/// Follow the feed for one chain until `genesis`'s subscription is dropped.
///
/// Reconnects indefinitely. `genesis` is discovered from the node rather
/// than configured, so a chain that respecs before launch needs no change
/// here.
pub async fn run(self, url: String, genesis: String) {
let mut backoff_logged = false;
loop {
match self.follow_once(&url, &genesis).await {
Ok(()) => {
tracing::info!(%url, "telemetry feed closed, reconnecting");
backoff_logged = false;
}
Err(e) => {
if backoff_logged {
tracing::debug!(%url, error = %e, "telemetry feed still down");
} else {
tracing::warn!(%url, error = %e, "telemetry feed lost");
backoff_logged = true;
}
}
}
{
let mut state = self.write();
state.connected = false;
// Node identities are per-connection: keeping them across a
// reconnect would report a population that no longer exists.
// Names are deliberately kept — an attribution made before the
// hiccup must still render.
state.chain_node_count = None;
}
tokio::time::sleep(RECONNECT_DELAY).await;
}
}
async fn follow_once(&self, url: &str, genesis: &str) -> Result<(), DataError> {
let (mut socket, _) = tokio_tungstenite::connect_async(url).await?;
socket
.send(tokio_tungstenite::tungstenite::Message::Text(format!(
"subscribe:{genesis}"
)))
.await?;
self.write().connected = true;
tracing::info!(%url, genesis = &genesis[..genesis.len().min(18)], "telemetry subscribed");
while let Some(msg) = socket.next().await {
// substrate-telemetry sends its JSON in **binary** frames, not text
// ones. This cost an afternoon: a client that handles only
// `Message::Text` connects, subscribes, reports itself healthy and
// silently receives nothing at all — no error, no decode failure,
// just an empty feed. (Python's `json.loads` accepts bytes, so a
// throwaway probe in Python works and hides the difference.) Both
// are accepted here; the payload is identical either way.
let payload = match msg? {
tokio_tungstenite::tungstenite::Message::Text(t) => t.as_bytes().to_vec(),
tokio_tungstenite::tungstenite::Message::Binary(b) => b.to_vec(),
tokio_tungstenite::tungstenite::Message::Close(_) => break,
_ => continue,
};
let Ok(Value::Array(items)) = serde_json::from_slice::<Value>(&payload) else {
continue;
};
self.ingest(&items);
}
Ok(())
}
/// Apply one feed frame: alternating action codes and payloads.
fn ingest(&self, items: &[Value]) {
let mut state = self.write();
for pair in items.chunks_exact(2) {
let (Some(code), payload) = (pair[0].as_u64(), &pair[1]) else {
continue;
};
match code {
ACTION_ADDED_NODE => state.added_node(payload),
ACTION_REMOVED_NODE => {
if let Some(id) = payload.as_u64() {
state.names.remove(&id);
}
}
ACTION_IMPORTED_BLOCK => state.imported_block(payload),
ACTION_ADDED_CHAIN => {
if let Some(count) = payload.get(2).and_then(Value::as_u64) {
state.chain_node_count = Some(count as u32);
}
}
_ => {}
}
}
}
fn read(&self) -> std::sync::RwLockReadGuard<'_, State> {
// A poisoned lock means a panic while holding it. Nothing in this
// module can panic under the lock (every access is bounds-checked), and
// taking the feed down over telemetry — which is optional by design —
// would be a worse outcome than continuing with the state as it stands.
self.state.read().unwrap_or_else(|e| e.into_inner())
}
fn write(&self) -> std::sync::RwLockWriteGuard<'_, State> {
self.state.write().unwrap_or_else(|e| e.into_inner())
}
}
impl State {
/// The stable identity to vote with: the peer id when the feed gave one,
/// else the per-connection node id.
fn node_key(&self, node_id: u64) -> NodeKey {
match self.names.get(&node_id) {
Some((_, peer)) if !peer.is_empty() => NodeKey::Peer(peer.clone()),
_ => NodeKey::Node(node_id),
}
}
fn added_node(&mut self, payload: &Value) {
let (Some(id), Some(details)) = (
payload.get(0).and_then(Value::as_u64),
payload.get(1).and_then(Value::as_array),
) else {
self.errors += 1;
return;
};
let field = |i: usize| {
details
.get(i)
.and_then(Value::as_str)
.unwrap_or_default()
.to_owned()
};
let name = field(0);
let peer = field(4);
if !peer.is_empty() && !name.is_empty() {
self.peer_names.insert(peer.clone(), name.clone());
}
self.names.insert(id, (name, peer));
}
fn imported_block(&mut self, payload: &Value) {
let (Some(node_id), Some(block)) = (
payload.get(0).and_then(Value::as_u64),
payload.get(1).and_then(Value::as_array),
) else {
self.errors += 1;
return;
};
let Some(hash) = block.get(1).and_then(Value::as_str) else {
self.errors += 1;
return;
};
// Propagation time relative to the first reporter. `null` on the very
// first report of a hash from some feed versions, `0` on others.
let propagation = block.get(4).and_then(Value::as_i64);
if !self.imports.contains_key(hash) {
if self.import_order.len() == MAX_TRACKED_IMPORTS
&& let Some(oldest) = self.import_order.pop_front()
{
self.imports.remove(&oldest);
}
self.import_order.push_back(hash.to_owned());
self.imports
.insert(hash.to_owned(), ImportRecord::default());
}
let Some(record) = self.imports.get_mut(hash) else {
return;
};
match propagation {
// A later reporter: its delay is the first reporter's lead. Keep
// the smallest, which is the lead over the *closest* competitor —
// the conservative figure, and the one that correctly refuses to
// credit an author whose blocks another node also sees instantly.
Some(ms) if ms > 0 => {
record.second_ms = Some(record.second_ms.map_or(ms, |cur| cur.min(ms)));
}
// Zero or absent: a first reporter. The second such report for a
// hash is a tie, and a tie means nobody led.
_ => match record.first {
None => record.first = Some(node_id),
Some(existing) if existing != node_id => record.second_ms = Some(0),
_ => {}
},
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
fn added_node(id: u64, name: &str, peer: &str) -> Vec<Value> {
vec![
json!(ACTION_ADDED_NODE),
json!([id, [name, "quantus-node", "0.11.1", null, peer, "linux"]]),
]
}
fn imported(node: u64, hash: &str, propagation: Option<i64>) -> Vec<Value> {
vec![
json!(ACTION_IMPORTED_BLOCK),
json!([
node,
[1_041_064, hash, 6000, 1_757_000_000_000i64, propagation]
]),
]
}
#[test]
fn the_first_reporter_leads_by_the_second_reporters_delay() {
let feed = TelemetryFeed::new();
feed.ingest(&added_node(1, "alice", "12D3KooWalice"));
feed.ingest(&added_node(2, "bob", "12D3KooWbob"));
feed.ingest(&imported(1, "0xdead", Some(0)));
feed.ingest(&imported(2, "0xdead", Some(85)));
assert_eq!(
feed.first_import("0xdead"),
Some((NodeKey::Peer("12D3KooWalice".into()), Some(85)))
);
assert_eq!(
feed.name_of(&NodeKey::Peer("12D3KooWalice".into()))
.as_deref(),
Some("alice")
);
}
#[test]
fn a_single_report_yields_no_lead() {
// Nobody else has spoken yet. Treating that as an infinite lead would
// hand the block to whichever node the feed happened to mention first.
let feed = TelemetryFeed::new();
feed.ingest(&added_node(1, "alice", "p1"));
feed.ingest(&imported(1, "0xbeef", Some(0)));
assert_eq!(
feed.first_import("0xbeef"),
Some((NodeKey::Peer("p1".into()), None))
);
}
#[test]
fn two_nodes_reporting_first_is_a_tie_with_no_lead() {
let feed = TelemetryFeed::new();
feed.ingest(&added_node(1, "alice", "p1"));
feed.ingest(&added_node(2, "bob", "p2"));
feed.ingest(&imported(1, "0xcafe", Some(0)));
feed.ingest(&imported(2, "0xcafe", Some(0)));
assert_eq!(feed.first_import("0xcafe").unwrap().1, Some(0));
}
#[test]
fn the_closest_competitor_sets_the_lead() {
let feed = TelemetryFeed::new();
feed.ingest(&added_node(1, "alice", "p1"));
feed.ingest(&imported(1, "0x01", Some(0)));
feed.ingest(&imported(2, "0x01", Some(300)));
feed.ingest(&imported(3, "0x01", Some(40)));
assert_eq!(feed.first_import("0x01").unwrap().1, Some(40));
}
#[test]
fn a_node_with_no_peer_id_falls_back_to_its_node_id() {
let feed = TelemetryFeed::new();
feed.ingest(&added_node(7, "carol", ""));
feed.ingest(&imported(7, "0x02", Some(0)));
assert_eq!(feed.first_import("0x02").unwrap().0, NodeKey::Node(7));
}
#[test]
fn the_chain_node_count_comes_from_the_feed() {
let feed = TelemetryFeed::new();
feed.ingest(&[json!(ACTION_ADDED_CHAIN), json!(["Planck", "0x4901", 142])]);
assert_eq!(feed.node_count(), Some(142));
}
#[test]
fn a_malformed_payload_is_counted_not_fatal() {
let feed = TelemetryFeed::new();
feed.ingest(&[json!(ACTION_ADDED_NODE), json!("not an array")]);
feed.ingest(&[json!(ACTION_IMPORTED_BLOCK), json!([1])]);
assert_eq!(feed.decode_errors(), 2);
// Still usable afterwards.
feed.ingest(&added_node(1, "alice", "p1"));
assert_eq!(
feed.name_of(&NodeKey::Peer("p1".into())).as_deref(),
Some("alice")
);
}
#[test]
fn tracked_imports_are_bounded() {
let feed = TelemetryFeed::new();
for i in 0..MAX_TRACKED_IMPORTS + 100 {
feed.ingest(&imported(1, &format!("0x{i:x}"), Some(0)));
}
assert_eq!(feed.read().imports.len(), MAX_TRACKED_IMPORTS);
// The oldest are gone, the newest are kept.
assert!(feed.first_import("0x0").is_none());
assert!(
feed.first_import(&format!("0x{:x}", MAX_TRACKED_IMPORTS + 50))
.is_some()
);
}
#[test]
fn an_unconnected_feed_reports_nothing_rather_than_failing() {
let feed = TelemetryFeed::new();
assert!(!feed.connected());
assert_eq!(feed.node_count(), None);
assert_eq!(feed.first_import("0xanything"), None);
assert_eq!(feed.name_of(&NodeKey::Node(1)), None);
}
}

347
crates/blackbeard-data/tests/store.rs Normal file
View File

@@ -0,0 +1,347 @@
//! Integration tests against a real Postgres.
//!
//! Skipped unless `TEST_DATABASE_URL` is set, so `cargo test` stays useful on a
//! machine with no database. CI and the dev loop set it to a throwaway server:
//!
//! ```sh
//! podman run -d --rm --name bb-pg -e POSTGRES_PASSWORD=dev \
//! -e POSTGRES_DB=blackbeard -p 55432:5432 docker.io/library/postgres:18-alpine
//! TEST_DATABASE_URL='postgres://postgres:dev@127.0.0.1:55432/blackbeard' cargo test
//! ```
//!
//! Each test gets its own schema rather than its own database — same isolation,
//! no `create database` race when tests run concurrently — and drops it on the
//! way out.
//!
//! These exist because the queries they exercise are the ones a unit test
//! cannot reach: an `unnest` batch upsert, the reorg semantics of the primary
//! key, and `date_bin` bucketing. Each has a failure mode that is silent rather
//! than loud — wrong rows, not an error.
use std::time::Duration;
use blackbeard_core::attribution::NodeKey;
use blackbeard_data::store::{BlockRecord, ChainRecord, Store};
use blackbeard_entities::{ChainId, MinerId};
use chrono::{DateTime, TimeZone, Utc};
use sqlx::postgres::PgPoolOptions;
/// A store on a private schema, plus the pool so the schema can be dropped.
struct Fixture {
store: Store,
pool: sqlx::PgPool,
schema: String,
}
impl Fixture {
async fn new(name: &str) -> Option<Self> {
// An empty value counts as unset. `export A=x B="$A"` in one statement
// expands `$A` before assigning it, which is an easy way to end up with
// an empty TEST_DATABASE_URL and a confusing parse error instead of a
// skip.
let url = std::env::var("TEST_DATABASE_URL")
.ok()
.filter(|u| !u.trim().is_empty())?;
let schema = format!("test_{name}");
let pool = PgPoolOptions::new()
.max_connections(2)
// Every connection in this pool lands in the test's own schema, so
// the migrations and the queries under test agree on where the
// tables are without any query needing to qualify a name.
.after_connect({
let schema = schema.clone();
move |conn, _| {
let schema = schema.clone();
Box::pin(async move {
sqlx::query(&format!("set search_path to {schema}"))
.execute(conn)
.await?;
Ok(())
})
}
})
.connect(&url)
.await
.expect("TEST_DATABASE_URL is set but unreachable");
sqlx::query(&format!("drop schema if exists {schema} cascade"))
.execute(&pool)
.await
.unwrap();
sqlx::query(&format!("create schema {schema}"))
.execute(&pool)
.await
.unwrap();
let store = Store::from_pool(pool.clone());
store.migrate().await.expect("migrations apply");
Some(Self {
store,
pool,
schema,
})
}
async fn cleanup(self) {
sqlx::query(&format!("drop schema {} cascade", self.schema))
.execute(&self.pool)
.await
.unwrap();
}
}
fn chain() -> ChainId {
ChainId("planck".into())
}
fn miner(n: u8) -> MinerId {
MinerId(format!("0x{n:064x}"))
}
fn at(secs: i64) -> DateTime<Utc> {
Utc.timestamp_opt(1_757_000_000 + secs, 0).unwrap()
}
fn block(height: u64, m: u8, secs: i64, difficulty: &str) -> BlockRecord {
BlockRecord {
chain: chain(),
height,
hash: format!("0x{height:064x}"),
miner: miner(m),
authored_at: Some(at(secs - 1)),
observed_at: at(secs),
difficulty: Some(difficulty.to_owned()),
}
}
async fn seed_chain(store: &Store) {
store
.upsert_chain(&ChainRecord {
chain: chain(),
display_name: "Planck Testnet".into(),
mainnet: false,
genesis: Some("0x4901".into()),
token_symbol: Some("PLK".into()),
token_decimals: Some(12),
target_block_time: 6.0,
})
.await
.unwrap();
}
macro_rules! fixture {
($name:literal) => {
match Fixture::new($name).await {
Some(f) => f,
None => {
eprintln!("skipping: TEST_DATABASE_URL not set");
return;
}
}
};
}
#[tokio::test]
async fn blocks_round_trip_through_the_batch_upsert() {
let f = fixture!("round_trip");
seed_chain(&f.store).await;
let blocks: Vec<_> = (1..=50)
.map(|h| block(h, (h % 3) as u8, h as i64 * 6, "41650875256"))
.collect();
assert_eq!(f.store.record_blocks(&blocks).await.unwrap(), 50);
assert_eq!(f.store.max_height(&chain()).await.unwrap(), Some(50));
// Oldest first, which is the order the rolling window replays in.
let recent = f.store.recent_blocks(&chain(), 10).await.unwrap();
assert_eq!(recent.len(), 10);
assert_eq!(recent.first().unwrap().height, 41);
assert_eq!(recent.last().unwrap().height, 50);
f.cleanup().await;
}
#[tokio::test]
async fn a_reorg_replaces_the_block_at_that_height() {
// The primary key is (chain, height), not the hash, precisely so this
// happens. A schema keyed on the hash would accumulate both sides of every
// fork and inflate the loser's leaderboard row forever.
let f = fixture!("reorg");
seed_chain(&f.store).await;
f.store
.record_blocks(&[block(100, 1, 600, "1000")])
.await
.unwrap();
let mut replacement = block(100, 2, 601, "1000");
replacement.hash = "0xdifferent".into();
f.store.record_blocks(&[replacement]).await.unwrap();
let rows = f.store.recent_blocks(&chain(), 10).await.unwrap();
assert_eq!(rows.len(), 1, "one block per height, not one per fork");
assert_eq!(rows[0].miner, miner(2));
let (blocks, _, _, _) = f.store.miner_totals(&chain(), &miner(1)).await.unwrap();
assert_eq!(blocks, 0, "the orphaned author keeps no credit");
f.cleanup().await;
}
#[tokio::test]
async fn an_empty_batch_is_a_no_op_not_an_error() {
let f = fixture!("empty_batch");
seed_chain(&f.store).await;
assert_eq!(f.store.record_blocks(&[]).await.unwrap(), 0);
assert_eq!(f.store.max_height(&chain()).await.unwrap(), None);
f.cleanup().await;
}
#[tokio::test]
async fn cumulative_work_sums_difficulty_beyond_any_integer_type() {
// Difficulty is a U512. If `difficulty` were bigint — or if the sum were
// read back through an i64 — this would overflow rather than answer.
let f = fixture!("work");
seed_chain(&f.store).await;
let huge = "1000000000000000000000000000000000000000";
f.store
.record_blocks(&[block(1, 1, 6, huge), block(2, 1, 12, huge)])
.await
.unwrap();
let (blocks, first, last, work) = f.store.miner_totals(&chain(), &miner(1)).await.unwrap();
assert_eq!(blocks, 2);
assert_eq!(first, Some(at(6)));
assert_eq!(last, Some(at(12)));
assert_eq!(work, "2000000000000000000000000000000000000000");
f.cleanup().await;
}
#[tokio::test]
async fn a_miner_with_no_blocks_totals_zero_rather_than_failing() {
let f = fixture!("absent_miner");
seed_chain(&f.store).await;
let (blocks, first, last, work) = f.store.miner_totals(&chain(), &miner(9)).await.unwrap();
assert_eq!((blocks, first, last, work.as_str()), (0, None, None, "0"));
f.cleanup().await;
}
#[tokio::test]
async fn the_series_buckets_on_fixed_boundaries() {
let f = fixture!("series");
seed_chain(&f.store).await;
// Two hours of blocks, one every ten minutes, alternating authors.
let blocks: Vec<_> = (0..12)
.map(|i| block(i + 1, (i % 2) as u8, i as i64 * 600, "100"))
.collect();
f.store.record_blocks(&blocks).await.unwrap();
let series = f
.store
.miner_series(&chain(), &miner(0), at(0), Duration::from_secs(3600))
.await
.unwrap();
// Anchored to the epoch, so every boundary lands on the hour regardless of
// when the range starts or when the query is made. A chart bucketed from
// the range's own start would re-cut itself on every refresh and appear to
// shift under the reader — which is why this assertion, not a tidier one
// about the bucket count, is the one worth making. Two hours of blocks
// starting mid-hour therefore straddle three buckets, not two.
assert!(series.len() >= 2);
for (bucket, _, _) in &series {
assert_eq!(bucket.timestamp() % 3600, 0, "buckets start on the hour");
}
assert_eq!(series.iter().map(|(_, mine, _)| mine).sum::<u32>(), 6);
assert_eq!(series.iter().map(|(_, _, total)| total).sum::<u32>(), 12);
// Buckets are contiguous and ascending.
for pair in series.windows(2) {
assert!(pair[1].0 > pair[0].0);
}
f.cleanup().await;
}
#[tokio::test]
async fn attributions_survive_a_restart() {
let f = fixture!("attribution");
seed_chain(&f.store).await;
f.store
.save_attributions(
&chain(),
&[
(
miner(1),
NodeKey::Peer("12D3KooWalice".into()),
40,
38,
Some("alice".into()),
),
(miner(2), NodeKey::Node(77), 12, 9, None),
],
)
.await
.unwrap();
let mut loaded = f.store.load_attributions(&chain()).await.unwrap();
loaded.sort_by(|a, b| a.miner.cmp(&b.miner));
assert_eq!(loaded.len(), 2);
assert_eq!(loaded[0].key, NodeKey::Peer("12D3KooWalice".into()));
assert_eq!(loaded[0].name.as_deref(), Some("alice"));
assert_eq!(loaded[0].attempts, 40);
assert_eq!(loaded[1].key, NodeKey::Node(77));
// A later save with no resolvable name must not erase the one we had — a
// telemetry hiccup would otherwise cost every miner its name permanently.
f.store
.save_attributions(
&chain(),
&[(
miner(1),
NodeKey::Peer("12D3KooWalice".into()),
41,
39,
None,
)],
)
.await
.unwrap();
let reloaded = f.store.load_attributions(&chain()).await.unwrap();
let alice = reloaded.iter().find(|a| a.miner == miner(1)).unwrap();
assert_eq!(alice.name.as_deref(), Some("alice"));
assert_eq!(alice.attempts, 41);
f.cleanup().await;
}
#[tokio::test]
async fn a_failed_poll_does_not_blank_out_discovered_chain_properties() {
let f = fixture!("chain_upsert");
seed_chain(&f.store).await;
// A poll that reached the node for nothing but the name.
f.store
.upsert_chain(&ChainRecord {
chain: chain(),
display_name: "Planck Testnet".into(),
mainnet: false,
genesis: None,
token_symbol: None,
token_decimals: None,
target_block_time: 6.0,
})
.await
.unwrap();
let row = sqlx::query_scalar::<_, Option<String>>("select genesis from chain where id = $1")
.bind(chain().as_str())
.fetch_one(&f.pool)
.await
.unwrap();
assert_eq!(row.as_deref(), Some("0x4901"));
f.cleanup().await;
}

18
crates/blackbeard-entities/Cargo.toml Normal file
View File

@@ -0,0 +1,18 @@
[package]
name = "blackbeard-entities"
description = "Domain types and wire protocol for blackbeard.observer. No I/O."
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
authors.workspace = true
repository.workspace = true
[dependencies]
chrono.workspace = true
serde.workspace = true
thiserror.workspace = true
ts-rs.workspace = true
[dev-dependencies]
serde_json.workspace = true

60
crates/blackbeard-entities/src/block.rs Normal file
View File

@@ -0,0 +1,60 @@
//! Observed blocks.
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use ts_rs::TS;
use crate::{BigUintDec, ChainId, MinerId};
/// A block as the observer recorded it.
///
/// Two timestamps, deliberately: `authored_at` comes from the block's own
/// `Timestamp::set` inherent, which the author stamps when it *builds* the
/// proposal, while `observed_at` is when this observer first saw the block.
/// The difference is propagation plus poll lag when negative, and a
/// future-dated block when positive. A miner whose blocks arrive in bursts
/// right after someone else's, with `authored_at` well before `observed_at`,
/// is withholding them.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, TS)]
#[ts(export, export_to = "BlockObservation.ts")]
pub struct BlockObservation {
/// Which chain.
pub chain: ChainId,
/// Block number.
#[ts(type = "number")]
pub height: u64,
/// Block hash, `0x`-prefixed.
pub hash: String,
/// Author's reward preimage from the `pow_` PreRuntime digest.
pub miner: MinerId,
/// The author's own timestamp from the block's timestamp inherent.
pub authored_at: Option<DateTime<Utc>>,
/// When this observer first saw the block.
pub observed_at: DateTime<Utc>,
/// Difficulty in force for this block: expected hashes to win it.
pub difficulty: Option<BigUintDec>,
}
/// A block trimmed for the live ticker: what scrolls past on the front page.
///
/// Separate from [`BlockObservation`] because the ticker is pushed on every
/// block to every connected browser, and shipping the full record — including
/// the difficulty string, which changes rarely — would multiply the socket's
/// bandwidth for nothing.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, TS)]
#[ts(export, export_to = "RecentBlock.ts")]
pub struct RecentBlock {
/// Block number.
#[ts(type = "number")]
pub height: u64,
/// Author's reward preimage.
pub miner: MinerId,
/// Display name for the author at the time the block was pushed.
pub display: String,
/// When this observer first saw it.
pub observed_at: DateTime<Utc>,
/// Seconds since the previous block, by observation. `None` for the first
/// block after a restart, where there is no previous observation to
/// subtract.
pub gap_seconds: Option<f64>,
}

126
crates/blackbeard-entities/src/chain.rs Normal file
View File

@@ -0,0 +1,126 @@
//! Chain identity and the per-chain live summary.
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use ts_rs::TS;
use crate::BigUintDec;
/// Short, stable, URL-safe name for a chain (`planck`, `quantus`).
///
/// This is the operator's name for the chain, not the genesis hash: it appears
/// in routes, in config, and in the WebSocket protocol, and it has to survive a
/// chain being specced before it launches. The genesis hash is discovered from
/// the node at runtime and carried in [`ChainInfo`].
#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize, TS)]
#[serde(transparent)]
#[ts(export, export_to = "ChainId.ts", type = "string")]
pub struct ChainId(pub String);
impl ChainId {
/// Borrow the name.
pub fn as_str(&self) -> &str {
&self.0
}
}
impl std::fmt::Display for ChainId {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(&self.0)
}
}
impl From<&str> for ChainId {
fn from(s: &str) -> Self {
Self(s.to_owned())
}
}
/// How much of a chain the observer can currently see.
///
/// A chain that is configured but not yet launched is a first-class state, not
/// an error: mainnet exists in config before it exists on the network, and the
/// UI has to say "not yet live" rather than "backend broken".
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, TS)]
#[serde(rename_all = "snake_case")]
#[ts(export, export_to = "ChainStatus.ts")]
pub enum ChainStatus {
/// Connected, node in sync, every metric trustworthy.
Live,
/// Connected, but the node is still importing history. Block interval — and
/// therefore network hashrate — is nominal until this clears.
Syncing,
/// Configured but the node has never answered. Expected for a chain that
/// has not launched.
Awaiting,
/// The node answered before and has stopped. Distinct from `Awaiting`
/// because it means something broke rather than something not yet started.
Unreachable,
}
/// Static-ish facts about a chain: what to call it and how to read its numbers.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, TS)]
#[ts(export, export_to = "ChainInfo.ts")]
pub struct ChainInfo {
/// Operator name, used in routes and the WS protocol.
pub id: ChainId,
/// What a human should see: "Quantus", "Planck Testnet".
pub display_name: String,
/// True for the production network, false for a testnet. Drives whether the
/// UI treats rewards as real.
pub mainnet: bool,
/// Genesis hash, discovered from the node. `None` until it first answers.
pub genesis: Option<String>,
/// Token ticker from `system_properties`, e.g. `QUAN`.
pub token_symbol: Option<String>,
/// Token decimals from `system_properties`. Balances are integers of the
/// smallest unit; this is how many places to shift them.
pub token_decimals: Option<u8>,
/// Target seconds per block the chain retargets towards. Used as the
/// hashrate denominator whenever a measured interval is not yet available.
pub target_block_time_seconds: f64,
/// Current reachability.
pub status: ChainStatus,
}
/// The live headline numbers for one chain: one row of the scoreboard's top.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, TS)]
#[ts(export, export_to = "ChainSummary.ts")]
pub struct ChainSummary {
/// Which chain this describes.
pub chain: ChainId,
/// Best block height seen.
#[ts(type = "number | null")]
pub height: Option<u64>,
/// Current mining difficulty: expected hashes per block.
pub difficulty: Option<BigUintDec>,
/// The ceiling difficulty can reach, from `QPoWApi_get_max_difficulty`.
/// The ratio of the two is how much headroom the network has left.
pub max_difficulty: Option<BigUintDec>,
/// Estimated network hashrate in hashes per second: difficulty divided by
/// the block interval.
pub network_hashrate: Option<f64>,
/// True when `network_hashrate` used the configured target rather than a
/// measured interval. The figure is nominal — say so in the UI rather than
/// presenting an estimate as an observation.
pub hashrate_from_target: bool,
/// Measured mean seconds per block at the tip, when enough tip samples
/// exist. `None` while syncing or just after a restart.
pub block_interval_seconds: Option<f64>,
/// The configured target, for comparison against the measured interval.
pub target_block_time_seconds: f64,
/// Blocks in the rolling leaderboard window.
pub window_blocks: u32,
/// Distinct reward preimages that authored in the window. A **lower bound**
/// on miner count: one operator may run several preimages, and several
/// operators could in principle share one.
pub distinct_miners: u32,
/// Nodes on this chain per the substrate-telemetry feed, when connected.
/// Compare against `distinct_miners`: node count flat while distinct
/// authors falls is small miners still running and simply being out-hashed.
pub telemetry_nodes: Option<u32>,
/// Whether the telemetry feed is currently connected.
pub telemetry_connected: bool,
/// When these numbers were computed.
pub updated_at: DateTime<Utc>,
}

47
crates/blackbeard-entities/src/error.rs Normal file
View File

@@ -0,0 +1,47 @@
//! Error types shared across the workspace.
use serde::{Deserialize, Serialize};
use ts_rs::TS;
/// Errors raised by the types themselves — parsing a window name, a malformed
/// preimage. Deliberately small: this crate does no I/O, so it has few ways to
/// fail.
#[derive(Debug, thiserror::Error)]
pub enum EntityError {
/// A window name that is not one of the four.
#[error("unknown window `{0}` (expected hour, six_hours, day or week)")]
UnknownWindow(String),
/// A miner id that is not 0x-prefixed 32-byte hex.
#[error("`{0}` is not a 32-byte 0x-prefixed reward preimage")]
MalformedMinerId(String),
/// A chain name that is not configured.
#[error("no chain named `{0}` is configured")]
UnknownChain(String),
}
/// The JSON body every failing REST response carries.
///
/// Shared with the frontend so error rendering is not a guess about what the
/// backend emits.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, TS)]
#[ts(export, export_to = "ApiError.ts")]
pub struct ApiError {
/// A stable machine-readable code (`unknown_chain`, `not_found`,
/// `chain_unavailable`). The frontend switches on this; `message` is for
/// humans and may be reworded at any time.
pub code: String,
/// Human-readable explanation.
pub message: String,
}
impl ApiError {
/// Build one.
pub fn new(code: &str, message: impl Into<String>) -> Self {
Self {
code: code.to_owned(),
message: message.into(),
}
}
}

79
crates/blackbeard-entities/src/lib.rs Normal file
View File

@@ -0,0 +1,79 @@
//! Domain types and the browser wire protocol for blackbeard.observer.
//!
//! Types only — no I/O, no async runtime, no chain client. Everything
//! downstream (`blackbeard-core`, `blackbeard-data`, `blackbeard-api`) depends
//! on this crate, and the TypeScript the frontend consumes is generated from it
//! by `ts-rs` (`cargo test -p blackbeard-entities` writes `web/src/api/generated/`).
//!
//! ## The one type that needs explaining
//!
//! Difficulty on this chain is a **U512** — expected hashes per block, routinely
//! past `1e30`. It does not fit an `f64` without losing exactness and does not
//! fit any integer type serde will put on the wire, so it crosses every boundary
//! as [`BigUintDec`]: a decimal string. Arithmetic on it happens in
//! `blackbeard-core`, which parses it once; nothing here does maths.
#![deny(missing_docs)]
mod block;
mod chain;
mod error;
mod miner;
mod ws;
pub use block::{BlockObservation, RecentBlock};
pub use chain::{ChainId, ChainInfo, ChainStatus, ChainSummary};
pub use error::{ApiError, EntityError};
pub use miner::{AttributionSource, LeaderboardRow, MinerDetail, MinerId, MinerSeriesPoint};
pub use ws::{ClientMessage, ServerMessage, Window};
use serde::{Deserialize, Serialize};
use ts_rs::TS;
/// An arbitrary-precision non-negative integer carried as a decimal string.
///
/// Difficulty and cumulative hash counts are U512 on this chain; JSON numbers
/// are IEEE-754 doubles, so putting them on the wire as numbers would silently
/// round them. The string is the canonical form on the wire, in the database
/// (`numeric`), and in the browser — the frontend formats it with `BigInt`.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize, TS)]
#[serde(transparent)]
#[ts(export, export_to = "BigUintDec.ts", type = "string")]
pub struct BigUintDec(pub String);
impl BigUintDec {
/// Zero, for the case where the chain has not yet answered.
pub fn zero() -> Self {
Self("0".into())
}
/// The decimal digits, without allocating.
pub fn as_str(&self) -> &str {
&self.0
}
}
impl std::fmt::Display for BigUintDec {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(&self.0)
}
}
impl From<String> for BigUintDec {
fn from(s: String) -> Self {
Self(s)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn biguint_is_transparent_on_the_wire() {
// The frontend parses this with BigInt(...), so it must be a bare JSON
// string and never an object wrapper.
let json = serde_json::to_string(&BigUintDec("1234567890123456789012345".into())).unwrap();
assert_eq!(json, "\"1234567890123456789012345\"");
}
}

148
crates/blackbeard-entities/src/miner.rs Normal file
View File

@@ -0,0 +1,148 @@
//! Miner identity, leaderboard rows, and per-miner history.
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use ts_rs::TS;
use crate::{BigUintDec, ChainId};
/// A miner's on-chain identity: the 32-byte wormhole reward **preimage**,
/// `0x`-prefixed lowercase hex.
///
/// This is the only miner identity the chain offers without cooperation. Every
/// block header carries its author's preimage in a `PreRuntime` digest with
/// engine id `pow_`, so authorship for *every* miner on the network is
/// derivable from headers alone — no indexer, no registration, no opt-in.
///
/// It is **public data, not a secret**: it is published from a miner's first
/// authored block onward, and spending the rewards needs a plonky2 proof of
/// knowledge of the underlying secret, which the preimage does not reveal. It
/// is also **static per wallet**, which is exactly what makes a leaderboard
/// possible — and what makes mining income permanently attributable.
#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize, TS)]
#[serde(transparent)]
#[ts(export, export_to = "MinerId.ts", type = "string")]
pub struct MinerId(pub String);
impl MinerId {
/// Borrow the hex.
pub fn as_str(&self) -> &str {
&self.0
}
/// The abbreviated form shown when a miner has no attributed node name:
/// `0x1234abcd…ef01`. Long enough to be distinguishable at a glance, short
/// enough to sit in a table cell.
pub fn abbreviated(&self) -> String {
if self.0.len() > 14 {
format!("{}…{}", &self.0[..10], &self.0[self.0.len() - 4..])
} else {
self.0.clone()
}
}
}
impl std::fmt::Display for MinerId {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(&self.0)
}
}
impl From<&str> for MinerId {
fn from(s: &str) -> Self {
Self(s.to_ascii_lowercase())
}
}
/// How a miner's display name was arrived at.
///
/// The distinction matters on screen: a telemetry-attributed name is an
/// inference from who reported a block first, not a claim the miner made, and
/// the UI must not present the two identically.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, TS)]
#[serde(rename_all = "snake_case")]
#[ts(export, export_to = "AttributionSource.ts")]
pub enum AttributionSource {
/// No name — the row shows the abbreviated preimage.
Preimage,
/// A substrate-telemetry node name, inferred because that node consistently
/// reported this author's blocks before anyone else. Carries a confidence.
Telemetry,
}
/// One row of the leaderboard.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, TS)]
#[ts(export, export_to = "LeaderboardRow.ts")]
pub struct LeaderboardRow {
/// 1-based position within the window. Ties break by most recent block.
pub rank: u32,
/// The miner's reward preimage — the stable key for this row.
pub miner: MinerId,
/// What to render: a telemetry node name when one is attributed, else the
/// abbreviated preimage.
pub display: String,
/// Where `display` came from.
pub attribution: AttributionSource,
/// Fraction of attributed blocks pointing at the named node, 0.0–1.0. Zero
/// when `attribution` is `Preimage`.
pub confidence: f32,
/// Blocks authored inside the window.
pub blocks: u32,
/// Share of the window's blocks, 0.0–1.0.
pub share: f64,
/// Implied hashrate in hashes per second: the miner's share of blocks times
/// the estimated network hashrate. A statistical estimate over a finite
/// window, not a measurement of the miner's hardware — a lucky small miner
/// will over-read and an unlucky large one will under-read.
pub hashrate_estimate: Option<f64>,
/// Height of this miner's most recent block in the window.
#[ts(type = "number")]
pub last_block_height: u64,
/// When the observer saw that block.
pub last_seen: DateTime<Utc>,
/// Longest run of consecutive blocks by this miner within the window. A run
/// far above what its share predicts is the signature of withheld blocks.
pub best_streak: u32,
}
/// A point on a miner's history chart.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, TS)]
#[ts(export, export_to = "MinerSeriesPoint.ts")]
pub struct MinerSeriesPoint {
/// Start of the bucket.
pub at: DateTime<Utc>,
/// Blocks authored inside the bucket.
pub blocks: u32,
/// Share of all blocks in the bucket, 0.0–1.0.
pub share: f64,
/// Implied hashrate over the bucket, hashes per second.
pub hashrate_estimate: Option<f64>,
}
/// Everything the miner detail page shows for one preimage.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, TS)]
#[ts(export, export_to = "MinerDetail.ts")]
pub struct MinerDetail {
/// Which chain.
pub chain: ChainId,
/// The miner.
pub miner: MinerId,
/// Current leaderboard row, absent if the miner authored nothing in the
/// window (a small miner between blocks, or one that has stopped).
pub current: Option<LeaderboardRow>,
/// Blocks authored since the observer's records begin — not since chain
/// genesis, unless the observer has watched from genesis.
#[ts(type = "number")]
pub blocks_observed: u64,
/// First block the observer attributed to this miner.
pub first_seen: Option<DateTime<Utc>>,
/// Most recent.
pub last_seen: Option<DateTime<Utc>>,
/// Bucketed history for charting, oldest first.
pub series: Vec<MinerSeriesPoint>,
/// Sum of the difficulty of every block this miner authored in the observed
/// record: the expected number of hashes it took to win them. The honest
/// cumulative measure of work done, and the one that does not shrink when
/// difficulty rises.
pub cumulative_work: BigUintDec,
}

166
crates/blackbeard-entities/src/ws.rs Normal file
View File

@@ -0,0 +1,166 @@
//! The browser wire protocol.
//!
//! One WebSocket per browser tab carries everything the page needs after the
//! initial handshake. The client subscribes to a chain and gets a
//! [`ServerMessage::Snapshot`] immediately, then deltas: a `Block` on every
//! block, a `Summary` when the headline numbers move, a `Leaderboard` when the
//! standings change. There is no polling and no refetch — the socket *is* the
//! data source, and a REST call exists only to serve the first paint and to
//! give something to `curl`.
//!
//! Both directions are externally tagged on `type`, so the TypeScript side gets
//! a discriminated union it can `switch` on exhaustively.
use serde::{Deserialize, Serialize};
use ts_rs::TS;
use crate::{ChainId, ChainInfo, ChainSummary, LeaderboardRow, RecentBlock};
/// How far back a leaderboard looks.
///
/// A **block count**, never a duration. A time window is meaningless while a
/// node is catching up: it imports historical blocks at disk speed, so "the
/// last hour" can contain half a million blocks. Block counts stay honest in
/// both regimes, and the UI renders the approximate duration beside them from
/// the measured interval.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Hash, Serialize, Deserialize, TS)]
#[serde(rename_all = "snake_case")]
#[ts(export, export_to = "Window.ts")]
pub enum Window {
/// 600 blocks — about an hour at a 6 s target.
Hour,
/// 3,600 blocks — about six hours. The default, matching the arena
/// exporter's window so the two agree.
#[default]
SixHours,
/// 14,400 blocks — about a day.
Day,
/// 100,800 blocks — about a week.
Week,
}
impl Window {
/// The window's length in blocks.
pub fn blocks(self) -> u32 {
match self {
Window::Hour => 600,
Window::SixHours => 3_600,
Window::Day => 14_400,
Window::Week => 100_800,
}
}
/// Every window, for building a selector.
pub fn all() -> [Window; 4] {
[Window::Hour, Window::SixHours, Window::Day, Window::Week]
}
}
impl std::str::FromStr for Window {
type Err = crate::EntityError;
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s {
"hour" => Ok(Window::Hour),
"six_hours" | "6h" => Ok(Window::SixHours),
"day" => Ok(Window::Day),
"week" => Ok(Window::Week),
other => Err(crate::EntityError::UnknownWindow(other.to_owned())),
}
}
}
/// Browser to server.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, TS)]
#[serde(tag = "type", rename_all = "snake_case")]
#[ts(export, export_to = "ClientMessage.ts")]
pub enum ClientMessage {
/// Watch a chain at a window. Replaces any previous subscription for that
/// chain — a client changing window sends this again rather than
/// unsubscribing first.
Subscribe {
/// Chain to watch.
chain: ChainId,
/// Leaderboard window.
window: Window,
},
/// Stop watching a chain.
Unsubscribe {
/// Chain to drop.
chain: ChainId,
},
/// Keepalive. The server answers [`ServerMessage::Pong`].
///
/// Browsers cannot send WebSocket ping frames from JavaScript, so an
/// application-level ping is the only way a tab can prove its socket is
/// still alive through a proxy that has gone quiet.
Ping,
}
/// Server to browser.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, TS)]
#[serde(tag = "type", rename_all = "snake_case")]
#[ts(export, export_to = "ServerMessage.ts")]
pub enum ServerMessage {
/// Sent unprompted on connect: which chains exist and how they are doing.
/// Lets the chain switcher render before any subscription is made.
Chains {
/// Every configured chain.
chains: Vec<ChainInfo>,
},
/// Everything about one chain, sent immediately on subscribe and never
/// again for that subscription unless the client resubscribes.
Snapshot {
/// Which chain.
chain: ChainId,
/// Window this snapshot's leaderboard covers.
window: Window,
/// Headline numbers.
summary: ChainSummary,
/// Full standings for the window.
leaderboard: Vec<LeaderboardRow>,
/// The tail of the block ticker, newest last.
recent_blocks: Vec<RecentBlock>,
},
/// A new block arrived.
Block {
/// Which chain.
chain: ChainId,
/// The block.
block: RecentBlock,
},
/// The headline numbers moved.
Summary {
/// Which chain.
chain: ChainId,
/// New numbers.
summary: ChainSummary,
},
/// The standings changed. Sent whole rather than as a diff: a leaderboard
/// is at most a few hundred rows, and a wholesale replace cannot drift out
/// of sync with the server the way an accumulated patch stream can.
Leaderboard {
/// Which chain.
chain: ChainId,
/// Window these standings cover.
window: Window,
/// The standings.
rows: Vec<LeaderboardRow>,
},
/// A chain's reachability changed — the node went away, or a chain that was
/// awaiting launch has started producing blocks.
ChainStatus {
/// Which chain.
chain: ChainId,
/// Its new state.
info: ChainInfo,
},
/// Answer to [`ClientMessage::Ping`].
Pong,
/// Something the client asked for could not be served. Not fatal — the
/// socket stays open.
Error {
/// Human-readable reason.
message: String,
},
}

237
readme.md Normal file
View File

@@ -0,0 +1,237 @@
# blackbeard.observer
Live miner leaderboard and network hashrate for the [Quantus](https://quantus.com)
blockchain and its Planck testnet, at **<https://blackbeard.observer>**
(`blackbeard.internal` from the mesh).
The point of it: a miner can find their own row, see what share of the network
they are actually winning, and watch it move — against everyone else, in real
time. Mining is a competition that normally gives its participants no scoreboard.
This is the scoreboard.
## How it knows who mined what
Every block header carries its author's 32-byte wormhole **reward preimage** in a
`PreRuntime` digest stamped with the engine id `pow_`. The node writes it there so
the `mining-rewards` pallet can read it back and derive the payout address
on-chain — and as a side effect, authorship for **every miner on the network** is
derivable from headers alone.
That is the whole mechanism, and its consequences are worth stating plainly:
- **No registration, no opt-in, no way to be left out.** Every miner appears the
moment they win a block. Nothing here depends on anyone's cooperation.
- **The preimage is public, not secret.** It is published from a miner's first
authored block onward. Spending the rewards needs a plonky2 proof of knowledge
of the underlying secret, which the preimage does not reveal — but it does make
mining income permanently attributable. The privacy in this design is at the
*exit*, not at receipt. `lair/quantus`'s `doc/wormhole-rewards.md` has the full
analysis.
- **A leaderboard row is a preimage, not a person.** One operator may run several;
the count of distinct preimages is a *lower bound* on miner count.
Difficulty comes from the `QPoWApi_get_difficulty` runtime call. Difficulty *is*
the expected number of hashes to win a block, so network hashrate is difficulty
divided by the block interval — and a miner's implied hashrate is their share of
recent blocks times that.
The mechanics were first worked out in `lair/quantus`'s Prometheus arena
exporter (`asset/arena/quantus-arena-exporter.py`), which remains the reference
for anyone checking this implementation against a second one.
## Names
The chain gives us preimages; a board of 64-character hex strings is honest and
unreadable. substrate-telemetry closes the gap indirectly.
The feed reports block imports with a propagation time, stamping the **first**
reporter of a hash with `0`. A node imports its own block before announcing it,
so the author — if it is on telemetry at all — is nearly always first. One block
proves nothing (measured on Planck, an author's lead over the second reporter is
50–620 ms, which a well-peered bystander can produce), so the observer **votes
over a rolling window** and only shows a name once enough blocks agree.
An attributed name is therefore **an inference, never a claim the miner made and
never an identity check**. The UI marks it as such, carries the confidence in a
tooltip, and falls back to the abbreviated preimage rather than guessing. Where
two miners resolve to the same node name — one operator with several preimages,
or two people who never renamed their node — every member of the colliding group
gets a preimage suffix, so no row is silently privileged as "the real one".
## Layout
Cargo workspace plus a Vite frontend, per `~/git/architecture/generic.md` §1.
```
crates/
blackbeard-entities/ domain types + the browser wire protocol; ts-rs exports
the TypeScript the frontend consumes. No I/O.
blackbeard-core/ header/SCALE decoding, hashrate maths, the rolling
window, telemetry attribution voting. Pure — no I/O,
no clock, no sockets, exercised entirely by unit tests.
blackbeard-data/ chain JSON-RPC, the telemetry feed, Postgres.
blackbeard-api/ the daemon: ingest tasks, REST, WebSocket fanout.
blackbeard-cli/ operator tools: probe, backfill, standings.
web/ Vite + React + SWC + TS. Static build, served by nginx.
asset/ systemd, firewalld, nginx, config template, bootstrap SQL.
script/infra-setup.sh one-time host provisioning, operator-run.
```
**The frontend's types are generated from the Rust.** `cargo test -p
blackbeard-entities` writes `web/src/api/generated/`; CI fails the build if the
committed output is stale. Edit the Rust DTO, never the TypeScript — otherwise
the two compile cleanly and disagree at runtime.
## Real time, without observables
One WebSocket per browser tab. On subscribe the client gets a full snapshot;
after that only deltas — a `block` on every block, a `summary` when the headline
numbers move, a `leaderboard` when the standings change. The page never polls and
never refetches.
The head stream is itself a push: `chain_subscribeNewHeads` over the node's
WebSocket, so the observer learns of a block the moment the node imports it. The
only polling anywhere is difficulty and sync state, on a four-second timer.
RxJS was considered and not used. This needs one stream, one reducer and one
subscriber list; `useSyncExternalStore` is React's own contract for exactly that,
gets concurrent rendering right, and costs no dependency and no idiom in every
component. The store is `web/src/api/socket.ts` — about 200 lines including the
reconnect and resubscribe logic.
Server-side, messages are serialised **once** per broadcast and every socket
writes the same `Arc<str>`; leaderboards are recomputed only for windows that
actually have a subscriber.
## Windows are block counts, not durations
Every window on the site — 1h, 6h, 24h, 7d — is a **block count** (600, 3 600,
14 400, 100 800). The labels are the approximate duration at the measured
interval, and the UI says "~".
This is not pedantry. A duration is meaningless while a node is catching up: it
imports historical blocks at disk speed, so "the last hour" can contain half a
million blocks. The same reasoning is why a *measured* block interval is only
used when the node is at the tip; while syncing, the configured target is used
and every hashrate on the page is labelled **nominal**.
## Data
Postgres on `magrathea.kosherinata.internal`, mTLS and passwordless — the host's
own certificate is the credential and `pg_ident.conf` maps its CN to the role.
There is no password anywhere in this repo and nowhere in the config to put one.
Blocks are keyed on `(chain, height)`, not on the block hash, deliberately: this
is a proof-of-work chain and it reorgs. An upsert on that key means a block
replacing another at the same height overwrites it, so the standings reflect the
canonical chain rather than the union of every fork the observer witnessed.
The database is what lets the site keep promises a rolling window cannot: "how
did I do last week", a hashrate line older than the process, and a leaderboard
that survives a deploy. On start the daemon replays the window and the held
attribution names out of it, so a restart does not serve an empty site.
## Build and run
```sh
# A throwaway Postgres for the dev loop and for the sqlx query cache.
podman run -d --rm --name bb-pg -e POSTGRES_PASSWORD=dev \
-e POSTGRES_DB=blackbeard -p 55432:5432 docker.io/library/postgres:18-alpine
export DATABASE_URL='postgres://postgres:dev@127.0.0.1:55432/blackbeard'
export TEST_DATABASE_URL="$DATABASE_URL"
cargo fmt --all && cargo clippy --all-targets -- -D warnings && cargo test --all
# The daemon, against a real node. BLACKBEARD_DEV_DATABASE_URL is the
# development-only path that skips mTLS; the systemd unit never sets it.
BLACKBEARD_DEV_DATABASE_URL="$DATABASE_URL" \
cargo run -p blackbeard-api -- --config dev-config.toml
cd web && pnpm install && pnpm dev # proxies /v1 to 127.0.0.1:25864
```
`cargo test` skips the database integration tests when `TEST_DATABASE_URL` is
unset, so it stays useful on a machine with no Postgres.
**Before adding a chain, probe it.** Every failure the probe reports is one the
deployed daemon would hit silently — an RPC without the QPoW runtime API gives a
site with no hashrate, and headers without a `pow_` digest give an empty
leaderboard, both of which look like a working deployment:
```sh
cargo run -p blackbeard-cli -- probe --rpc-url http://bob.hanzalova.internal:9944
```
## Deploy
CI-driven (`architecture/deployment-gitea-actions.md`): push to `main`, or run
the workflow from the Actions UI. Hosts, ports and paths live in
`.gitea/workflows/deploy.yaml` and nowhere else.
One-time host provisioning is operator-run, from a workstation with full sudo:
```sh
./script/infra-setup.sh --pubkey ~/.ssh/id_gitea_ci.pub
```
Re-run it whenever the deploy gains a new file to ship — each deploy job
preflights the target's sudoers against the grants in that script and fails up
front naming what is missing, rather than dying partway through an rsync.
Two things it deliberately does not automate, both because they touch shared
infrastructure: the public Let's Encrypt certificate (`external-tls.md`) and the
split-horizon `blackbeard.internal` record, which must be added to **both** site
routers' Unbound or it `NXDOMAIN`s everywhere but one site.
| Port | What | Where |
| --- | --- | --- |
| `25864` | `blackbeard-api` REST + WebSocket | `bob`, mesh address only, plain HTTP behind nginx |
Registered in `architecture/port-allocations.md` §5; derived from the service
name per §3.
## Mainnet
Quantus mainnet is not yet live. It is a commented-out `[[chains]]` block in
`asset/config/config.toml.tmpl`, waiting on a published chain spec and an
endpoint.
A chain may be configured **before it launches**: the observer reports it as
`awaiting`, the UI says so plainly, and it comes alive on its own the moment the
node starts answering. No redeploy, no restart. Adding it is a config change and
a `probe` run.
## Deviations from house convention
- **`runs-on: infra` for the deploy jobs, not `fedora-43`.** The targets are
mesh-only `.internal` names and the fedora runners have no route to the mesh.
Same reasoning as `lair/quantus` and `lair/mail`.
- **Dark theme only.** `generic.md` says nothing about themes, but the dataviz
conventions expect a selected dark mode alongside light. This site commits to
one look: it is a scoreboard for a proof-of-work chain, and a light variant
would be a second palette to validate for a context this content does not have.
- **`BLACKBEARD_DEV_DATABASE_URL`** bypasses the mandated mTLS Postgres
connection. It exists so `cargo run` works on a workstation with no
`pg_ident` mapping on the fleet cluster. It is an environment variable rather
than a config key precisely so it cannot be reached by editing a deployed
config, and taking it logs a warning loud enough to spot in the journal.
## What this cannot tell you
- **Hashrate figures are estimates, not measurements.** A miner's implied
hashrate is their share of blocks won over a finite window times the network
estimate. Winning blocks is a Poisson process: a miner holding 1% of the
network will quite ordinarily show anywhere from 2 to 11 blocks in a
600-block window. Short windows flatter the lucky and libel the unlucky.
- **Distinct miners is a lower bound.** One operator, several preimages.
- **Orphan and stale-work loss is not measured.** Answering "do I win the share
of blocks my hashrate predicts?" honestly needs a hashrate the observer cannot
see. `lair/quantus` issue #1 tracks the same gap from the exporter side.
- **A telemetry name is an inference.** See *Names* above.
## Related
- `lair/quantus` — the node and miner deployment, the arena exporter these
mechanics come from, and the measured hashrate numbers behind the fleet.
- `~/git/architecture` — the house conventions this project follows.

6
rust-toolchain.toml Normal file
View File

@@ -0,0 +1,6 @@
# Pinned to match the CI runner image (gitea-runners.md). Bump here and in the
# workflow together, never one without the other.
[toolchain]
channel = "stable"
components = ["rustfmt", "clippy"]
targets = ["x86_64-unknown-linux-musl"]

1
rustfmt.toml Normal file
View File

@@ -0,0 +1 @@
edition = "2024"

343
script/infra-setup.sh Executable file
View File

@@ -0,0 +1,343 @@
#!/usr/bin/env bash
#
# One-time host provisioning for blackbeard.observer.
#
# Run by an OPERATOR from a workstation with full sudo ssh to the targets — not
# by CI. The runner deploys as a scoped `gitea_ci` user and deliberately has no
# rights to create accounts, read certificate keys, or reload nginx on a shared
# edge proxy.
#
# ./script/infra-setup.sh --pubkey ~/.ssh/id_gitea_ci.pub
#
# Idempotent, and it skips past unreachable hosts so one offline node does not
# block the rest. Re-run it whenever the deploy gains a new file to ship: every
# deploy job preflights the target's sudoers against the grants below and fails
# up front naming what is missing, so the two cannot silently drift.
#
# Roles (all run by default; pass --role to narrow):
#
# api the host running blackbeard-api beside quantus-node
# edge the site's nginx proxy: vhosts, webroot, internal cert
# database Postgres roles, database, and the pg_ident CN mapping
#
# Conventions: architecture/generic.md §8-§11, deployment-gitea-actions.md.
set -euo pipefail
# --- infra truth, matching .gitea/workflows/deploy.yaml -----------------------
API_HOST="${API_HOST:-bob.hanzalova.internal}"
API_PORT="${API_PORT:-25864}"
EDGE_HOST="${EDGE_HOST:-hanzalova.internal}"
PG_PRIMARY="${PG_PRIMARY:-magrathea.kosherinata.internal}"
# The standby needs the same ident mapping: pg_ident.conf contents are NOT
# replicated, and a failover to a server missing it locks the app out.
PG_STANDBY="${PG_STANDBY:-frankie.hanzalova.internal}"
PG_VERSION="${PG_VERSION:-18}"
WEBROOT="${WEBROOT:-/var/www/blackbeard.observer}"
PUBLIC_NAME="${PUBLIC_NAME:-blackbeard.observer}"
INTERNAL_NAME="${INTERNAL_NAME:-blackbeard.internal}"
DB_NAME="${DB_NAME:-blackbeard}"
DB_ROLE="${DB_ROLE:-blackbeard_rw}"
PUBKEY=""
ROLES="api edge database"
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
info() { printf '\033[36m==\033[0m %s\n' "$*"; }
warn() { printf '\033[33m!!\033[0m %s\n' "$*" >&2; }
die() { printf '\033[31mXX\033[0m %s\n' "$*" >&2; exit 1; }
usage() {
sed -n '3,30p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'
exit "${1:-0}"
}
while [ $# -gt 0 ]; do
case "$1" in
--pubkey) PUBKEY="$2"; shift 2 ;;
--role) ROLES="$2"; shift 2 ;;
-h|--help) usage 0 ;;
*) die "unknown argument: $1 (try --help)" ;;
esac
done
has_role() { [[ " $ROLES " == *" $1 "* ]]; }
# Reachability is checked once per host and the result reused, so an offline
# host produces one clear message rather than a failure per step.
reachable() {
local host="$1"
if ssh -o ConnectTimeout=8 -o BatchMode=yes "$host" true; then
return 0
fi
warn "$host is unreachable — skipping its steps"
return 1
}
# --- the scoped sudoers grants ------------------------------------------------
#
# These strings are the single source of truth for what CI may do on each host.
# The deploy workflow's preflight extracts the paths from THIS FILE and compares
# them against `sudo -n -l` on the target, so adding a file to the deploy means
# adding a line here and re-running this script — nothing else keeps them in
# step.
#
# `:` and `=` are reserved in sudoers and must be escaped inside command
# arguments, or visudo rejects the file.
api_sudoers() {
cat <<'EOF'
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /usr/local/bin/blackbeard-api
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /usr/local/bin/blackbeard
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/blackbeard/config.toml
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/sysusers.d/blackbeard.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/systemd/system/blackbeard-api.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/systemd/system/blackbeard-api-cert.path
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/systemd/system/blackbeard-api-cert-reload.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/firewalld/services/blackbeard-api.xml
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemd-sysusers
gitea_ci ALL=(root) NOPASSWD: /usr/bin/install -d -o root -g blackbeard -m 0750 /etc/blackbeard
gitea_ci ALL=(root) NOPASSWD: /usr/bin/setfacl -m u\:blackbeard\:r /etc/pki/tls/private/*
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/restorecon -R /usr/local/bin/blackbeard-api /usr/local/bin/blackbeard /etc/blackbeard
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/semanage port -l
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/semanage port -a -t http_port_t -p tcp 25864
gitea_ci ALL=(root) NOPASSWD: /usr/bin/firewall-cmd --reload
gitea_ci ALL=(root) NOPASSWD: /usr/bin/firewall-cmd --get-default-zone
gitea_ci ALL=(root) NOPASSWD: /usr/bin/firewall-cmd --zone=* --query-service=blackbeard-api
gitea_ci ALL=(root) NOPASSWD: /usr/bin/firewall-cmd --permanent --zone=* --add-service=blackbeard-api
gitea_ci ALL=(root) NOPASSWD: /usr/bin/firewall-cmd --zone=* --add-service=blackbeard-api
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl enable blackbeard-api.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now blackbeard-api-cert.path
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl restart blackbeard-api.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl is-active blackbeard-api.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sudo -u blackbeard /usr/local/bin/blackbeard-api --config /etc/blackbeard/config.toml --check
EOF
}
edge_sudoers() {
cat <<'EOF'
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /var/www/blackbeard.observer/
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/restorecon -R /var/www/blackbeard.observer
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/nginx -t
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl reload nginx
EOF
}
# --- gitea_ci -----------------------------------------------------------------
provision_gitea_ci() {
local host="$1" sudoers_body="$2"
[ -n "$PUBKEY" ] || die "--pubkey is required to provision gitea_ci (the runner's public key)"
[ -f "$PUBKEY" ] || die "$PUBKEY does not exist"
info "$host: gitea_ci account"
# A real shell, never nologin. The deploy runs `ssh gitea_ci@host <command>`;
# a nologin shell authenticates the key and then refuses the command with
# "This account is currently not available", which reads as an auth problem
# rather than a shell one. Every pre-existing gitea_ci on the fleet has bash,
# and one provisioned with nologin can never be deployed to — so repair it
# rather than leaving it.
ssh "$host" sudo bash -euo pipefail <<'REMOTE'
if ! id gitea_ci >/dev/null 2>&1; then
useradd --system --create-home --home-dir /var/lib/gitea_ci --shell /bin/bash gitea_ci
fi
current=$(getent passwd gitea_ci | cut -d: -f7)
if [ "$current" != "/bin/bash" ]; then
echo "repairing gitea_ci shell: $current -> /bin/bash"
usermod --shell /bin/bash gitea_ci
fi
install -d -o gitea_ci -g gitea_ci -m 0700 /var/lib/gitea_ci/.ssh
# Lets the deploy capture `journalctl -u <unit>` after a restart without
# a sudoers entry for it.
usermod -aG systemd-journal gitea_ci
REMOTE
rsync -a --chown gitea_ci:gitea_ci --chmod 0600 --rsync-path 'sudo rsync' \
"$PUBKEY" "$host:/var/lib/gitea_ci/.ssh/authorized_keys"
info "$host: scoped sudoers"
# Named <app>_gitea_ci, not bare gitea_ci, so several apps can drop their own
# files on a shared host without clobbering each other.
printf '%s\n' "$sudoers_body" | \
ssh "$host" 'sudo tee /etc/sudoers.d/blackbeard_gitea_ci > /dev/null && sudo chmod 0440 /etc/sudoers.d/blackbeard_gitea_ci'
# Verify before leaving: a syntax error in a sudoers drop-in can lock every
# sudo on the host, not just this one.
ssh "$host" sudo visudo -cf /etc/sudoers.d/blackbeard_gitea_ci
}
# --- roles --------------------------------------------------------------------
role_api() {
reachable "$API_HOST" || return 0
provision_gitea_ci "$API_HOST" "$(api_sudoers)"
info "$API_HOST: service account and directories"
rsync -a --rsync-path 'sudo rsync' \
"$REPO_ROOT/asset/systemd/blackbeard.sysusers.conf" \
"$API_HOST:/etc/sysusers.d/blackbeard.conf"
ssh "$API_HOST" sudo bash -euo pipefail <<'REMOTE'
systemd-sysusers
install -d -o root -g blackbeard -m 0750 /etc/blackbeard
install -d -o blackbeard -g blackbeard -m 0750 /var/lib/blackbeard
# The mTLS credential for Postgres. The key is not world-readable; the
# service account is granted read access here and again on every deploy,
# because a certificate rotation replaces the file and drops the ACL.
setfacl -m "u:blackbeard:r" "/etc/pki/tls/private/$(hostname -f).pem"
REMOTE
info "$API_HOST: SELinux port label"
ssh "$API_HOST" sudo bash -euo pipefail <<REMOTE
if semanage port -l | grep -qE "^http_port_t.*\b$API_PORT\b"; then
echo "port $API_PORT already labelled http_port_t"
else
semanage port -a -t http_port_t -p tcp "$API_PORT"
fi
REMOTE
}
role_edge() {
reachable "$EDGE_HOST" || return 0
provision_gitea_ci "$EDGE_HOST" "$(edge_sudoers)"
info "$EDGE_HOST: webroot"
ssh "$EDGE_HOST" sudo bash -euo pipefail <<REMOTE
install -d -o root -g root -m 0755 "$WEBROOT"
# An unlabelled webroot makes nginx return 403 for every file, with
# nothing in the nginx error log to explain it.
restorecon -R "$WEBROOT"
REMOTE
info "$EDGE_HOST: internal certificate for $INTERNAL_NAME"
# Minted from the internal step-ca and renewed by a templated step@ unit
# (architecture/internal-tls.md). Certs are issued with a 24-hour expiry, so
# the timer is not optional.
ssh "$EDGE_HOST" sudo bash -euo pipefail <<REMOTE
install -d -m 0755 /etc/nginx/tls/cert
install -d -m 0700 /etc/nginx/tls/key
if [ -f "/etc/nginx/tls/cert/$INTERNAL_NAME.pem" ]; then
echo "$INTERNAL_NAME certificate already present"
else
echo "MISSING: /etc/nginx/tls/cert/$INTERNAL_NAME.pem" >&2
echo "Mint it with the lair provisioner per architecture/internal-tls.md §4," >&2
echo "then re-run this script. The vhost is NOT installed until it exists:" >&2
echo "nginx -t fails on a missing ssl_certificate and blocks every reload." >&2
exit 1
fi
systemctl enable --now "step@$(basename "$INTERNAL_NAME" .internal).timer" || \
echo "step@ renewal timer not armed — renew $INTERNAL_NAME manually until it is"
REMOTE
info "$EDGE_HOST: nginx configuration"
rsync -a --rsync-path 'sudo rsync' \
"$REPO_ROOT/asset/nginx/blackbeard-upstream.conf" \
"$EDGE_HOST:/etc/nginx/conf.d/blackbeard-upstream.conf"
rsync -a --rsync-path 'sudo rsync' \
"$REPO_ROOT/asset/nginx/$PUBLIC_NAME.conf" \
"$REPO_ROOT/asset/nginx/$INTERNAL_NAME.conf" \
"$EDGE_HOST:/etc/nginx/sites-available/"
ssh "$EDGE_HOST" sudo bash -euo pipefail <<REMOTE
# sites-enabled holds only symlinks, and relative ones.
ln -sfn "../sites-available/$PUBLIC_NAME.conf" "/etc/nginx/sites-enabled/$PUBLIC_NAME.conf"
ln -sfn "../sites-available/$INTERNAL_NAME.conf" "/etc/nginx/sites-enabled/$INTERNAL_NAME.conf"
# The vhosts use \$connection_upgrade for the WebSocket upgrade; without
# the map the socket silently degrades to a hanging request and the page
# shows "reconnecting" forever with nothing in any log.
if ! grep -rqs 'connection_upgrade' /etc/nginx/conf.d/ /etc/nginx/nginx.conf; then
cat > /etc/nginx/conf.d/websocket-upgrade.conf <<'MAP'
map \$http_upgrade \$connection_upgrade {
default upgrade;
'' close;
}
MAP
echo "installed the connection_upgrade map"
else
echo "connection_upgrade map already present"
fi
# nginx -t parses without binding, so it catches syntax and missing
# certs but not a port owned across http{} and stream{}. Verify the
# reload landed rather than trusting the test.
nginx -t
systemctl reload nginx
sleep 1
systemctl is-active --quiet nginx || { echo "nginx did not come back after reload" >&2; exit 1; }
REMOTE
cat <<EOF
$EDGE_HOST is configured, but two steps are NOT automated here:
1. The public certificate for $PUBLIC_NAME (Let's Encrypt, certbot,
Cloudflare DNS-01) — see architecture/external-tls.md.
2. Split-horizon DNS for $INTERNAL_NAME, on BOTH site routers. A record on
only one router NXDOMAINs everywhere else:
for site in hanzalova kosherinata; do
opn-cli --config ~/.opn-cli/\$site.yml unbound host create \\
--hostname blackbeard --domain internal --rr A --server <edge mesh ip>
done
`create` only saves; POST /api/unbound/service/reconfigure on each router
to apply.
A public DNS record for $PUBLIC_NAME goes in Cloudflare, unproxied, CNAMEd to
the site indirection name rather than carrying a site address directly —
architecture/public-dns.md.
EOF
}
role_database() {
reachable "$PG_PRIMARY" || return 0
info "$PG_PRIMARY: roles and database"
# Roles and databases are created on the PRIMARY only — replication carries
# them to the standby.
ssh "$PG_PRIMARY" sudo -u postgres bash -euo pipefail <<REMOTE
if psql -tAc "select 1 from pg_database where datname = '$DB_NAME'" | grep -q 1; then
echo "database $DB_NAME already exists"
else
# createdb, not the bootstrap SQL: create database cannot run inside
# a transaction or a DO block.
psql -c "create role $DB_ROLE with login" || true
createdb -O "$DB_ROLE" "$DB_NAME"
fi
REMOTE
rsync -a --rsync-path 'sudo rsync' "$REPO_ROOT/asset/sql/bootstrap.sql" \
"$PG_PRIMARY:/tmp/blackbeard-bootstrap.sql"
ssh "$PG_PRIMARY" "sudo -u postgres psql -v ON_ERROR_STOP=1 -f /tmp/blackbeard-bootstrap.sql && sudo rm -f /tmp/blackbeard-bootstrap.sql"
# The CN → role mapping, on BOTH servers. pg_ident.conf contents are not
# replicated, and a failover to a server missing this mapping locks the app
# out entirely — with an authentication error that looks like a certificate
# problem.
local api_fqdn
api_fqdn=$(ssh "$API_HOST" hostname -f 2>/dev/null || echo "$API_HOST")
for server in "$PG_PRIMARY" "$PG_STANDBY"; do
reachable "$server" || continue
info "$server: pg_ident mapping for $api_fqdn -> $DB_ROLE"
printf 'cert_cn %s %s\n' "$api_fqdn" "$DB_ROLE" | \
ssh "$server" "sudo install -d -m 0700 -o postgres -g postgres /var/lib/pgsql/$PG_VERSION/data/pg_ident.conf.d && sudo tee /var/lib/pgsql/$PG_VERSION/data/pg_ident.conf.d/$api_fqdn.conf > /dev/null"
ssh "$server" "sudo chown postgres:postgres /var/lib/pgsql/$PG_VERSION/data/pg_ident.conf.d/$api_fqdn.conf"
# Reload, not restart: pg_ident is re-read on SIGHUP.
ssh "$server" "sudo systemctl reload postgresql-$PG_VERSION"
done
info "verifying the mapping from $API_HOST"
if reachable "$API_HOST"; then
ssh "$API_HOST" "sudo -u blackbeard psql 'host=$PG_PRIMARY port=5432 dbname=$DB_NAME user=$DB_ROLE sslmode=verify-full sslrootcert=/etc/pki/ca-trust/source/anchors/root-internal.pem sslcert=/etc/pki/tls/misc/\$(hostname -f).pem sslkey=/etc/pki/tls/private/\$(hostname -f).pem' -tAc 'select current_user'" \
|| warn "the app host could not authenticate to $PG_PRIMARY — check the CN mapping and the key ACL"
fi
}
# --- run ----------------------------------------------------------------------
info "roles: $ROLES"
has_role api && role_api
has_role edge && role_edge
has_role database && role_database
info "done"

6
web/.prettierrc Normal file
View File

@@ -0,0 +1,6 @@
{
"semi": false,
"singleQuote": true,
"printWidth": 100,
"trailingComma": "all"
}

20
web/eslint.config.js Normal file
View File

@@ -0,0 +1,20 @@
import js from '@eslint/js'
import globals from 'globals'
import reactHooks from 'eslint-plugin-react-hooks'
import reactRefresh from 'eslint-plugin-react-refresh'
import tseslint from 'typescript-eslint'
export default tseslint.config(
// Generated by ts-rs from the Rust entities crate; edit the Rust, not these.
{ ignores: ['dist', 'src/api/generated'] },
{
extends: [js.configs.recommended, ...tseslint.configs.recommended],
files: ['**/*.{ts,tsx}'],
languageOptions: { ecmaVersion: 2022, globals: globals.browser },
plugins: { 'react-hooks': reactHooks, 'react-refresh': reactRefresh },
rules: {
...reactHooks.configs.recommended.rules,
'react-refresh/only-export-components': ['warn', { allowConstantExport: true }],
},
},
)

36
web/index.html Normal file
View File

@@ -0,0 +1,36 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>blackbeard.observer — Quantus mining leaderboard</title>
<meta
name="description"
content="Live miner leaderboard and network hashrate for the Quantus blockchain and its Planck testnet. Every author decoded from block headers — no registration, no opt-in."
/>
<meta name="color-scheme" content="dark" />
<!-- Painted before the stylesheet loads, so there is no white flash on a
page whose entire design is dark. -->
<meta name="theme-color" content="#0d0b09" />
<link rel="icon" href="/sigil.svg" type="image/svg+xml" />
<meta property="og:title" content="blackbeard.observer" />
<meta property="og:description" content="Live Quantus mining leaderboard and network hashrate." />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://blackbeard.observer/" />
<style>
/* Inline so the background is right on the first paint rather than after
the CSS bundle arrives. */
html { background: #0d0b09; }
</style>
</head>
<body>
<div id="root"></div>
<noscript>
blackbeard.observer streams the chain over a WebSocket and needs JavaScript. The same data is
available without it from the API at <code>/v1/chains</code>.
</noscript>
<script type="module" src="/src/main.tsx"></script>
</body>
</html>

35
web/package.json Normal file
View File

@@ -0,0 +1,35 @@
{
"name": "blackbeard-web",
"private": true,
"version": "0.1.0",
"type": "module",
"packageManager": "pnpm@10.30.3",
"scripts": {
"dev": "vite",
"build": "tsc -b && vite build",
"preview": "vite preview",
"lint": "eslint .",
"format": "prettier --write src",
"typecheck": "tsc -b --noEmit"
},
"dependencies": {
"react": "^19.1.1",
"react-dom": "^19.1.1",
"react-router-dom": "^7.8.0"
},
"devDependencies": {
"@eslint/js": "^9.32.0",
"@types/node": "^26.4.1",
"@types/react": "^19.1.9",
"@types/react-dom": "^19.1.7",
"@vitejs/plugin-react-swc": "^4.0.0",
"eslint": "^9.32.0",
"eslint-plugin-react-hooks": "^5.2.0",
"eslint-plugin-react-refresh": "^0.4.20",
"globals": "^16.3.0",
"prettier": "^3.4.2",
"typescript": "~5.8.3",
"typescript-eslint": "^8.39.0",
"vite": "^7.1.0"
}
}

1897
web/pnpm-lock.yaml generated Normal file
View File

File diff suppressed because it is too large Load Diff

7
web/public/sigil.svg Normal file
View File

@@ -0,0 +1,7 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 34 40" width="34" height="40">
<rect width="34" height="40" fill="#0d0b09"/>
<path d="M17 1 L32 7 V20 C32 29 25 35 17 39 C9 35 2 29 2 20 V7 Z" fill="none" stroke="#bd8829" stroke-width="1.6"/>
<path d="M17 8 L24 12 V21 C24 26 20.5 29.5 17 31.5 C13.5 29.5 10 26 10 21 V12 Z" fill="#bd882922"/>
<path d="M17 11 V28" stroke="#d8453a" stroke-width="1.6"/>
<path d="M13 15 L17 11 L21 15" fill="none" stroke="#d8453a" stroke-width="1.6"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 503 B

214
web/src/App.tsx Normal file
View File

@@ -0,0 +1,214 @@
/**
* The page.
*
* One route, effectively: a chain, a window, and optionally one miner in focus.
* State lives in the URL hash rather than a router, because the whole
* navigable surface is `#/planck/day` and `#/miner/0x…` — pulling in a router
* to parse two segments would be more code than parsing them.
*/
import { useCallback, useEffect, useMemo, useState } from 'react'
import type { Window as WindowName } from './api/generated/Window'
import { BlockTicker } from './components/BlockTicker'
import { Leaderboard } from './components/Leaderboard'
import { MinerPanel } from './components/MinerPanel'
import { StatBar } from './components/StatBar'
import { seconds, windowSpan } from './lib/format'
import { useObserver, usePinnedMiners, useWatch } from './lib/store'
const WINDOWS: { id: WindowName; label: string; blocks: number }[] = [
{ id: 'hour', label: '1h', blocks: 600 },
{ id: 'six_hours', label: '6h', blocks: 3600 },
{ id: 'day', label: '24h', blocks: 14400 },
{ id: 'week', label: '7d', blocks: 100800 },
]
interface Route {
chain: string | null
window: WindowName
miner: string | null
}
function parseHash(hash: string): Route {
const parts = hash.replace(/^#\/?/, '').split('/').filter(Boolean)
if (parts[0] === 'miner' && parts[1]) {
return { chain: null, window: 'six_hours', miner: parts[1].toLowerCase() }
}
const window = WINDOWS.find((w) => w.id === parts[1])?.id ?? 'six_hours'
return { chain: parts[0] ?? null, window, miner: null }
}
export default function App() {
const state = useObserver()
const [route, setRoute] = useState<Route>(() => parseHash(window.location.hash))
const { pinned } = usePinnedMiners()
useEffect(() => {
const onHash = () => setRoute(parseHash(window.location.hash))
window.addEventListener('hashchange', onHash)
return () => window.removeEventListener('hashchange', onHash)
}, [])
// Default to the first chain the backend reports — which is the first one in
// its config, so the operator decides what the front page shows.
const chain = route.chain ?? state.chains[0]?.id ?? null
const info = useMemo(() => state.chains.find((c) => c.id === chain) ?? null, [state.chains, chain])
useWatch(chain, route.window)
const go = useCallback((next: Partial<Route>) => {
const merged = { ...parseHash(window.location.hash), ...next }
window.location.hash = merged.miner
? `/miner/${merged.miner}`
: `/${merged.chain ?? ''}/${merged.window}`
}, [])
const selectMiner = useCallback(
(miner: string) => {
// Keep the chain and window; the miner panel opens above the board rather
// than replacing the page, so the standings stay visible behind it.
setRoute((r) => ({ ...r, miner }))
},
[],
)
const interval = state.summary?.block_interval_seconds ?? state.summary?.target_block_time_seconds ?? 6
const activeWindow = WINDOWS.find((w) => w.id === route.window) ?? WINDOWS[1]!
return (
<div className="shell">
<header className="masthead">
<div className="mark">
<Sigil />
<div>
<div className="mark-name">
blackbeard<span>.observer</span>
</div>
<div className="mark-tagline">quantus mining · this is sparta</div>
</div>
</div>
<div className="masthead-right">
{state.chains.length > 1 && (
<div className="segmented" role="group" aria-label="Chain">
{state.chains.map((c) => (
<button
key={c.id}
aria-pressed={c.id === chain}
disabled={c.status === 'awaiting'}
title={c.status === 'awaiting' ? `${c.display_name} has not launched yet` : c.display_name}
onClick={() => go({ chain: c.id, miner: null })}
>
{c.display_name}
</button>
))}
</div>
)}
<div className="segmented" role="group" aria-label="Leaderboard window">
{WINDOWS.map((w) => (
<button
key={w.id}
aria-pressed={w.id === route.window}
title={`${w.blocks.toLocaleString('en-US')} blocks — ${windowSpan(w.blocks, interval)}`}
onClick={() => go({ window: w.id, miner: null })}
>
{w.label}
</button>
))}
</div>
<div className={`status status-${state.connection}`}>
<span className="status-dot" />
{state.connection}
</div>
</div>
</header>
{info?.status === 'awaiting' && (
<div className="banner banner-warn">
<strong>{info.display_name}</strong> has not started producing blocks yet. This page will
come alive on its own the moment it does — no refresh needed.
</div>
)}
{info?.status === 'unreachable' && (
<div className="banner banner-warn">
The observer has lost contact with the {info.display_name} node. The standings below are
the last it saw.
</div>
)}
{info?.status === 'syncing' && (
<div className="banner banner-warn">
The node is still importing history. Block interval — and therefore every hashrate on this
page — is nominal until it reaches the tip.
</div>
)}
<StatBar summary={state.summary} windowName={route.window} />
{route.miner && chain && (
<MinerPanel
chain={chain}
miner={route.miner}
windowName={route.window}
onClose={() => setRoute((r) => ({ ...r, miner: null }))}
/>
)}
<div className="two-col">
<section className="panel">
<div className="panel-head">
<h2 className="panel-title">The Standings</h2>
<span className="eyebrow">
last {activeWindow.blocks.toLocaleString('en-US')} blocks ·{' '}
{windowSpan(activeWindow.blocks, interval)}
{pinned.length > 0 && ` · ${pinned.length} marked yours`}
</span>
</div>
<Leaderboard rows={state.leaderboard} ready={state.ready} onSelect={selectMiner} />
</section>
<section className="panel">
<div className="panel-head">
<h2 className="panel-title">Live Blocks</h2>
<span className="eyebrow">
{state.summary?.block_interval_seconds
? `${seconds(state.summary.block_interval_seconds)} apart`
: 'measuring'}
</span>
</div>
<BlockTicker blocks={state.blocks} onSelect={selectMiner} />
</section>
</div>
<footer className="footer">
<span>
Every miner on this board is derived from the <code>pow_</code> digest in each block
header — no registration, no opt-in, no way to be left out.
</span>
<span>
Hashrates are estimates from blocks won over a finite window, not measurements of anyone's
hardware.
</span>
</footer>
</div>
)
}
/** The mark: a spear-point shield, drawn rather than shipped as an asset. */
function Sigil() {
return (
<svg width="34" height="40" viewBox="0 0 34 40" aria-hidden="true">
<path
d="M17 1 L32 7 V20 C32 29 25 35 17 39 C9 35 2 29 2 20 V7 Z"
fill="none"
stroke="var(--data)"
strokeWidth="1.6"
/>
<path d="M17 8 L24 12 V21 C24 26 20.5 29.5 17 31.5 C13.5 29.5 10 26 10 21 V12 Z" fill="var(--data-wash)" />
<path d="M17 11 V28" stroke="var(--accent)" strokeWidth="1.6" />
<path d="M13 15 L17 11 L21 15" fill="none" stroke="var(--accent)" strokeWidth="1.6" />
</svg>
)
}

19
web/src/api/generated/ApiError.ts Normal file
View File

@@ -0,0 +1,19 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* The JSON body every failing REST response carries.
*
* Shared with the frontend so error rendering is not a guess about what the
* backend emits.
*/
export type ApiError = {
/**
* A stable machine-readable code (`unknown_chain`, `not_found`,
* `chain_unavailable`). The frontend switches on this; `message` is for
* humans and may be reworded at any time.
*/
code: string,
/**
* Human-readable explanation.
*/
message: string, };

10
web/src/api/generated/AttributionSource.ts Normal file
View File

@@ -0,0 +1,10 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* How a miner's display name was arrived at.
*
* The distinction matters on screen: a telemetry-attributed name is an
* inference from who reported a block first, not a claim the miner made, and
* the UI must not present the two identically.
*/
export type AttributionSource = "preimage" | "telemetry";

11
web/src/api/generated/BigUintDec.ts Normal file
View File

@@ -0,0 +1,11 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* An arbitrary-precision non-negative integer carried as a decimal string.
*
* Difficulty and cumulative hash counts are U512 on this chain; JSON numbers
* are IEEE-754 doubles, so putting them on the wire as numbers would silently
* round them. The string is the canonical form on the wire, in the database
* (`numeric`), and in the browser — the frontend formats it with `BigInt`.
*/
export type BigUintDec = string;

45
web/src/api/generated/BlockObservation.ts Normal file
View File

@@ -0,0 +1,45 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { BigUintDec } from "./BigUintDec";
import type { ChainId } from "./ChainId";
import type { MinerId } from "./MinerId";
/**
* A block as the observer recorded it.
*
* Two timestamps, deliberately: `authored_at` comes from the block's own
* `Timestamp::set` inherent, which the author stamps when it *builds* the
* proposal, while `observed_at` is when this observer first saw the block.
* The difference is propagation plus poll lag when negative, and a
* future-dated block when positive. A miner whose blocks arrive in bursts
* right after someone else's, with `authored_at` well before `observed_at`,
* is withholding them.
*/
export type BlockObservation = {
/**
* Which chain.
*/
chain: ChainId,
/**
* Block number.
*/
height: number,
/**
* Block hash, `0x`-prefixed.
*/
hash: string,
/**
* Author's reward preimage from the `pow_` PreRuntime digest.
*/
miner: MinerId,
/**
* The author's own timestamp from the block's timestamp inherent.
*/
authored_at: string | null,
/**
* When this observer first saw the block.
*/
observed_at: string,
/**
* Difficulty in force for this block: expected hashes to win it.
*/
difficulty: BigUintDec | null, };

11
web/src/api/generated/ChainId.ts Normal file
View File

@@ -0,0 +1,11 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* Short, stable, URL-safe name for a chain (`planck`, `quantus`).
*
* This is the operator's name for the chain, not the genesis hash: it appears
* in routes, in config, and in the WebSocket protocol, and it has to survive a
* chain being specced before it launches. The genesis hash is discovered from
* the node at runtime and carried in [`ChainInfo`].
*/
export type ChainId = string;

43
web/src/api/generated/ChainInfo.ts Normal file
View File

@@ -0,0 +1,43 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { ChainId } from "./ChainId";
import type { ChainStatus } from "./ChainStatus";
/**
* Static-ish facts about a chain: what to call it and how to read its numbers.
*/
export type ChainInfo = {
/**
* Operator name, used in routes and the WS protocol.
*/
id: ChainId,
/**
* What a human should see: "Quantus", "Planck Testnet".
*/
display_name: string,
/**
* True for the production network, false for a testnet. Drives whether the
* UI treats rewards as real.
*/
mainnet: boolean,
/**
* Genesis hash, discovered from the node. `None` until it first answers.
*/
genesis: string | null,
/**
* Token ticker from `system_properties`, e.g. `QUAN`.
*/
token_symbol: string | null,
/**
* Token decimals from `system_properties`. Balances are integers of the
* smallest unit; this is how many places to shift them.
*/
token_decimals: number | null,
/**
* Target seconds per block the chain retargets towards. Used as the
* hashrate denominator whenever a measured interval is not yet available.
*/
target_block_time_seconds: number,
/**
* Current reachability.
*/
status: ChainStatus, };

10
web/src/api/generated/ChainStatus.ts Normal file
View File

@@ -0,0 +1,10 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* How much of a chain the observer can currently see.
*
* A chain that is configured but not yet launched is a first-class state, not
* an error: mainnet exists in config before it exists on the network, and the
* UI has to say "not yet live" rather than "backend broken".
*/
export type ChainStatus = "live" | "syncing" | "awaiting" | "unreachable";

69
web/src/api/generated/ChainSummary.ts Normal file
View File

@@ -0,0 +1,69 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { BigUintDec } from "./BigUintDec";
import type { ChainId } from "./ChainId";
/**
* The live headline numbers for one chain: one row of the scoreboard's top.
*/
export type ChainSummary = {
/**
* Which chain this describes.
*/
chain: ChainId,
/**
* Best block height seen.
*/
height: number | null,
/**
* Current mining difficulty: expected hashes per block.
*/
difficulty: BigUintDec | null,
/**
* The ceiling difficulty can reach, from `QPoWApi_get_max_difficulty`.
* The ratio of the two is how much headroom the network has left.
*/
max_difficulty: BigUintDec | null,
/**
* Estimated network hashrate in hashes per second: difficulty divided by
* the block interval.
*/
network_hashrate: number | null,
/**
* True when `network_hashrate` used the configured target rather than a
* measured interval. The figure is nominal — say so in the UI rather than
* presenting an estimate as an observation.
*/
hashrate_from_target: boolean,
/**
* Measured mean seconds per block at the tip, when enough tip samples
* exist. `None` while syncing or just after a restart.
*/
block_interval_seconds: number | null,
/**
* The configured target, for comparison against the measured interval.
*/
target_block_time_seconds: number,
/**
* Blocks in the rolling leaderboard window.
*/
window_blocks: number,
/**
* Distinct reward preimages that authored in the window. A **lower bound**
* on miner count: one operator may run several preimages, and several
* operators could in principle share one.
*/
distinct_miners: number,
/**
* Nodes on this chain per the substrate-telemetry feed, when connected.
* Compare against `distinct_miners`: node count flat while distinct
* authors falls is small miners still running and simply being out-hashed.
*/
telemetry_nodes: number | null,
/**
* Whether the telemetry feed is currently connected.
*/
telemetry_connected: boolean,
/**
* When these numbers were computed.
*/
updated_at: string, };

20
web/src/api/generated/ClientMessage.ts Normal file
View File

@@ -0,0 +1,20 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { ChainId } from "./ChainId";
import type { Window } from "./Window";
/**
* Browser to server.
*/
export type ClientMessage = { "type": "subscribe",
/**
* Chain to watch.
*/
chain: ChainId,
/**
* Leaderboard window.
*/
window: Window, } | { "type": "unsubscribe",
/**
* Chain to drop.
*/
chain: ChainId, } | { "type": "ping" };

58
web/src/api/generated/LeaderboardRow.ts Normal file
View File

@@ -0,0 +1,58 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { AttributionSource } from "./AttributionSource";
import type { MinerId } from "./MinerId";
/**
* One row of the leaderboard.
*/
export type LeaderboardRow = {
/**
* 1-based position within the window. Ties break by most recent block.
*/
rank: number,
/**
* The miner's reward preimage — the stable key for this row.
*/
miner: MinerId,
/**
* What to render: a telemetry node name when one is attributed, else the
* abbreviated preimage.
*/
display: string,
/**
* Where `display` came from.
*/
attribution: AttributionSource,
/**
* Fraction of attributed blocks pointing at the named node, 0.0–1.0. Zero
* when `attribution` is `Preimage`.
*/
confidence: number,
/**
* Blocks authored inside the window.
*/
blocks: number,
/**
* Share of the window's blocks, 0.0–1.0.
*/
share: number,
/**
* Implied hashrate in hashes per second: the miner's share of blocks times
* the estimated network hashrate. A statistical estimate over a finite
* window, not a measurement of the miner's hardware — a lucky small miner
* will over-read and an unlucky large one will under-read.
*/
hashrate_estimate: number | null,
/**
* Height of this miner's most recent block in the window.
*/
last_block_height: number,
/**
* When the observer saw that block.
*/
last_seen: string,
/**
* Longest run of consecutive blocks by this miner within the window. A run
* far above what its share predicts is the signature of withheld blocks.
*/
best_streak: number, };

48
web/src/api/generated/MinerDetail.ts Normal file
View File

@@ -0,0 +1,48 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { BigUintDec } from "./BigUintDec";
import type { ChainId } from "./ChainId";
import type { LeaderboardRow } from "./LeaderboardRow";
import type { MinerId } from "./MinerId";
import type { MinerSeriesPoint } from "./MinerSeriesPoint";
/**
* Everything the miner detail page shows for one preimage.
*/
export type MinerDetail = {
/**
* Which chain.
*/
chain: ChainId,
/**
* The miner.
*/
miner: MinerId,
/**
* Current leaderboard row, absent if the miner authored nothing in the
* window (a small miner between blocks, or one that has stopped).
*/
current: LeaderboardRow | null,
/**
* Blocks authored since the observer's records begin — not since chain
* genesis, unless the observer has watched from genesis.
*/
blocks_observed: number,
/**
* First block the observer attributed to this miner.
*/
first_seen: string | null,
/**
* Most recent.
*/
last_seen: string | null,
/**
* Bucketed history for charting, oldest first.
*/
series: Array<MinerSeriesPoint>,
/**
* Sum of the difficulty of every block this miner authored in the observed
* record: the expected number of hashes it took to win them. The honest
* cumulative measure of work done, and the one that does not shrink when
* difficulty rises.
*/
cumulative_work: BigUintDec, };

18
web/src/api/generated/MinerId.ts Normal file
View File

@@ -0,0 +1,18 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* A miner's on-chain identity: the 32-byte wormhole reward **preimage**,
* `0x`-prefixed lowercase hex.
*
* This is the only miner identity the chain offers without cooperation. Every
* block header carries its author's preimage in a `PreRuntime` digest with
* engine id `pow_`, so authorship for *every* miner on the network is
* derivable from headers alone — no indexer, no registration, no opt-in.
*
* It is **public data, not a secret**: it is published from a miner's first
* authored block onward, and spending the rewards needs a plonky2 proof of
* knowledge of the underlying secret, which the preimage does not reveal. It
* is also **static per wallet**, which is exactly what makes a leaderboard
* possible — and what makes mining income permanently attributable.
*/
export type MinerId = string;

22
web/src/api/generated/MinerSeriesPoint.ts Normal file
View File

@@ -0,0 +1,22 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* A point on a miner's history chart.
*/
export type MinerSeriesPoint = {
/**
* Start of the bucket.
*/
at: string,
/**
* Blocks authored inside the bucket.
*/
blocks: number,
/**
* Share of all blocks in the bucket, 0.0–1.0.
*/
share: number,
/**
* Implied hashrate over the bucket, hashes per second.
*/
hashrate_estimate: number | null, };

34
web/src/api/generated/RecentBlock.ts Normal file
View File

@@ -0,0 +1,34 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { MinerId } from "./MinerId";
/**
* A block trimmed for the live ticker: what scrolls past on the front page.
*
* Separate from [`BlockObservation`] because the ticker is pushed on every
* block to every connected browser, and shipping the full record — including
* the difficulty string, which changes rarely — would multiply the socket's
* bandwidth for nothing.
*/
export type RecentBlock = {
/**
* Block number.
*/
height: number,
/**
* Author's reward preimage.
*/
miner: MinerId,
/**
* Display name for the author at the time the block was pushed.
*/
display: string,
/**
* When this observer first saw it.
*/
observed_at: string,
/**
* Seconds since the previous block, by observation. `None` for the first
* block after a restart, where there is no previous observation to
* subtract.
*/
gap_seconds: number | null, };

76
web/src/api/generated/ServerMessage.ts Normal file
View File

@@ -0,0 +1,76 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { ChainId } from "./ChainId";
import type { ChainInfo } from "./ChainInfo";
import type { ChainSummary } from "./ChainSummary";
import type { LeaderboardRow } from "./LeaderboardRow";
import type { RecentBlock } from "./RecentBlock";
import type { Window } from "./Window";
/**
* Server to browser.
*/
export type ServerMessage = { "type": "chains",
/**
* Every configured chain.
*/
chains: Array<ChainInfo>, } | { "type": "snapshot",
/**
* Which chain.
*/
chain: ChainId,
/**
* Window this snapshot's leaderboard covers.
*/
window: Window,
/**
* Headline numbers.
*/
summary: ChainSummary,
/**
* Full standings for the window.
*/
leaderboard: Array<LeaderboardRow>,
/**
* The tail of the block ticker, newest last.
*/
recent_blocks: Array<RecentBlock>, } | { "type": "block",
/**
* Which chain.
*/
chain: ChainId,
/**
* The block.
*/
block: RecentBlock, } | { "type": "summary",
/**
* Which chain.
*/
chain: ChainId,
/**
* New numbers.
*/
summary: ChainSummary, } | { "type": "leaderboard",
/**
* Which chain.
*/
chain: ChainId,
/**
* Window these standings cover.
*/
window: Window,
/**
* The standings.
*/
rows: Array<LeaderboardRow>, } | { "type": "chain_status",
/**
* Which chain.
*/
chain: ChainId,
/**
* Its new state.
*/
info: ChainInfo, } | { "type": "pong" } | { "type": "error",
/**
* Human-readable reason.
*/
message: string, };

12
web/src/api/generated/Window.ts Normal file
View File

@@ -0,0 +1,12 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* How far back a leaderboard looks.
*
* A **block count**, never a duration. A time window is meaningless while a
* node is catching up: it imports historical blocks at disk speed, so "the
* last hour" can contain half a million blocks. Block counts stay honest in
* both regimes, and the UI renders the approximate duration beside them from
* the measured interval.
*/
export type Window = "hour" | "six_hours" | "day" | "week";

53
web/src/api/rest.ts Normal file
View File

@@ -0,0 +1,53 @@
/**
* The few things the socket does not push.
*
* A miner's own history is a query, not a stream — it is asked for once when
* someone opens their page and does not change until the next block they win.
* Pushing it to every connected tab would be strictly worse.
*/
import type { ApiError } from './generated/ApiError'
import type { MinerDetail } from './generated/MinerDetail'
import type { Window as WindowName } from './generated/Window'
const BASE = import.meta.env.VITE_API_BASE_URL ?? '/v1'
/** An error the API described, as opposed to a network failure. */
export class RequestFailed extends Error {
constructor(
readonly status: number,
readonly code: string,
message: string,
) {
super(message)
this.name = 'RequestFailed'
}
}
async function get<T>(path: string, signal?: AbortSignal): Promise<T> {
const response = await fetch(`${BASE}${path}`, signal ? { signal } : {})
if (!response.ok) {
// The API answers failures with a typed body; fall back to the status text
// for anything that got as far as nginx but not as far as the API.
const body = (await response.json().catch(() => null)) as ApiError | null
throw new RequestFailed(
response.status,
body?.code ?? 'http_error',
body?.message ?? response.statusText,
)
}
return (await response.json()) as T
}
/** One miner's standing and history. */
export function fetchMiner(
chain: string,
miner: string,
windowName: WindowName,
signal?: AbortSignal,
): Promise<MinerDetail> {
return get<MinerDetail>(
`/chains/${encodeURIComponent(chain)}/miners/${encodeURIComponent(miner)}?window=${windowName}`,
signal,
)
}

248
web/src/api/socket.ts Normal file
View File

@@ -0,0 +1,248 @@
/**
* The live connection.
*
* One WebSocket for the whole tab. It reconnects on its own, resubscribes to
* whatever the page was watching, and hands React a snapshot object that only
* changes identity when something in it actually changed — so `useSyncExternalStore`
* re-renders exactly the components whose data moved.
*
* Why not observables: this needs one stream, one reducer and one subscriber
* list. RxJS would add a dependency and an idiom to every component to express
* what `useSyncExternalStore` already does natively, and React's own store
* contract is the thing that gets concurrent rendering right.
*/
import type { ChainInfo } from './generated/ChainInfo'
import type { ChainSummary } from './generated/ChainSummary'
import type { ClientMessage } from './generated/ClientMessage'
import type { LeaderboardRow } from './generated/LeaderboardRow'
import type { RecentBlock } from './generated/RecentBlock'
import type { ServerMessage } from './generated/ServerMessage'
import type { Window as WindowName } from './generated/Window'
/** How the socket is doing, for the status light in the header. */
export type Connection = 'connecting' | 'live' | 'reconnecting' | 'offline'
/** Everything the UI renders, in one immutable object. */
export interface ObserverState {
connection: Connection
/** Every configured chain, in the order the backend lists them. */
chains: ChainInfo[]
/** Which chain the page is watching. */
chain: string | null
/** Which leaderboard window. */
window: WindowName
summary: ChainSummary | null
leaderboard: LeaderboardRow[]
/** Newest first — the order the ticker renders in. */
blocks: RecentBlock[]
/** True once the first snapshot for the current subscription has landed. */
ready: boolean
}
const INITIAL: ObserverState = {
connection: 'connecting',
chains: [],
chain: null,
window: 'six_hours',
summary: null,
leaderboard: [],
blocks: [],
ready: false,
}
/** Blocks kept in the ticker. Matches the backend's replay length. */
const TICKER_LIMIT = 40
/**
* Reconnect backoff. Capped low: this is a live scoreboard, and a miner
* watching their standing would rather the page retry briskly than back off to
* a minute after a brief network blip.
*/
const RECONNECT_MIN_MS = 1_000
const RECONNECT_MAX_MS = 15_000
/**
* Application-level keepalive.
*
* Browsers cannot send WebSocket ping frames from JavaScript, so a socket that
* has been silently dropped by an intermediary looks identical to a quiet chain
* — and on a chain averaging a block every few seconds, "quiet" is a real
* state. This ping is the only way the tab can tell the two apart.
*/
const PING_INTERVAL_MS = 25_000
function socketUrl(): string {
const configured = import.meta.env.VITE_WS_URL
if (configured) return configured
// Same origin by default: in production nginx serves the bundle and proxies
// /v1 to the API, and in development Vite proxies it. Deriving the URL means
// no build-time value to get wrong per environment.
const protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:'
return `${protocol}//${window.location.host}/v1/ws`
}
/**
* The store. One instance per tab, created in `main.tsx`.
*/
export class Observer {
private state: ObserverState = INITIAL
private listeners = new Set<() => void>()
private socket: WebSocket | null = null
private reconnectDelay = RECONNECT_MIN_MS
private reconnectTimer: number | null = null
private pingTimer: number | null = null
private closed = false
/** `useSyncExternalStore` subscribe. */
subscribe = (listener: () => void): (() => void) => {
this.listeners.add(listener)
return () => {
this.listeners.delete(listener)
}
}
/**
* `useSyncExternalStore` snapshot.
*
* Must return a stable reference between changes — returning a fresh object
* here would re-render every subscriber on every tick and, in React 19, throw
* for an unstable snapshot.
*/
getSnapshot = (): ObserverState => this.state
/** Open the socket. Idempotent. */
connect(): void {
if (this.socket || this.closed) return
const socket = new WebSocket(socketUrl())
this.socket = socket
socket.onopen = () => {
this.reconnectDelay = RECONNECT_MIN_MS
this.patch({ connection: 'live' })
// The server does not remember subscriptions across sockets, so the
// reconnect has to restate what this tab is watching. Without it a
// recovered connection would deliver the chain list and then nothing.
if (this.state.chain) {
this.send({ type: 'subscribe', chain: this.state.chain, window: this.state.window })
}
this.pingTimer = window.setInterval(() => this.send({ type: 'ping' }), PING_INTERVAL_MS)
}
socket.onmessage = (event: MessageEvent<string>) => {
let message: ServerMessage
try {
message = JSON.parse(event.data) as ServerMessage
} catch {
return
}
this.apply(message)
}
socket.onclose = () => {
this.socket = null
if (this.pingTimer !== null) {
window.clearInterval(this.pingTimer)
this.pingTimer = null
}
if (this.closed) return
this.patch({ connection: 'reconnecting', ready: false })
this.reconnectTimer = window.setTimeout(() => this.connect(), this.reconnectDelay)
this.reconnectDelay = Math.min(this.reconnectDelay * 2, RECONNECT_MAX_MS)
}
// `onerror` is always followed by `onclose`, so reconnection is handled
// there and this only exists to stop the browser logging an unhandled one.
socket.onerror = () => {}
}
/** Close for good. Used when the app unmounts. */
disconnect(): void {
this.closed = true
if (this.reconnectTimer !== null) window.clearTimeout(this.reconnectTimer)
if (this.pingTimer !== null) window.clearInterval(this.pingTimer)
this.socket?.close()
this.socket = null
}
/** Watch a chain at a window. Safe to call before the socket is open. */
watch(chain: string, windowName: WindowName): void {
if (this.state.chain === chain && this.state.window === windowName) return
// Clear the previous chain's data rather than letting it linger under the
// new chain's name: a stale leaderboard under a different heading is worse
// than an empty one, because it looks correct.
this.patch({
chain,
window: windowName,
summary: null,
leaderboard: [],
blocks: [],
ready: false,
})
this.send({ type: 'subscribe', chain, window: windowName })
}
private send(message: ClientMessage): void {
if (this.socket?.readyState === WebSocket.OPEN) {
this.socket.send(JSON.stringify(message))
}
}
private apply(message: ServerMessage): void {
switch (message.type) {
case 'chains':
this.patch({ chains: message.chains })
break
case 'snapshot':
// A snapshot for a chain the page has since navigated away from would
// overwrite the current one; ignore it rather than flicker.
if (message.chain !== this.state.chain) return
this.patch({
summary: message.summary,
leaderboard: message.leaderboard,
blocks: [...message.recent_blocks].reverse(),
ready: true,
})
break
case 'summary':
if (message.chain !== this.state.chain) return
this.patch({ summary: message.summary })
break
case 'leaderboard':
if (message.chain !== this.state.chain || message.window !== this.state.window) return
this.patch({ leaderboard: message.rows })
break
case 'block': {
if (message.chain !== this.state.chain) return
// A reorg replaces the block at a height rather than appending a second
// one, so the ticker matches the chain rather than accumulating both
// sides of every fork.
const rest = this.state.blocks.filter((b) => b.height !== message.block.height)
this.patch({ blocks: [message.block, ...rest].slice(0, TICKER_LIMIT) })
break
}
case 'chain_status':
this.patch({
chains: this.state.chains.map((c) => (c.id === message.chain ? message.info : c)),
})
break
case 'error':
console.warn('[observer]', message.message)
break
case 'pong':
break
}
}
private patch(next: Partial<ObserverState>): void {
this.state = { ...this.state, ...next }
for (const listener of this.listeners) listener()
}
}

87
web/src/components/BlockTicker.tsx Normal file
View File

@@ -0,0 +1,87 @@
/**
* The live block feed.
*
* The site's proof that it is live: blocks appear here the moment the node
* imports them, with no refresh and no polling. Each row flashes once on
* arrival — suppressed under `prefers-reduced-motion`, where the numbers alone
* still carry everything.
*/
import { useEffect, useRef, useState } from 'react'
import type { RecentBlock } from '../api/generated/RecentBlock'
import { ago, height, seconds, shortMiner } from '../lib/format'
import { usePinnedMiners } from '../lib/store'
export function BlockTicker({
blocks,
onSelect,
}: {
blocks: RecentBlock[]
onSelect: (miner: string) => void
}) {
const { isPinned } = usePinnedMiners()
const seen = useRef<Set<number>>(new Set())
// `ago()` is a pure function of the clock, so nothing re-renders it on its
// own. One interval for the whole list, rather than a timer per row.
const [, tick] = useState(0)
useEffect(() => {
const id = window.setInterval(() => tick((n) => n + 1), 1000)
return () => window.clearInterval(id)
}, [])
const fresh = new Set<number>()
for (const block of blocks) {
if (!seen.current.has(block.height)) fresh.add(block.height)
}
useEffect(() => {
for (const block of blocks) seen.current.add(block.height)
})
if (blocks.length === 0) {
return <p className="empty">Waiting for the next block…</p>
}
return (
<div>
{blocks.map((block) => {
const mine = isPinned(block.miner)
const named = block.display !== shortMiner(block.miner)
return (
<div
key={block.height}
className={[
'ticker-row',
mine ? 'mine' : '',
fresh.has(block.height) ? 'fresh' : '',
]
.filter(Boolean)
.join(' ')}
>
<span className="ticker-height">#{height(block.height)}</span>
<a
href={`#/miner/${block.miner}`}
title={block.miner}
onClick={(e) => {
e.preventDefault()
onSelect(block.miner)
}}
style={{
overflow: 'hidden',
textOverflow: 'ellipsis',
fontFamily: named ? 'inherit' : 'var(--font-mono)',
fontSize: named ? 'inherit' : '12px',
color: named ? 'var(--text-primary)' : 'var(--text-secondary)',
}}
>
{block.display}
</a>
<span className="ticker-gap">
{block.gap_seconds !== null ? `+${seconds(block.gap_seconds)}` : ago(block.observed_at)}
</span>
</div>
)
})}
</div>
)
}

155
web/src/components/Leaderboard.tsx Normal file
View File

@@ -0,0 +1,155 @@
/**
* The standings.
*
* A table, not a chart. Eight-plus miners each carrying a name, a count, a
* share, a hashrate and a streak is exactly the case the form heuristic sends
* to a table — and a miner looking for their own row wants to *read* it, not
* hunt for a colour in a legend.
*
* The one graphical element is the share bar: single hue, magnitude only. A
* pinned miner's row is marked three ways at once — a left rule, a wash, and an
* explicit "YOU" chip — so identity never rests on colour alone.
*/
import type { LeaderboardRow } from '../api/generated/LeaderboardRow'
import { ago, hashrate, share, shortMiner } from '../lib/format'
import { usePinnedMiners } from '../lib/store'
function Row({
row,
max,
mine,
onToggle,
onSelect,
}: {
row: LeaderboardRow
max: number
mine: boolean
onToggle: (miner: string) => void
onSelect: (miner: string) => void
}) {
const named = row.attribution === 'telemetry'
return (
<tr className={mine ? 'mine' : undefined}>
<td className={`rank rank-${row.rank}`}>{row.rank}</td>
<td className="left">
<div className="miner-cell">
<button
className="pin-button"
aria-pressed={mine}
title={mine ? 'Remove from your miners' : 'Mark this miner as yours'}
onClick={() => onToggle(row.miner)}
>
{mine ? '★' : '☆'}
</button>
<a
href={`#/miner/${row.miner}`}
className={named ? 'miner-name' : 'miner-name anonymous'}
title={row.miner}
onClick={(e) => {
e.preventDefault()
onSelect(row.miner)
}}
>
{named ? row.display : shortMiner(row.miner)}
</a>
{mine && <span className="chip chip-you">You</span>}
{named && (
<span
className="chip chip-named"
// An attributed name is an inference from who reported the block
// first, not a claim the miner made. The chip and the confidence
// in its tooltip keep that visible rather than presenting a guess
// as a fact.
title={`Name inferred from telemetry — ${Math.round(row.confidence * 100)}% of recent blocks agree`}
>
node
</span>
)}
</div>
</td>
<td className="numeral">{row.blocks}</td>
{/* Bar and number in one cell: two columns for one quantity cost ~80px
and pushed the streak and last-block columns off the table at ordinary
window widths. The bar is the magnitude, the number is its direct
label — they belong together. */}
<td className="share-cell">
<span className="numeral">{share(row.share)}</span>
<div className="share-bar" aria-hidden="true">
<i style={{ width: `${max > 0 ? (row.share / max) * 100 : 0}%` }} />
</div>
</td>
<td className="numeral">{hashrate(row.hashrate_estimate)}</td>
<td className="numeral streak" title="Longest run of consecutive blocks in this window">
{row.best_streak > 1 ? `×${row.best_streak}` : '—'}
</td>
<td className="numeral last-seen" style={{ color: 'var(--text-muted)' }}>
{ago(row.last_seen)}
</td>
</tr>
)
}
export function Leaderboard({
rows,
ready,
onSelect,
}: {
rows: LeaderboardRow[]
ready: boolean
onSelect: (miner: string) => void
}) {
const { isPinned, toggle } = usePinnedMiners()
// Bars are scaled to the leader, not to 100%: with twenty miners the top
// share is often under 15%, and full-scale bars would all be slivers.
const max = rows.length > 0 ? Math.max(...rows.map((r) => r.share)) : 0
if (rows.length === 0) {
return (
<p className="empty">
{ready
? 'No blocks observed in this window yet. A shorter window may show more.'
: 'Waiting for the first snapshot…'}
</p>
)
}
return (
<div className="scroll-x">
<table className="board">
<caption className="visually-hidden">
Miners ranked by blocks won in the selected window
</caption>
<thead>
<tr>
<th scope="col">#</th>
<th scope="col" className="left">
Miner
</th>
<th scope="col">Blocks</th>
<th scope="col">Share</th>
<th scope="col">Est. hashrate</th>
<th scope="col" className="streak">
Streak
</th>
<th scope="col" className="last-seen">
Last block
</th>
</tr>
</thead>
<tbody>
{rows.map((row) => (
<Row
key={row.miner}
row={row}
max={max}
mine={isPinned(row.miner)}
onToggle={toggle}
onSelect={onSelect}
/>
))}
</tbody>
</table>
</div>
)
}

139
web/src/components/MinerPanel.tsx Normal file
View File

@@ -0,0 +1,139 @@
/**
* One miner's page.
*
* The reason the site exists: a miner types (or clicks) their reward preimage
* and sees where they stand and how they have been doing. Everything here is
* public chain data — there is nothing to log into, and the "this one is mine"
* mark lives only in this browser.
*/
import { useEffect, useState } from 'react'
import type { MinerDetail } from '../api/generated/MinerDetail'
import type { Window as WindowName } from '../api/generated/Window'
import { RequestFailed, fetchMiner } from '../api/rest'
import { ago, bigNumber, hashrate, height, share, shortMiner } from '../lib/format'
import { usePinnedMiners } from '../lib/store'
import { ShareChart } from './ShareChart'
export function MinerPanel({
chain,
miner,
windowName,
onClose,
}: {
chain: string
miner: string
windowName: WindowName
onClose: () => void
}) {
const [detail, setDetail] = useState<MinerDetail | null>(null)
const [error, setError] = useState<string | null>(null)
const { isPinned, toggle } = usePinnedMiners()
const mine = isPinned(miner)
useEffect(() => {
const controller = new AbortController()
setDetail(null)
setError(null)
fetchMiner(chain, miner, windowName, controller.signal)
.then(setDetail)
.catch((e: unknown) => {
// An abort is this component being replaced, not a failure to report.
if (controller.signal.aborted) return
setError(e instanceof RequestFailed ? e.message : 'Could not reach the observer.')
})
return () => controller.abort()
// Re-fetched on window change because the history range follows the window.
}, [chain, miner, windowName])
const row = detail?.current ?? null
const named = row?.attribution === 'telemetry'
return (
<section className="panel" style={{ marginBottom: 26 }}>
<div className="panel-head">
<div>
<div className="eyebrow">Miner</div>
<h2 className="panel-title" style={{ marginTop: 4 }}>
{named ? row.display : shortMiner(miner)}
</h2>
<div
className="numeral"
style={{ fontSize: 11, color: 'var(--text-muted)', marginTop: 4, overflowWrap: 'anywhere' }}
>
{miner}
</div>
</div>
<div style={{ display: 'flex', gap: 10, alignItems: 'center' }}>
<button
className="segmented"
style={{ padding: '7px 13px', font: 'inherit', fontSize: 12, letterSpacing: '0.1em', textTransform: 'uppercase', color: mine ? 'var(--accent-bright)' : 'var(--text-secondary)', cursor: 'pointer' }}
aria-pressed={mine}
onClick={() => toggle(miner)}
>
{mine ? '★ Yours' : '☆ Mark as yours'}
</button>
<button
className="segmented"
style={{ padding: '7px 13px', font: 'inherit', fontSize: 12, letterSpacing: '0.1em', textTransform: 'uppercase', color: 'var(--text-secondary)', cursor: 'pointer' }}
onClick={onClose}
>
Close
</button>
</div>
</div>
{error && <p className="empty">{error}</p>}
{!error && !detail && <p className="empty">Reading the record…</p>}
{detail && (
<>
<div className="stats" style={{ margin: 0, border: 0, borderBottom: 'var(--rule)' }}>
<div className="stat">
<div className="stat-label">Rank</div>
<div className="stat-value">{row ? `#${row.rank}` : '—'}</div>
<div className="stat-note">
{row ? `${share(row.share)} of the window` : 'no blocks in this window'}
</div>
</div>
<div className="stat">
<div className="stat-label">Est. hashrate</div>
<div className="stat-value">{hashrate(row?.hashrate_estimate)}</div>
<div className="stat-note">from share of blocks won</div>
</div>
<div className="stat">
<div className="stat-label">Blocks observed</div>
<div className="stat-value">{height(detail.blocks_observed)}</div>
<div className="stat-note">
{detail.first_seen ? `first seen ${ago(detail.first_seen)}` : 'never seen'}
</div>
</div>
<div className="stat">
<div className="stat-label">Cumulative work</div>
<div className="stat-value" title={detail.cumulative_work}>
{bigNumber(detail.cumulative_work)}
</div>
{/* Difficulty summed over every block won. It does not shrink when
difficulty rises, which a block count effectively does. */}
<div className="stat-note">expected hashes, all blocks</div>
</div>
<div className="stat">
<div className="stat-label">Best streak</div>
<div className="stat-value">{row && row.best_streak > 1 ? `×${row.best_streak}` : '—'}</div>
<div className="stat-note">consecutive blocks</div>
</div>
</div>
<div style={{ padding: '16px 18px 18px' }}>
<div className="eyebrow" style={{ marginBottom: 10 }}>
Share of network blocks
</div>
<ShareChart points={detail.series} />
</div>
</>
)}
</section>
)
}

211
web/src/components/ShareChart.tsx Normal file
View File

@@ -0,0 +1,211 @@
/**
* One miner's share of the network over time.
*
* A single-series area chart: the job is change-over-time for one quantity, so
* there is one hue and no legend — the panel title already says what is
* plotted, and a legend box with one swatch would restate it.
*
* Inline SVG rather than a charting library. The whole chart is one path, one
* fill and two axes; a library would be a hundred kilobytes and a second set of
* theming rules to keep in step with the CSS custom properties above it.
*
* The viewBox tracks the container's measured width rather than being fixed and
* scaled to fit. A fixed viewBox stretched to a full-width panel scales its
* *height* with it — a 190-unit chart became 300px tall on a wide screen, and
* the area fill went from a wash to a slab. Measuring keeps one SVG unit at one
* CSS pixel, which also makes the crosshair land exactly under the pointer.
*/
import { useEffect, useMemo, useRef, useState } from 'react'
import type { MinerSeriesPoint } from '../api/generated/MinerSeriesPoint'
import { hashrate, share } from '../lib/format'
const HEIGHT = 200
const PAD = { top: 16, right: 18, bottom: 26, left: 46 }
/** Used until the container has been measured, and if ResizeObserver is absent. */
const FALLBACK_WIDTH = 720
interface Placed {
point: MinerSeriesPoint
x: number
y: number
}
function useMeasuredWidth(): [React.RefObject<HTMLDivElement | null>, number] {
const ref = useRef<HTMLDivElement>(null)
const [width, setWidth] = useState(FALLBACK_WIDTH)
useEffect(() => {
const node = ref.current
if (!node) return
const observer = new ResizeObserver(([entry]) => {
if (entry) setWidth(Math.max(320, entry.contentRect.width))
})
observer.observe(node)
return () => observer.disconnect()
}, [])
return [ref, width]
}
function timeLabel(ms: number, spanMs: number): string {
// A week of data wants dates; an hour wants clock time. Showing both at all
// ranges is noise, and showing neither leaves a time series with no time.
return new Date(ms).toLocaleString(undefined, {
...(spanMs > 86_400_000 ? { month: 'short', day: 'numeric' } : {}),
hour: '2-digit',
minute: '2-digit',
})
}
export function ShareChart({ points }: { points: MinerSeriesPoint[] }) {
const [wrap, width] = useMeasuredWidth()
const [hover, setHover] = useState<Placed | null>(null)
const chart = useMemo(() => {
if (points.length === 0) return null
const times = points.map((p) => new Date(p.at).getTime())
const t0 = Math.min(...times)
const t1 = Math.max(...times)
const span = t1 - t0 || 1
// The y-axis starts at zero — a share chart whose baseline floats makes a
// steady miner look volatile. Headroom above the peak keeps the line off
// the top edge.
const peak = Math.max(...points.map((p) => p.share), 0.0001)
const top = peak * 1.15
const plotW = width - PAD.left - PAD.right
const plotH = HEIGHT - PAD.top - PAD.bottom
const baseline = PAD.top + plotH
const x = (t: number) => PAD.left + ((t - t0) / span) * plotW
const y = (v: number) => baseline - (v / top) * plotH
const placed: Placed[] = points.map((point, i) => ({
point,
x: x(times[i] ?? t0),
y: y(point.share),
}))
const line = placed
.map((p, i) => `${i === 0 ? 'M' : 'L'}${p.x.toFixed(1)} ${p.y.toFixed(1)}`)
.join(' ')
const first = placed[0]
const last = placed[placed.length - 1]
const area =
first && last
? `${line} L${last.x.toFixed(1)} ${baseline} L${first.x.toFixed(1)} ${baseline} Z`
: ''
const yTicks = [0, top / 2, top].map((v) => ({ v, y: y(v) }))
// Three x labels — ends and middle. More would collide on a narrow panel,
// and the tooltip carries the exact time for any point.
const xTicks = [0, 0.5, 1].map((f) => ({ t: t0 + span * f, x: x(t0 + span * f) }))
return { placed, area, line, yTicks, xTicks, baseline, span, last }
}, [points, width])
if (!chart) {
return <p className="empty">No history recorded for this miner yet.</p>
}
const onMove = (event: React.PointerEvent<SVGSVGElement>) => {
const rect = event.currentTarget.getBoundingClientRect()
const local = event.clientX - rect.left
let nearest: Placed | null = null
let best = Infinity
for (const p of chart.placed) {
const d = Math.abs(p.x - local)
if (d < best) {
best = d
nearest = p
}
}
setHover(nearest)
}
return (
<div ref={wrap} style={{ position: 'relative' }}>
<svg
className="chart"
width={width}
height={HEIGHT}
viewBox={`0 0 ${width} ${HEIGHT}`}
role="img"
aria-label={`Share of network blocks over time, from ${share(chart.placed[0]?.point.share ?? 0)} to ${share(chart.last?.point.share ?? 0)}`}
onPointerMove={onMove}
onPointerLeave={() => setHover(null)}
>
{chart.yTicks.map((t) => (
<g key={t.v}>
<line className="grid" x1={PAD.left} x2={width - PAD.right} y1={t.y} y2={t.y} />
<text className="chart-axis" x={PAD.left - 8} y={t.y + 3.5} textAnchor="end">
{share(t.v)}
</text>
</g>
))}
{chart.xTicks.map((t, i) => (
<text
key={t.x}
className="chart-axis"
x={t.x}
y={HEIGHT - 8}
textAnchor={i === 0 ? 'start' : i === 2 ? 'end' : 'middle'}
>
{timeLabel(t.t, chart.span)}
</text>
))}
<path className="area" d={chart.area} />
<path className="line" d={chart.line} />
{hover && (
<>
<line className="crosshair" x1={hover.x} x2={hover.x} y1={PAD.top} y2={chart.baseline} />
{/* 2px surface ring so the marker stays legible where it sits on the
line; the ring is part of the hit target, not just spacing. */}
<circle className="marker" cx={hover.x} cy={hover.y} r={4.5} />
</>
)}
{/* One direct label, on the endpoint — the current value is the one a
reader is looking for. Labelling every point would be unreadable. */}
{!hover && chart.last && (
<text
className="chart-axis"
x={chart.last.x - 7}
// Clamped inside the plot area. A miner whose latest share is zero
// puts the endpoint on the baseline, and an unclamped label lands on
// top of the time axis.
y={Math.min(Math.max(chart.last.y - 9, PAD.top + 10), chart.baseline - 8)}
textAnchor="end"
style={{ fill: 'var(--data-bright)', fontSize: 11 }}
>
{share(chart.last.point.share)}
</text>
)}
</svg>
{hover && (
<div
className="tooltip"
style={{
left: hover.x,
top: 4,
// Flip past the midpoint so the tooltip never runs off the right
// edge of the panel.
transform: hover.x > width / 2 ? 'translate(-100%, 0)' : 'translate(10px, 0)',
}}
>
<div className="tooltip-label">{timeLabel(new Date(hover.point.at).getTime(), chart.span)}</div>
<div>
<strong>{share(hover.point.share)}</strong> of blocks
</div>
<div style={{ color: 'var(--text-secondary)' }}>
{hover.point.blocks} block{hover.point.blocks === 1 ? '' : 's'} ·{' '}
{hashrate(hover.point.hashrate_estimate)}
</div>
</div>
)}
</div>
)
}

94
web/src/components/StatBar.tsx Normal file
View File

@@ -0,0 +1,94 @@
/**
* The headline numbers.
*
* A KPI row of stat tiles, not a chart: these are five single current values,
* and a bar chart of five unrelated quantities would be a chart of nothing.
* Network hashrate leads as the hero figure because it is the one number that
* answers "how hard is this to win right now".
*/
import type { ChainSummary } from '../api/generated/ChainSummary'
import type { Window as WindowName } from '../api/generated/Window'
import { bigNumber, hashrate, height, seconds, windowSpan } from '../lib/format'
function Stat({
label,
value,
note,
hero,
noteClass,
title,
}: {
label: string
value: string
note?: string
hero?: boolean
noteClass?: string
title?: string
}) {
return (
<div className={hero ? 'stat stat-hero' : 'stat'}>
<div className="stat-label">{label}</div>
<div className="stat-value" title={title}>
{value}
</div>
<div className={noteClass ? `stat-note ${noteClass}` : 'stat-note'}>{note ?? ''}</div>
</div>
)
}
export function StatBar({
summary,
windowName,
}: {
summary: ChainSummary | null
windowName: WindowName
}) {
if (!summary) {
return (
<div className="stats">
{['Network hashrate', 'Difficulty', 'Block time', 'Height', 'Miners'].map((label) => (
<Stat key={label} label={label} value="—" />
))}
</div>
)
}
const interval = summary.block_interval_seconds ?? summary.target_block_time_seconds
const windowBlocks = { hour: 600, six_hours: 3600, day: 14400, week: 100800 }[windowName]
return (
<div className="stats">
<Stat
hero
label="Network hashrate"
value={hashrate(summary.network_hashrate)}
// Presenting an estimate as a measurement is the easiest way to lie
// with this number, so the tile says which it is every time.
note={summary.hashrate_from_target ? 'nominal — from target block time' : 'measured at tip'}
{...(summary.hashrate_from_target ? { noteClass: 'nominal' } : {})}
/>
<Stat
label="Difficulty"
value={bigNumber(summary.difficulty)}
note="expected hashes per block"
{...(summary.difficulty ? { title: summary.difficulty } : {})}
/>
<Stat
label="Block time"
value={seconds(summary.block_interval_seconds)}
note={`target ${seconds(summary.target_block_time_seconds)}`}
/>
<Stat label="Height" value={height(summary.height)} note="best block seen" />
<Stat
label="Miners"
value={String(summary.distinct_miners)}
note={
summary.telemetry_nodes !== null
? `${summary.telemetry_nodes} node${summary.telemetry_nodes === 1 ? '' : 's'} on telemetry`
: `winning in ${windowSpan(windowBlocks, interval)}`
}
/>
</div>
)
}

707
web/src/index.css Normal file
View File

@@ -0,0 +1,707 @@
/*
* blackbeard.observer — the design system.
*
* Dark, and only dark. This is a deliberate commitment rather than an omission:
* the site is a scoreboard for a proof-of-work chain, read in the same rooms as
* the miners it reports on, and a light variant would be a second palette to
* validate for the sake of a context this content does not have. `color-scheme`
* is stated so the browser paints form controls and scrollbars to match instead
* of flashing white.
*
* The two data colours were validated with the dataviz palette checker against
* the chart surface (#14110d): bronze #bd8829 passes the lightness band, chroma
* floor and 3:1 contrast. Bronze and crimson are adjacent hues and do NOT pass
* as a categorical pair — which is why no chart here uses them as two series.
* Every chart is single-series (magnitude, one hue); identity is carried by
* labels and row treatment, never by a second colour a reader has to tell apart.
*/
:root {
color-scheme: dark;
/* Surfaces, warm-black — a cold grey reads as a developer tool, not a arena. */
--surface-0: #0d0b09;
--surface-1: #14110d;
--surface-2: #1c1813;
--surface-3: #262019;
--border: #2f2719;
--border-strong: #453923;
/* Ink. Contrast against --surface-1: 16.0, 6.3, 3.7. */
--text-primary: #f2ece0;
--text-secondary: #a2937c;
--text-muted: #7a6c59;
/* The single data hue: magnitude, share bars, the history area. */
--data: #bd8829;
--data-bright: #e0a63a;
--data-wash: rgba(189, 136, 41, 0.1);
/* Identity and alarm. Never a second series alongside --data. */
--accent: #d8453a;
--accent-bright: #f05a4c;
--accent-wash: rgba(216, 69, 58, 0.1);
--good: #3f9d76;
--warn: #d9a441;
--font-display: 'Iowan Old Style', 'Palatino Linotype', Palatino, 'Book Antiqua', Georgia, serif;
--font-body: ui-sans-serif, system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif;
--font-mono: ui-monospace, 'SF Mono', 'JetBrains Mono', Menlo, Consolas, monospace;
--rule: 1px solid var(--border);
--shadow: 0 1px 0 rgba(255, 255, 255, 0.03) inset;
}
* {
box-sizing: border-box;
}
html,
body,
#root {
height: 100%;
}
body {
margin: 0;
background:
/* A faint warm glow behind the masthead, so the page has a top rather than
being a flat field. Fixed so it does not travel with the scroll. */
radial-gradient(120% 60% at 50% -10%, rgba(189, 136, 41, 0.08), transparent 60%),
var(--surface-0);
background-attachment: fixed;
color: var(--text-primary);
font-family: var(--font-body);
font-size: 15px;
line-height: 1.5;
-webkit-font-smoothing: antialiased;
}
a {
color: inherit;
text-decoration: none;
}
/* Visible focus everywhere. The site is a table of links; keyboard users need
to see where they are, and the default ring is invisible on this ground. */
:focus-visible {
outline: 2px solid var(--data-bright);
outline-offset: 2px;
border-radius: 2px;
}
/* ---- type ---------------------------------------------------------------- */
.display {
font-family: var(--font-display);
font-weight: 700;
letter-spacing: 0.02em;
}
.eyebrow {
font-size: 11px;
font-weight: 600;
letter-spacing: 0.16em;
text-transform: uppercase;
color: var(--text-muted);
}
.numeral {
font-family: var(--font-mono);
font-variant-numeric: tabular-nums;
/* Tabular figures are the whole point: a column of hashrates that shifts
horizontally on every update is unreadable on a page that updates every
few seconds. */
}
/* ---- layout -------------------------------------------------------------- */
.shell {
max-width: 1180px;
margin: 0 auto;
padding: 0 20px 72px;
}
.panel {
background: var(--surface-1);
border: var(--rule);
box-shadow: var(--shadow);
}
.panel-head {
display: flex;
align-items: baseline;
justify-content: space-between;
gap: 12px;
padding: 14px 18px;
border-bottom: var(--rule);
flex-wrap: wrap;
}
.panel-title {
font-family: var(--font-display);
font-size: 17px;
font-weight: 700;
letter-spacing: 0.04em;
text-transform: uppercase;
margin: 0;
}
/* ---- masthead ------------------------------------------------------------ */
.masthead {
display: flex;
align-items: center;
justify-content: space-between;
gap: 20px;
padding: 22px 0 18px;
flex-wrap: wrap;
}
.mark {
display: flex;
align-items: center;
gap: 12px;
}
.mark-name {
font-family: var(--font-display);
font-size: 25px;
font-weight: 700;
letter-spacing: 0.04em;
line-height: 1;
}
.mark-name span {
color: var(--data);
}
.mark-tagline {
font-size: 11px;
letter-spacing: 0.2em;
text-transform: uppercase;
color: var(--text-muted);
margin-top: 5px;
}
.masthead-right {
display: flex;
align-items: center;
gap: 16px;
flex-wrap: wrap;
}
/* ---- connection light ---------------------------------------------------- */
.status {
display: inline-flex;
align-items: center;
gap: 7px;
font-size: 11px;
letter-spacing: 0.14em;
text-transform: uppercase;
color: var(--text-secondary);
}
.status-dot {
width: 7px;
height: 7px;
border-radius: 50%;
background: var(--text-muted);
flex: none;
}
.status-live .status-dot {
background: var(--good);
box-shadow: 0 0 0 3px rgba(63, 157, 118, 0.18);
}
.status-reconnecting .status-dot,
.status-connecting .status-dot {
background: var(--warn);
animation: pulse 1.1s ease-in-out infinite;
}
.status-offline .status-dot {
background: var(--accent);
}
@keyframes pulse {
50% {
opacity: 0.25;
}
}
/* ---- chain switcher & window selector ------------------------------------ */
.segmented {
display: inline-flex;
border: var(--rule);
background: var(--surface-1);
}
.segmented button {
appearance: none;
border: 0;
background: transparent;
color: var(--text-secondary);
font: inherit;
font-size: 12px;
letter-spacing: 0.1em;
text-transform: uppercase;
padding: 7px 13px;
cursor: pointer;
white-space: nowrap;
}
.segmented button + button {
border-left: var(--rule);
}
.segmented button:hover:not(:disabled) {
color: var(--text-primary);
background: var(--surface-2);
}
.segmented button[aria-pressed='true'] {
background: var(--data-wash);
color: var(--data-bright);
}
.segmented button:disabled {
color: var(--text-muted);
cursor: not-allowed;
}
/* ---- stat tiles ---------------------------------------------------------- */
.stats {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(178px, 1fr));
gap: 1px;
background: var(--border);
border: var(--rule);
margin-bottom: 26px;
}
.stat {
background: var(--surface-1);
padding: 15px 18px 16px;
min-width: 0;
}
/* Two columns and five tiles leaves a blank cell. Letting the hero span the row
makes six, which fills the grid — and gives the headline number the width it
deserves on the screen where it is most cramped. */
@media (max-width: 620px) {
.stat-hero {
grid-column: 1 / -1;
}
}
.stat-label {
display: flex;
align-items: center;
gap: 6px;
font-size: 10px;
font-weight: 600;
letter-spacing: 0.15em;
text-transform: uppercase;
color: var(--text-muted);
}
.stat-value {
font-family: var(--font-mono);
font-variant-numeric: tabular-nums;
font-size: 25px;
font-weight: 600;
line-height: 1.15;
margin-top: 7px;
overflow-wrap: anywhere;
}
.stat-hero .stat-value {
font-size: 33px;
color: var(--data-bright);
}
.stat-note {
font-size: 11px;
color: var(--text-muted);
margin-top: 5px;
min-height: 1.2em;
}
.stat-note.nominal {
color: var(--warn);
}
/* ---- leaderboard --------------------------------------------------------- */
.board {
width: 100%;
border-collapse: collapse;
}
.board th {
font-size: 10px;
font-weight: 600;
letter-spacing: 0.14em;
text-transform: uppercase;
color: var(--text-muted);
text-align: right;
padding: 10px 14px;
border-bottom: var(--rule);
white-space: nowrap;
}
.board th:first-child,
.board th.left {
text-align: left;
}
.board td {
padding: 9px 14px;
border-bottom: 1px solid rgba(47, 39, 25, 0.55);
text-align: right;
white-space: nowrap;
}
.board td:first-child,
.board td.left {
text-align: left;
}
.board tbody tr:hover {
background: var(--surface-2);
}
/* The pinned row: a left rule and a wash, plus an explicit "YOU" chip. Three
redundant cues, none of them colour alone. */
.board tbody tr.mine {
background: var(--accent-wash);
box-shadow: inset 3px 0 0 var(--accent);
}
.board tbody tr.mine:hover {
background: rgba(216, 69, 58, 0.16);
}
.rank {
font-family: var(--font-mono);
font-variant-numeric: tabular-nums;
font-size: 14px;
color: var(--text-muted);
width: 1%;
}
.rank-1,
.rank-2,
.rank-3 {
color: var(--data-bright);
font-weight: 700;
}
.miner-cell {
display: flex;
align-items: center;
gap: 9px;
min-width: 0;
}
.miner-name {
font-weight: 600;
overflow: hidden;
text-overflow: ellipsis;
}
.miner-name.anonymous {
font-family: var(--font-mono);
font-weight: 400;
font-size: 13px;
color: var(--text-secondary);
}
.chip {
font-size: 9px;
font-weight: 700;
letter-spacing: 0.12em;
text-transform: uppercase;
padding: 2px 6px;
border: 1px solid currentColor;
flex: none;
line-height: 1.4;
}
.chip-you {
color: var(--accent-bright);
}
.chip-named {
color: var(--text-muted);
}
/* Share: the percentage as the direct label, the bar beneath it as magnitude.
One hue, 4px rounded data-end at the far side, square against the baseline it
grows from. */
.share-cell {
/* Wide enough that the bar has usable resolution between a 20% leader and
a 0.6% tail; narrower and every small miner's bar is the same sliver. */
min-width: 92px;
}
.share-bar {
position: relative;
height: 5px;
margin-top: 4px;
background: var(--surface-3);
}
.share-bar > i {
display: block;
height: 100%;
background: var(--data);
border-radius: 0 3px 3px 0;
}
tr.mine .share-bar > i {
background: var(--accent);
}
/* Below this width the table would need a horizontal scrollbar, and a column
behind a scrollbar is a column nobody reads. "Last block" is the one a reader
can most easily do without — the live ticker beside the table already says
who mined most recently. */
@media (max-width: 1320px) {
.board .last-seen {
display: none;
}
}
/* On a phone the table still scrolls inside its own box — which is correct for
a data table — but dropping the least load-bearing column and tightening the
cells means far less of it is behind that scroll. Blocks, share and estimated
hashrate, the three a miner actually came for, stay visible without scrolling. */
@media (max-width: 720px) {
.board .streak {
display: none;
}
.board th,
.board td {
padding: 9px 9px;
}
.miner-name {
max-width: 150px;
}
}
.pin-button {
appearance: none;
background: transparent;
border: 0;
color: var(--text-muted);
cursor: pointer;
font: inherit;
font-size: 15px;
line-height: 1;
padding: 3px 5px;
}
.pin-button:hover {
color: var(--data-bright);
}
.pin-button[aria-pressed='true'] {
color: var(--accent-bright);
}
/* ---- block ticker -------------------------------------------------------- */
.ticker-row {
display: grid;
grid-template-columns: auto 1fr auto;
gap: 12px;
align-items: baseline;
padding: 8px 18px;
border-bottom: 1px solid rgba(47, 39, 25, 0.55);
font-size: 13px;
}
.ticker-row.mine {
background: var(--accent-wash);
box-shadow: inset 3px 0 0 var(--accent);
}
/* Newly arrived blocks flash once. Suppressed for reduced-motion, below. */
.ticker-row.fresh {
animation: arrive 1.4s ease-out;
}
@keyframes arrive {
from {
background: var(--data-wash);
}
}
.ticker-height {
font-family: var(--font-mono);
font-variant-numeric: tabular-nums;
color: var(--text-muted);
font-size: 12px;
}
.ticker-gap {
font-family: var(--font-mono);
font-variant-numeric: tabular-nums;
color: var(--text-muted);
font-size: 12px;
}
/* ---- chart --------------------------------------------------------------- */
.chart {
width: 100%;
display: block;
touch-action: none;
}
.chart .grid {
stroke: var(--border);
stroke-width: 1;
}
.chart .area {
fill: var(--data-wash);
}
.chart .line {
fill: none;
stroke: var(--data);
stroke-width: 2;
stroke-linejoin: round;
stroke-linecap: round;
}
.chart .crosshair {
stroke: var(--border-strong);
stroke-width: 1;
}
.chart .marker {
fill: var(--data);
stroke: var(--surface-1);
stroke-width: 2;
}
.chart-axis {
font-family: var(--font-mono);
font-size: 10px;
fill: var(--text-muted);
}
.tooltip {
position: absolute;
pointer-events: none;
background: var(--surface-3);
border: 1px solid var(--border-strong);
padding: 7px 10px;
font-size: 12px;
line-height: 1.45;
white-space: nowrap;
z-index: 5;
}
.tooltip-label {
color: var(--text-muted);
font-size: 10px;
letter-spacing: 0.1em;
text-transform: uppercase;
}
/* ---- misc ---------------------------------------------------------------- */
.empty {
padding: 40px 18px;
text-align: center;
color: var(--text-muted);
font-size: 13px;
}
.banner {
display: flex;
align-items: center;
gap: 10px;
padding: 11px 16px;
border: 1px solid var(--border-strong);
background: var(--surface-2);
font-size: 13px;
margin-bottom: 20px;
}
.banner-warn {
border-color: rgba(217, 164, 65, 0.4);
color: var(--warn);
}
.footer {
margin-top: 44px;
padding-top: 20px;
border-top: var(--rule);
font-size: 12px;
color: var(--text-muted);
display: flex;
gap: 18px;
flex-wrap: wrap;
justify-content: space-between;
}
.footer a {
color: var(--text-secondary);
text-decoration: underline;
text-underline-offset: 3px;
}
/* Wide content scrolls inside its own box; the page never scrolls sideways. */
.scroll-x {
overflow-x: auto;
}
.two-col {
display: grid;
grid-template-columns: minmax(0, 1.75fr) minmax(0, 1fr);
gap: 22px;
align-items: start;
}
@media (max-width: 880px) {
.two-col {
grid-template-columns: minmax(0, 1fr);
}
}
/* Motion is decoration here — the numbers carry the information, so it can all
go without losing anything. */
@media (prefers-reduced-motion: reduce) {
*,
*::before,
*::after {
animation-duration: 0.001ms !important;
animation-iteration-count: 1 !important;
transition-duration: 0.001ms !important;
}
}
.visually-hidden {
position: absolute;
width: 1px;
height: 1px;
padding: 0;
margin: -1px;
overflow: hidden;
clip: rect(0 0 0 0);
white-space: nowrap;
border: 0;
}

23
web/src/lib/ObserverProvider.tsx Normal file
View File

@@ -0,0 +1,23 @@
import { useEffect } from 'react'
import type { ReactNode } from 'react'
import type { Observer } from '../api/socket'
import { ObserverContext } from './observer-context'
/** Provides the single socket for the tab. */
export function ObserverProvider({
observer,
children,
}: {
observer: Observer
children: ReactNode
}) {
useEffect(() => {
observer.connect()
// Deliberately not disconnecting on unmount: in React 19's StrictMode the
// effect runs twice in development, and tearing the socket down between
// them produces a reconnect storm that looks like a bug in the backend. The
// socket's lifetime is the tab's, and the tab closing closes it.
}, [observer])
return <ObserverContext.Provider value={observer}>{children}</ObserverContext.Provider>
}

94
web/src/lib/format.ts Normal file
View File

@@ -0,0 +1,94 @@
/**
* Formatting.
*
* All of it in one place because most of these numbers are wrong-looking if
* formatted naively — a hashrate in bare hashes per second is fourteen digits,
* a difficulty is thirty, and a share of 0.0004 rounds to "0%".
*/
/** SI-ish hashrate. Miners talk in MH/s and GH/s; nobody says "6.9e9 H/s". */
export function hashrate(hs: number | null | undefined): string {
if (hs === null || hs === undefined || !Number.isFinite(hs)) return '—'
const units = ['H/s', 'kH/s', 'MH/s', 'GH/s', 'TH/s', 'PH/s', 'EH/s']
let value = hs
let unit = 0
while (value >= 1000 && unit < units.length - 1) {
value /= 1000
unit += 1
}
const digits = value >= 100 ? 0 : value >= 10 ? 1 : 2
return `${value.toFixed(digits)} ${units[unit]}`
}
/**
* A very large integer, given as a decimal string, in compact form.
*
* Difficulty arrives as a string precisely so it is not rounded in transit
* (see `BigUintDec` in the Rust entities crate); it is rounded here, once, for
* display, and the full value goes in the element's `title`.
*/
export function bigNumber(decimal: string | null | undefined): string {
if (!decimal) return '—'
const digits = decimal.replace(/^0+(?=\d)/, '')
const suffixes = ['', 'K', 'M', 'B', 'T', 'Q']
const magnitude = Math.floor((digits.length - 1) / 3)
if (magnitude === 0) return digits
if (magnitude >= suffixes.length) {
// Past quadrillions, an exponent is more honest than inventing a suffix.
return `${digits[0]}.${digits.slice(1, 3)}e${digits.length - 1}`
}
const whole = digits.slice(0, digits.length - magnitude * 3)
const fraction = digits.slice(whole.length, whole.length + 2)
return `${whole}.${fraction}${suffixes[magnitude]}`
}
/** A 0–1 share as a percentage. Small shares keep a decimal so they aren't 0%. */
export function share(fraction: number): string {
const pct = fraction * 100
if (pct === 0) return '0%'
if (pct < 0.1) return '<0.1%'
return `${pct.toFixed(pct < 10 ? 1 : 0)}%`
}
/** Seconds, phrased the way a block interval reads. */
export function seconds(value: number | null | undefined): string {
if (value === null || value === undefined || !Number.isFinite(value)) return '—'
if (value < 10) return `${value.toFixed(1)}s`
if (value < 120) return `${Math.round(value)}s`
return `${(value / 60).toFixed(1)}m`
}
/** How long ago, in the shortest form that is still unambiguous. */
export function ago(iso: string, now: number = Date.now()): string {
const delta = Math.max(0, (now - new Date(iso).getTime()) / 1000)
if (delta < 5) return 'just now'
if (delta < 60) return `${Math.floor(delta)}s ago`
if (delta < 3600) return `${Math.floor(delta / 60)}m ago`
if (delta < 86400) return `${Math.floor(delta / 3600)}h ago`
return `${Math.floor(delta / 86400)}d ago`
}
/** Block heights with thousands separators. */
export function height(value: number | null | undefined): string {
return value === null || value === undefined ? '—' : value.toLocaleString('en-US')
}
/** The abbreviated form of a reward preimage, matching the backend's. */
export function shortMiner(miner: string): string {
return miner.length > 14 ? `${miner.slice(0, 10)}…${miner.slice(-4)}` : miner
}
/**
* Approximately how long a window of `blocks` covers, at a measured or target
* interval.
*
* The window is a block count, not a duration — deliberately, because a
* duration is meaningless while a node is catching up. This is the human gloss
* on it, and it says "about" for a reason.
*/
export function windowSpan(blocks: number, intervalSeconds: number): string {
const total = blocks * intervalSeconds
if (total < 5400) return `~${Math.round(total / 60)} min`
if (total < 172800) return `~${Math.round(total / 3600)} h`
return `~${Math.round(total / 86400)} d`
}

20
web/src/lib/observer-context.ts Normal file
View File

@@ -0,0 +1,20 @@
/**
* The context the provider fills and the hooks read.
*
* Its own module so `ObserverProvider.tsx` can export only a component and
* `store.ts` only hooks — React Fast Refresh gives up on a file that exports
* both, which in development means every socket change forces a full reload.
*/
import { createContext, useContext } from 'react'
import type { Observer } from '../api/socket'
export const ObserverContext = createContext<Observer | null>(null)
/** The tab's socket. Throws outside a provider rather than failing silently. */
export function useObserverInstance(): Observer {
const observer = useContext(ObserverContext)
if (!observer) throw new Error('useObserver must be used inside an ObserverProvider')
return observer
}

108
web/src/lib/store.ts Normal file
View File

@@ -0,0 +1,108 @@
/**
* React's view of the socket, plus the one piece of purely local state: which
* miner is *yours*.
*/
import { useCallback, useEffect, useMemo, useSyncExternalStore } from 'react'
import type { ObserverState } from '../api/socket'
import { useObserverInstance } from './observer-context'
/** The whole live state. */
export function useObserver(): ObserverState {
const observer = useObserverInstance()
return useSyncExternalStore(observer.subscribe, observer.getSnapshot, observer.getSnapshot)
}
/** Subscribe to a chain and window; re-subscribes when either changes. */
export function useWatch(chain: string | null, windowName: ObserverState['window']): void {
const observer = useObserverInstance()
useEffect(() => {
if (chain) observer.watch(chain, windowName)
}, [observer, chain, windowName])
}
const PINNED_KEY = 'blackbeard.pinned-miners'
/**
* The miners this browser calls its own.
*
* There are no accounts here and there is nothing to log into: a reward
* preimage is public, so "this one is mine" is a claim the site has no way to
* verify and no reason to. It lives in `localStorage`, which means it is
* per-browser, survives a refresh, and never reaches the server. That is the
* whole feature — and it is why nothing on the site is gated behind it.
*/
export function usePinnedMiners(): {
pinned: string[]
isPinned: (miner: string) => boolean
toggle: (miner: string) => void
} {
const pinned = useSyncExternalStore(subscribePinned, readPinned, () => EMPTY)
const isPinned = useCallback((miner: string) => pinned.includes(miner.toLowerCase()), [pinned])
const toggle = useCallback((miner: string) => {
const id = miner.toLowerCase()
const current = readPinned()
const next = current.includes(id) ? current.filter((m) => m !== id) : [...current, id]
writePinned(next)
}, [])
return useMemo(() => ({ pinned, isPinned, toggle }), [pinned, isPinned, toggle])
}
const EMPTY: string[] = []
let cache: string[] = EMPTY
let cacheRaw: string | null = null
const pinnedListeners = new Set<() => void>()
function subscribePinned(listener: () => void): () => void {
pinnedListeners.add(listener)
// `storage` fires in *other* tabs, so pinning a miner in one tab updates the
// highlight in the others without either of them reloading.
const onStorage = (event: StorageEvent) => {
if (event.key === PINNED_KEY) listener()
}
window.addEventListener('storage', onStorage)
return () => {
pinnedListeners.delete(listener)
window.removeEventListener('storage', onStorage)
}
}
/**
* Read the pinned list, memoised on the raw string.
*
* `useSyncExternalStore` demands a stable reference between changes; parsing
* fresh JSON on every call returns a new array each time and React throws.
*/
function readPinned(): string[] {
let raw: string | null = null
try {
raw = window.localStorage.getItem(PINNED_KEY)
} catch {
// Private windows and "block site data" both throw rather than return null.
return EMPTY
}
if (raw === cacheRaw) return cache
cacheRaw = raw
try {
const parsed: unknown = raw ? JSON.parse(raw) : []
cache = Array.isArray(parsed) ? parsed.filter((m): m is string => typeof m === 'string') : EMPTY
} catch {
cache = EMPTY
}
return cache
}
function writePinned(next: string[]): void {
try {
window.localStorage.setItem(PINNED_KEY, JSON.stringify(next))
} catch {
// Storage unavailable: the pin lasts for this render and no longer. Better
// than refusing the click.
}
cacheRaw = null
for (const listener of pinnedListeners) listener()
}

22
web/src/main.tsx Normal file
View File

@@ -0,0 +1,22 @@
import { StrictMode } from 'react'
import { createRoot } from 'react-dom/client'
import App from './App'
import { Observer } from './api/socket'
import { ObserverProvider } from './lib/ObserverProvider'
import './index.css'
// One socket per tab, created here rather than inside a component so React 19's
// double-invoked StrictMode effects cannot open two.
const observer = new Observer()
const root = document.getElementById('root')
if (!root) throw new Error('#root is missing from index.html')
createRoot(root).render(
<StrictMode>
<ObserverProvider observer={observer}>
<App />
</ObserverProvider>
</StrictMode>,
)

16
web/src/vite-env.d.ts vendored Normal file
View File

@@ -0,0 +1,16 @@
/// <reference types="vite/client" />
interface ImportMetaEnv {
/**
* Override the REST base. Defaults to `/v1` on the page's own origin, which
* is what both nginx and the dev proxy serve — so this is only needed when
* the frontend is hosted apart from the API.
*/
readonly VITE_API_BASE_URL?: string
/** Override the WebSocket URL. Defaults to `/v1/ws` on the page's origin. */
readonly VITE_WS_URL?: string
}
interface ImportMeta {
readonly env: ImportMetaEnv
}

23
web/tsconfig.app.json Normal file
View File

@@ -0,0 +1,23 @@
{
"compilerOptions": {
"target": "ES2022",
"lib": ["ES2022", "DOM", "DOM.Iterable"],
"module": "ESNext",
"moduleResolution": "bundler",
"jsx": "react-jsx",
"types": ["vite/client"],
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noUncheckedIndexedAccess": true,
"noFallthroughCasesInSwitch": true,
"exactOptionalPropertyTypes": true,
"verbatimModuleSyntax": true,
"isolatedModules": true,
"skipLibCheck": true,
"noEmit": true,
"tsBuildInfoFile": "./node_modules/.tmp/tsconfig.app.tsbuildinfo"
},
"include": ["src"]
}

4
web/tsconfig.json Normal file
View File

@@ -0,0 +1,4 @@
{
"files": [],
"references": [{ "path": "./tsconfig.app.json" }, { "path": "./tsconfig.node.json" }]
}

16
web/tsconfig.node.json Normal file
View File

@@ -0,0 +1,16 @@
{
"compilerOptions": {
"target": "ES2022",
"lib": ["ES2023"],
"module": "ESNext",
"moduleResolution": "bundler",
"types": ["node"],
"strict": true,
"skipLibCheck": true,
"noEmit": true,
"isolatedModules": true,
"verbatimModuleSyntax": true,
"tsBuildInfoFile": "./node_modules/.tmp/tsconfig.node.tsbuildinfo"
},
"include": ["vite.config.ts"]
}

26
web/vite.config.ts Normal file
View File

@@ -0,0 +1,26 @@
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react-swc'
// The built output is static and served by nginx from /var/www/blackbeard.observer;
// there is no Node.js in production (architecture/generic.md §4).
export default defineConfig({
plugins: [react()],
build: {
// Source maps ship: the bundle is public, the code is GPL, and a stack
// trace from a miner's browser is worth far more than the bytes.
sourcemap: true,
},
server: {
port: 5173,
// `pnpm dev` talks to a locally-run blackbeard-api so the browser sees one
// origin and the WebSocket upgrade needs no CORS dance — the same shape
// nginx presents in production.
proxy: {
'/v1': {
target: 'http://127.0.0.1:25864',
changeOrigin: true,
ws: true,
},
},
},
})