feat(deploy): stake BUH_NODE_PORT on 31415 + testnet peering doc
Some checks failed
deploy / Build buh-api + buh-cli (static musl) (push) Waiting to run
deploy / Deploy buh node to slartibartfast (push) Has been cancelled

Settle buh's standardised ingress port on 31415/tcp — unassigned in IANA,
clear of the crowded 8443/9443/6443 appliance space, and below the Linux
ephemeral range so it's stable to forward from the edge. Updated the code
default (config.rs node_bind), the CI workflow env, firewalld service,
operator-model deploy.sh/manifest.yml, and the asset readme.

Also drop the SELinux `semanage port` step (and its two sudoers rules):
buh-api runs as unconfined_service_t and may bind any port, so BUH_NODE_PORT
needs no http_port_t label. The firewalld rules are keyed on the service
NAME (buh-node), not the port — so a port change is now purely repo-side and
needs no infra-setup re-run on the host.

Add doc/testnet.md: the per-node-CA trust model, the 31415 convention, a
node roster (incl. the slarti dogfood node + its CA fingerprint), and the
mutual trust / peer ping workflow for joining the testnet.

Gate: cargo fmt + clippy -D warnings clean; config renders node_bind
0.0.0.0:31415; sudoers passes visudo -c; deploy.yml valid YAML.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EB3LjarCdXxqrJ4tFLn8LB
This commit is contained in:
2026-06-30 18:05:45 +03:00
parent 0a6d05c5da
commit 9d5919b119
8 changed files with 147 additions and 18 deletions

View File

@@ -41,7 +41,7 @@ concurrency:
env: env:
# --- infra truth --- # --- infra truth ---
NODE_HOST: slartibartfast.kosherinata.internal NODE_HOST: slartibartfast.kosherinata.internal
NODE_PORT: "8443" # BUH_NODE_PORT — the only edge port (PQ-mTLS ingress) NODE_PORT: "31415" # BUH_NODE_PORT — buh's staked port; the only edge port (PQ-mTLS ingress)
ADMIN_PORT: "8081" # loopback operator admin API (health probe target) ADMIN_PORT: "8081" # loopback operator admin API (health probe target)
NODE_SANS: '["slartibartfast.kosherinata.internal"]' # TOML array for the leaf SANs NODE_SANS: '["slartibartfast.kosherinata.internal"]' # TOML array for the leaf SANs
MUSL_TARGET: x86_64-unknown-linux-musl MUSL_TARGET: x86_64-unknown-linux-musl
@@ -169,22 +169,22 @@ jobs:
asset/firewalld/buh-node.xml \ asset/firewalld/buh-node.xml \
gitea_ci@"${NODE_HOST}":/etc/firewalld/services/buh-node.xml gitea_ci@"${NODE_HOST}":/etc/firewalld/services/buh-node.xml
- name: Apply SELinux + firewalld (idempotent) - name: Apply SELinux contexts + firewalld (idempotent)
run: | run: |
ssh gitea_ci@"${NODE_HOST}" ' ssh gitea_ci@"${NODE_HOST}" '
set -euo pipefail set -euo pipefail
sudo /usr/sbin/restorecon -R /usr/local/bin/buh-api /usr/local/bin/buh-cli /etc/buh /var/lib/buh sudo /usr/sbin/restorecon -R /usr/local/bin/buh-api /usr/local/bin/buh-cli /etc/buh /var/lib/buh
# Label the PQ-mTLS ingress port (guarded so re-runs are no-ops; 8443 # No SELinux port labeling: buh-api runs as unconfined_service_t (a plain
# is usually already http_port_t). # systemd unit with no policy module), which may bind any port — so
if ! sudo /usr/sbin/semanage port -l | grep -E "^http_port_t" | grep -qw '"${NODE_PORT}"'; then # BUH_NODE_PORT needs no http_port_t entry. Keeping it out means a port
sudo /usr/sbin/semanage port -a -t http_port_t -p tcp '"${NODE_PORT}"' # change is purely repo-side: the firewalld rules below are keyed on the
fi # service NAME (buh-node), and the freshly-shipped buh-node.xml carries the
# firewalld only learns a freshly-shipped custom service after reload. # new port, so a reload picks it up with no sudoers/infra-setup change.
sudo /usr/bin/firewall-cmd --reload sudo /usr/bin/firewall-cmd --reload
if ! sudo /usr/bin/firewall-cmd --query-service=buh-node; then if ! sudo /usr/bin/firewall-cmd --query-service=buh-node; then
sudo /usr/bin/firewall-cmd --add-service=buh-node --permanent sudo /usr/bin/firewall-cmd --add-service=buh-node --permanent
sudo /usr/bin/firewall-cmd --reload fi
fi' sudo /usr/bin/firewall-cmd --reload'
- name: Restart buh-node - name: Restart buh-node
run: | run: |

View File

@@ -50,7 +50,7 @@ here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
: "${PKI_ENABLED:=true}" : "${PKI_ENABLED:=true}"
: "${PKI_DIR:=${STATE_DIR}/pki}" : "${PKI_DIR:=${STATE_DIR}/pki}"
: "${NODE_BIND:=0.0.0.0:8443}" : "${NODE_BIND:=0.0.0.0:31415}"
: "${PKI_SANS:=[\"localhost\"]}" : "${PKI_SANS:=[\"localhost\"]}"
: "${LEAF_TTL_HOURS:=48}" : "${LEAF_TTL_HOURS:=48}"
: "${ROTATE_EVERY_HOURS:=24}" : "${ROTATE_EVERY_HOURS:=24}"

View File

@@ -5,5 +5,5 @@
the relay/blob API served over X25519MLKEM768 mutual TLS, with peers pinned per-CA. The plain the relay/blob API served over X25519MLKEM768 mutual TLS, with peers pinned per-CA. The plain
loopback health port (127.0.0.1:8080) is never opened. Forward this port from the edge loopback health port (127.0.0.1:8080) is never opened. Forward this port from the edge
(OPNsense/router) to the node host.</description> (OPNsense/router) to the node host.</description>
<port protocol="tcp" port="8443"/> <port protocol="tcp" port="31415"/>
</service> </service>

View File

@@ -33,7 +33,7 @@ environments:
pki: pki:
enabled: true enabled: true
dir: /var/lib/buh/pki # CA key + cert; the node generates these on first start dir: /var/lib/buh/pki # CA key + cert; the node generates these on first start
node_bind: 0.0.0.0:8443 # BUH_NODE_PORT — the only port exposed at the edge node_bind: 0.0.0.0:31415 # BUH_NODE_PORT — the only port exposed at the edge
sans: [node1.example.com] # hostnames/IPs the leaf answers to sans: [node1.example.com] # hostnames/IPs the leaf answers to
leaf_ttl_hours: 48 leaf_ttl_hours: 48
rotate_every_hours: 24 # in-process leaf rotation (no step-ca, no .path units) rotate_every_hours: 24 # in-process leaf rotation (no step-ca, no .path units)

View File

@@ -23,7 +23,7 @@ plane, no shared database, and no central PKI**:
process* (`[relay].sweep_interval_seconds`), because Turso locks the datastore exclusively — a process* (`[relay].sweep_interval_seconds`), because Turso locks the datastore exclusively — a
separate `buh-cli sweep` cannot open the DB while the daemon holds it. separate `buh-cli sweep` cannot open the DB while the daemon holds it.
- `systemd/buh.sysusers.conf` — the unprivileged `buh` service account. - `systemd/buh.sysusers.conf` — the unprivileged `buh` service account.
- `firewalld/buh-node.xml` — opens **`BUH_NODE_PORT` (8443)**, the single PQ-mTLS ingress port. - `firewalld/buh-node.xml` — opens **`BUH_NODE_PORT` (31415)**, the single PQ-mTLS ingress port.
The plain loopback health port is never exposed. The plain loopback health port is never exposed.
- `deploy.sh` — installs the above on the local host, renders the config, has the node generate - `deploy.sh` — installs the above on the local host, renders the config, has the node generate
its CA, and prints the CA fingerprint to share with peers. its CA, and prints the CA fingerprint to share with peers.

View File

@@ -20,10 +20,12 @@ gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemd-sysusers
gitea_ci ALL=(root) NOPASSWD: /usr/bin/install -d -o root -g buh -m 0750 /etc/buh gitea_ci ALL=(root) NOPASSWD: /usr/bin/install -d -o root -g buh -m 0750 /etc/buh
gitea_ci ALL=(root) NOPASSWD: /usr/bin/install -d -o buh -g buh -m 0750 /var/lib/buh gitea_ci ALL=(root) NOPASSWD: /usr/bin/install -d -o buh -g buh -m 0750 /var/lib/buh
# --- SELinux + firewalld --- # --- SELinux contexts + firewalld ---
# No `semanage port` rules: buh-api runs as unconfined_service_t and may bind any
# port, so BUH_NODE_PORT needs no http_port_t label. The firewalld rules below are
# keyed on the service NAME (buh-node), not the port number, so changing
# BUH_NODE_PORT is purely repo-side and needs no edit here / no infra-setup re-run.
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/restorecon -R /usr/local/bin/buh-api /usr/local/bin/buh-cli /etc/buh /var/lib/buh gitea_ci ALL=(root) NOPASSWD: /usr/sbin/restorecon -R /usr/local/bin/buh-api /usr/local/bin/buh-cli /etc/buh /var/lib/buh
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/semanage port -l
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/semanage port -a -t http_port_t -p tcp 8443
gitea_ci ALL=(root) NOPASSWD: /usr/bin/firewall-cmd --reload gitea_ci ALL=(root) NOPASSWD: /usr/bin/firewall-cmd --reload
gitea_ci ALL=(root) NOPASSWD: /usr/bin/firewall-cmd --query-service\=buh-node gitea_ci ALL=(root) NOPASSWD: /usr/bin/firewall-cmd --query-service\=buh-node
gitea_ci ALL=(root) NOPASSWD: /usr/bin/firewall-cmd --add-service\=buh-node --permanent gitea_ci ALL=(root) NOPASSWD: /usr/bin/firewall-cmd --add-service\=buh-node --permanent

View File

@@ -131,7 +131,7 @@ impl Default for AppConfig {
pki: PkiConfig { pki: PkiConfig {
enabled: false, enabled: false,
dir: "/var/lib/buh/pki".to_string(), dir: "/var/lib/buh/pki".to_string(),
node_bind: "0.0.0.0:8443".to_string(), node_bind: "0.0.0.0:31415".to_string(),
sans: vec!["localhost".to_string()], sans: vec!["localhost".to_string()],
leaf_ttl_hours: 48, leaf_ttl_hours: 48,
rotate_every_hours: 24, rotate_every_hours: 24,

127
doc/testnet.md Normal file
View File

@@ -0,0 +1,127 @@
# buh testnet — peering guide
How to stand up a buh node, join it to the testnet, and establish a peering with
another node. buh is an **anti-hub**: there is no central server, no shared root
CA, and no registry. A "testnet" is just a set of nodes that have chosen to trust
each other's certificate authorities.
## Trust model in one paragraph
Every node is **its own CA**. On first start it generates a CA (`ca.cert.der` +
`ca.key.pem` under its pki dir) and rotates a short-lived TLS leaf from it in
process. Its identity is the **CA fingerprint** — the lowercase hex SHA-256 of the
CA certificate DER (`:` separators are accepted on input but not required). Two
nodes talk over **mutual** PQ-mTLS (X25519MLKEM768): a handshake succeeds only
when **each side has pinned the other's CA fingerprint**. Trust is therefore
explicit, pairwise, and symmetric — there is nothing to "join", only peers to
exchange fingerprints with. Removing trust (`peer distrust`) takes effect on the
peer's **next handshake**, with no restart.
## The staked port: 31415
buh standardises on **`BUH_NODE_PORT = 31415/tcp`** — the single PQ-mTLS ingress
port a node exposes (chosen to avoid the crowded `8443`/`9443`/`6443` space; it is
unassigned in IANA and below the Linux ephemeral range, so it is safe to forward).
Everything else stays local:
| Port | Bind | Purpose | Exposed? |
|------|------|---------|----------|
| `31415` | `0.0.0.0:31415` | PQ-mTLS relay/blob API (peers + clients) | **yes — forward from the edge** |
| `8081` | `127.0.0.1:8081` | operator admin API (peer-trust mgmt) | no — loopback only, no auth |
| `8080` | `127.0.0.1:8080` | plain health/debug (only when pki is off) | no |
Forward `31415/tcp` from your edge (OPNsense/router) to the node host. Peers reach
you at `your-edge-hostname:31415`.
## Testnet roster
Record each node's reachable address and CA fingerprint here as it joins.
| Node | Edge address (`host:31415`) | CA fingerprint | Role |
|------|------------------------------|----------------|------|
| `slartibartfast` (dogfood) | _\<edge-fqdn\>_:31415 — internal mesh `slartibartfast.kosherinata.internal:31415` | `3c8f125861f3c39f849a469cb32ef599000c71896ea1ccc8a5baaad7419ef808` | relay + blob (fs) |
> Fill in slarti's `<edge-fqdn>` once edge port-forwarding to `31415` is in place.
> Re-keying a node (`buh-cli ca rotate --force`) changes its fingerprint — update
> this table and every peer must re-pin.
## Join the testnet
### 1. Stand up your node
Either model from `asset/readme.md`:
- **CI** — push to a buh checkout whose `.gitea/workflows/deploy.yml` targets your
host (the dogfood node uses this).
- **Manual** — `sudo PKI_SANS='["your-node.example"]' ./asset/deploy.sh` on the host.
Confirm it is healthy (on the host, via the loopback admin API):
```sh
curl -fsS http://127.0.0.1:8081/admin/info
# {"ca_fingerprint":"…","trusted_peers":N}
```
### 2. Learn your CA fingerprint
```sh
buh-cli ca show # prints this node's CA fingerprint
```
This is the value you hand to peers. Exchange fingerprints **out of band** (signal,
in person, an existing secure channel) — buh deliberately has no fingerprint
directory to lie to you.
### 3. Trust your peer, and have them trust you
Peering is symmetric — **both** sides run a `trust`:
```sh
# on YOUR node — pin the peer's CA
buh-cli peer trust 3c8f125861f3c39f849a469cb32ef599000c71896ea1ccc8a5baaad7419ef808 \
--note "slartibartfast (dogfood)"
# on the PEER node — they pin YOUR CA
buh-cli peer trust <your-ca-fingerprint> --note "my-node"
```
`peer` commands talk to the running node's loopback admin API, so trust changes are
live — they apply on the next handshake without a restart. (With the daemon
stopped, the CLI falls back to opening the datastore directly.) Review with:
```sh
buh-cli peer list
```
### 4. Verify the peering
```sh
buh-cli peer ping <peer-edge-host>:31415
```
`peer ping` performs a real mutual PQ-mTLS handshake and reports the peer's
advertised CA fingerprint + health. It **succeeds only when both directions of
trust are in place** — if it fails, the usual cause is that one side hasn't pinned
the other yet (or the fingerprints don't match what was exchanged).
## Operations
- **Revoke a peer:** `buh-cli peer distrust <ca-fp>` — refused on their next
handshake, no restart.
- **Re-key this node:** `buh-cli ca rotate --force` — destructive; generates a new
CA (old one backed up to `*.bak`), so **every** peer must re-pin the new
fingerprint and the roster must be updated.
- **Keep the admin API loopback:** it has no auth; the daemon refuses a
non-loopback `[admin].bind`, and it must never be opened in firewalld.
- **Changing the port:** `BUH_NODE_PORT` is repo-side only — edit
`NODE_PORT`/`node_bind` and `asset/firewalld/buh-node.xml`; the firewalld rules
are keyed on the service name, and the node runs unconfined (no SELinux port
label needed), so no host re-provisioning is required. Re-point edge forwarding
to the new port.
## Reference
- CA fingerprint = lowercase hex SHA-256 of the CA cert DER (`buh-cli ca show`).
- State: `/var/lib/buh` (`relay.db`, `pki/`, `blobs/`); config: `/etc/buh/config.toml`.
- CLI talks to a running node via `--admin-url` (default `http://127.0.0.1:8081`,
env `BUH_ADMIN_URL`).