feat(deploy): stake BUH_NODE_PORT on 31415 + testnet peering doc
Settle buh's standardised ingress port on 31415/tcp — unassigned in IANA, clear of the crowded 8443/9443/6443 appliance space, and below the Linux ephemeral range so it's stable to forward from the edge. Updated the code default (config.rs node_bind), the CI workflow env, firewalld service, operator-model deploy.sh/manifest.yml, and the asset readme. Also drop the SELinux `semanage port` step (and its two sudoers rules): buh-api runs as unconfined_service_t and may bind any port, so BUH_NODE_PORT needs no http_port_t label. The firewalld rules are keyed on the service NAME (buh-node), not the port — so a port change is now purely repo-side and needs no infra-setup re-run on the host. Add doc/testnet.md: the per-node-CA trust model, the 31415 convention, a node roster (incl. the slarti dogfood node + its CA fingerprint), and the mutual trust / peer ping workflow for joining the testnet. Gate: cargo fmt + clippy -D warnings clean; config renders node_bind 0.0.0.0:31415; sudoers passes visudo -c; deploy.yml valid YAML. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EB3LjarCdXxqrJ4tFLn8LB
This commit is contained in:
@@ -41,7 +41,7 @@ concurrency:
|
||||
env:
|
||||
# --- infra truth ---
|
||||
NODE_HOST: slartibartfast.kosherinata.internal
|
||||
NODE_PORT: "8443" # BUH_NODE_PORT — the only edge port (PQ-mTLS ingress)
|
||||
NODE_PORT: "31415" # BUH_NODE_PORT — buh's staked port; the only edge port (PQ-mTLS ingress)
|
||||
ADMIN_PORT: "8081" # loopback operator admin API (health probe target)
|
||||
NODE_SANS: '["slartibartfast.kosherinata.internal"]' # TOML array for the leaf SANs
|
||||
MUSL_TARGET: x86_64-unknown-linux-musl
|
||||
@@ -169,22 +169,22 @@ jobs:
|
||||
asset/firewalld/buh-node.xml \
|
||||
gitea_ci@"${NODE_HOST}":/etc/firewalld/services/buh-node.xml
|
||||
|
||||
- name: Apply SELinux + firewalld (idempotent)
|
||||
- name: Apply SELinux contexts + firewalld (idempotent)
|
||||
run: |
|
||||
ssh gitea_ci@"${NODE_HOST}" '
|
||||
set -euo pipefail
|
||||
sudo /usr/sbin/restorecon -R /usr/local/bin/buh-api /usr/local/bin/buh-cli /etc/buh /var/lib/buh
|
||||
# Label the PQ-mTLS ingress port (guarded so re-runs are no-ops; 8443
|
||||
# is usually already http_port_t).
|
||||
if ! sudo /usr/sbin/semanage port -l | grep -E "^http_port_t" | grep -qw '"${NODE_PORT}"'; then
|
||||
sudo /usr/sbin/semanage port -a -t http_port_t -p tcp '"${NODE_PORT}"'
|
||||
fi
|
||||
# firewalld only learns a freshly-shipped custom service after reload.
|
||||
# No SELinux port labeling: buh-api runs as unconfined_service_t (a plain
|
||||
# systemd unit with no policy module), which may bind any port — so
|
||||
# BUH_NODE_PORT needs no http_port_t entry. Keeping it out means a port
|
||||
# change is purely repo-side: the firewalld rules below are keyed on the
|
||||
# service NAME (buh-node), and the freshly-shipped buh-node.xml carries the
|
||||
# new port, so a reload picks it up with no sudoers/infra-setup change.
|
||||
sudo /usr/bin/firewall-cmd --reload
|
||||
if ! sudo /usr/bin/firewall-cmd --query-service=buh-node; then
|
||||
sudo /usr/bin/firewall-cmd --add-service=buh-node --permanent
|
||||
sudo /usr/bin/firewall-cmd --reload
|
||||
fi'
|
||||
fi
|
||||
sudo /usr/bin/firewall-cmd --reload'
|
||||
|
||||
- name: Restart buh-node
|
||||
run: |
|
||||
|
||||
@@ -50,7 +50,7 @@ here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
: "${PKI_ENABLED:=true}"
|
||||
: "${PKI_DIR:=${STATE_DIR}/pki}"
|
||||
: "${NODE_BIND:=0.0.0.0:8443}"
|
||||
: "${NODE_BIND:=0.0.0.0:31415}"
|
||||
: "${PKI_SANS:=[\"localhost\"]}"
|
||||
: "${LEAF_TTL_HOURS:=48}"
|
||||
: "${ROTATE_EVERY_HOURS:=24}"
|
||||
|
||||
@@ -5,5 +5,5 @@
|
||||
the relay/blob API served over X25519MLKEM768 mutual TLS, with peers pinned per-CA. The plain
|
||||
loopback health port (127.0.0.1:8080) is never opened. Forward this port from the edge
|
||||
(OPNsense/router) to the node host.</description>
|
||||
<port protocol="tcp" port="8443"/>
|
||||
<port protocol="tcp" port="31415"/>
|
||||
</service>
|
||||
|
||||
@@ -33,7 +33,7 @@ environments:
|
||||
pki:
|
||||
enabled: true
|
||||
dir: /var/lib/buh/pki # CA key + cert; the node generates these on first start
|
||||
node_bind: 0.0.0.0:8443 # BUH_NODE_PORT — the only port exposed at the edge
|
||||
node_bind: 0.0.0.0:31415 # BUH_NODE_PORT — the only port exposed at the edge
|
||||
sans: [node1.example.com] # hostnames/IPs the leaf answers to
|
||||
leaf_ttl_hours: 48
|
||||
rotate_every_hours: 24 # in-process leaf rotation (no step-ca, no .path units)
|
||||
|
||||
@@ -23,7 +23,7 @@ plane, no shared database, and no central PKI**:
|
||||
process* (`[relay].sweep_interval_seconds`), because Turso locks the datastore exclusively — a
|
||||
separate `buh-cli sweep` cannot open the DB while the daemon holds it.
|
||||
- `systemd/buh.sysusers.conf` — the unprivileged `buh` service account.
|
||||
- `firewalld/buh-node.xml` — opens **`BUH_NODE_PORT` (8443)**, the single PQ-mTLS ingress port.
|
||||
- `firewalld/buh-node.xml` — opens **`BUH_NODE_PORT` (31415)**, the single PQ-mTLS ingress port.
|
||||
The plain loopback health port is never exposed.
|
||||
- `deploy.sh` — installs the above on the local host, renders the config, has the node generate
|
||||
its CA, and prints the CA fingerprint to share with peers.
|
||||
|
||||
@@ -20,10 +20,12 @@ gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemd-sysusers
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/install -d -o root -g buh -m 0750 /etc/buh
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/install -d -o buh -g buh -m 0750 /var/lib/buh
|
||||
|
||||
# --- SELinux + firewalld ---
|
||||
# --- SELinux contexts + firewalld ---
|
||||
# No `semanage port` rules: buh-api runs as unconfined_service_t and may bind any
|
||||
# port, so BUH_NODE_PORT needs no http_port_t label. The firewalld rules below are
|
||||
# keyed on the service NAME (buh-node), not the port number, so changing
|
||||
# BUH_NODE_PORT is purely repo-side and needs no edit here / no infra-setup re-run.
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/restorecon -R /usr/local/bin/buh-api /usr/local/bin/buh-cli /etc/buh /var/lib/buh
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/semanage port -l
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/semanage port -a -t http_port_t -p tcp 8443
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/firewall-cmd --reload
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/firewall-cmd --query-service\=buh-node
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/firewall-cmd --add-service\=buh-node --permanent
|
||||
|
||||
@@ -131,7 +131,7 @@ impl Default for AppConfig {
|
||||
pki: PkiConfig {
|
||||
enabled: false,
|
||||
dir: "/var/lib/buh/pki".to_string(),
|
||||
node_bind: "0.0.0.0:8443".to_string(),
|
||||
node_bind: "0.0.0.0:31415".to_string(),
|
||||
sans: vec!["localhost".to_string()],
|
||||
leaf_ttl_hours: 48,
|
||||
rotate_every_hours: 24,
|
||||
|
||||
127
doc/testnet.md
Normal file
127
doc/testnet.md
Normal file
@@ -0,0 +1,127 @@
|
||||
# buh testnet — peering guide
|
||||
|
||||
How to stand up a buh node, join it to the testnet, and establish a peering with
|
||||
another node. buh is an **anti-hub**: there is no central server, no shared root
|
||||
CA, and no registry. A "testnet" is just a set of nodes that have chosen to trust
|
||||
each other's certificate authorities.
|
||||
|
||||
## Trust model in one paragraph
|
||||
|
||||
Every node is **its own CA**. On first start it generates a CA (`ca.cert.der` +
|
||||
`ca.key.pem` under its pki dir) and rotates a short-lived TLS leaf from it in
|
||||
process. Its identity is the **CA fingerprint** — the lowercase hex SHA-256 of the
|
||||
CA certificate DER (`:` separators are accepted on input but not required). Two
|
||||
nodes talk over **mutual** PQ-mTLS (X25519MLKEM768): a handshake succeeds only
|
||||
when **each side has pinned the other's CA fingerprint**. Trust is therefore
|
||||
explicit, pairwise, and symmetric — there is nothing to "join", only peers to
|
||||
exchange fingerprints with. Removing trust (`peer distrust`) takes effect on the
|
||||
peer's **next handshake**, with no restart.
|
||||
|
||||
## The staked port: 31415
|
||||
|
||||
buh standardises on **`BUH_NODE_PORT = 31415/tcp`** — the single PQ-mTLS ingress
|
||||
port a node exposes (chosen to avoid the crowded `8443`/`9443`/`6443` space; it is
|
||||
unassigned in IANA and below the Linux ephemeral range, so it is safe to forward).
|
||||
Everything else stays local:
|
||||
|
||||
| Port | Bind | Purpose | Exposed? |
|
||||
|------|------|---------|----------|
|
||||
| `31415` | `0.0.0.0:31415` | PQ-mTLS relay/blob API (peers + clients) | **yes — forward from the edge** |
|
||||
| `8081` | `127.0.0.1:8081` | operator admin API (peer-trust mgmt) | no — loopback only, no auth |
|
||||
| `8080` | `127.0.0.1:8080` | plain health/debug (only when pki is off) | no |
|
||||
|
||||
Forward `31415/tcp` from your edge (OPNsense/router) to the node host. Peers reach
|
||||
you at `your-edge-hostname:31415`.
|
||||
|
||||
## Testnet roster
|
||||
|
||||
Record each node's reachable address and CA fingerprint here as it joins.
|
||||
|
||||
| Node | Edge address (`host:31415`) | CA fingerprint | Role |
|
||||
|------|------------------------------|----------------|------|
|
||||
| `slartibartfast` (dogfood) | _\<edge-fqdn\>_:31415 — internal mesh `slartibartfast.kosherinata.internal:31415` | `3c8f125861f3c39f849a469cb32ef599000c71896ea1ccc8a5baaad7419ef808` | relay + blob (fs) |
|
||||
|
||||
> Fill in slarti's `<edge-fqdn>` once edge port-forwarding to `31415` is in place.
|
||||
> Re-keying a node (`buh-cli ca rotate --force`) changes its fingerprint — update
|
||||
> this table and every peer must re-pin.
|
||||
|
||||
## Join the testnet
|
||||
|
||||
### 1. Stand up your node
|
||||
|
||||
Either model from `asset/readme.md`:
|
||||
|
||||
- **CI** — push to a buh checkout whose `.gitea/workflows/deploy.yml` targets your
|
||||
host (the dogfood node uses this).
|
||||
- **Manual** — `sudo PKI_SANS='["your-node.example"]' ./asset/deploy.sh` on the host.
|
||||
|
||||
Confirm it is healthy (on the host, via the loopback admin API):
|
||||
|
||||
```sh
|
||||
curl -fsS http://127.0.0.1:8081/admin/info
|
||||
# {"ca_fingerprint":"…","trusted_peers":N}
|
||||
```
|
||||
|
||||
### 2. Learn your CA fingerprint
|
||||
|
||||
```sh
|
||||
buh-cli ca show # prints this node's CA fingerprint
|
||||
```
|
||||
|
||||
This is the value you hand to peers. Exchange fingerprints **out of band** (signal,
|
||||
in person, an existing secure channel) — buh deliberately has no fingerprint
|
||||
directory to lie to you.
|
||||
|
||||
### 3. Trust your peer, and have them trust you
|
||||
|
||||
Peering is symmetric — **both** sides run a `trust`:
|
||||
|
||||
```sh
|
||||
# on YOUR node — pin the peer's CA
|
||||
buh-cli peer trust 3c8f125861f3c39f849a469cb32ef599000c71896ea1ccc8a5baaad7419ef808 \
|
||||
--note "slartibartfast (dogfood)"
|
||||
|
||||
# on the PEER node — they pin YOUR CA
|
||||
buh-cli peer trust <your-ca-fingerprint> --note "my-node"
|
||||
```
|
||||
|
||||
`peer` commands talk to the running node's loopback admin API, so trust changes are
|
||||
live — they apply on the next handshake without a restart. (With the daemon
|
||||
stopped, the CLI falls back to opening the datastore directly.) Review with:
|
||||
|
||||
```sh
|
||||
buh-cli peer list
|
||||
```
|
||||
|
||||
### 4. Verify the peering
|
||||
|
||||
```sh
|
||||
buh-cli peer ping <peer-edge-host>:31415
|
||||
```
|
||||
|
||||
`peer ping` performs a real mutual PQ-mTLS handshake and reports the peer's
|
||||
advertised CA fingerprint + health. It **succeeds only when both directions of
|
||||
trust are in place** — if it fails, the usual cause is that one side hasn't pinned
|
||||
the other yet (or the fingerprints don't match what was exchanged).
|
||||
|
||||
## Operations
|
||||
|
||||
- **Revoke a peer:** `buh-cli peer distrust <ca-fp>` — refused on their next
|
||||
handshake, no restart.
|
||||
- **Re-key this node:** `buh-cli ca rotate --force` — destructive; generates a new
|
||||
CA (old one backed up to `*.bak`), so **every** peer must re-pin the new
|
||||
fingerprint and the roster must be updated.
|
||||
- **Keep the admin API loopback:** it has no auth; the daemon refuses a
|
||||
non-loopback `[admin].bind`, and it must never be opened in firewalld.
|
||||
- **Changing the port:** `BUH_NODE_PORT` is repo-side only — edit
|
||||
`NODE_PORT`/`node_bind` and `asset/firewalld/buh-node.xml`; the firewalld rules
|
||||
are keyed on the service name, and the node runs unconfined (no SELinux port
|
||||
label needed), so no host re-provisioning is required. Re-point edge forwarding
|
||||
to the new port.
|
||||
|
||||
## Reference
|
||||
|
||||
- CA fingerprint = lowercase hex SHA-256 of the CA cert DER (`buh-cli ca show`).
|
||||
- State: `/var/lib/buh` (`relay.db`, `pki/`, `blobs/`); config: `/etc/buh/config.toml`.
|
||||
- CLI talks to a running node via `--admin-url` (default `http://127.0.0.1:8081`,
|
||||
env `BUH_ADMIN_URL`).
|
||||
Reference in New Issue
Block a user