diff --git a/generic.md b/generic.md index 2dad34b..f9ac398 100644 --- a/generic.md +++ b/generic.md @@ -390,6 +390,8 @@ If the service must run as root or with extra capabilities, document the reason All hosts run `firewalld`. Every service that listens on a port must ship a **named firewalld service definition** rather than opening bare ports in a zone. The service name matches the systemd unit name (minus `.service`). +**Choosing the port itself** — an uncommon number in the registered range, never a crowded alt-HTTP port (`8080`/`8081`/…) or the ephemeral range, recorded in a fleet registry — is covered in **`port-allocations.md`**. The number there must match this XML, the service's bind config, and the SELinux `semanage port` label (§10). + ### Why named services Named services are self-documenting (`firewall-cmd --list-services` tells you what's actually running), removable atomically on app decommission, and survive zone reassignment without reconfiguration. diff --git a/port-allocations.md b/port-allocations.md new file mode 100644 index 0000000..51881e3 --- /dev/null +++ b/port-allocations.md @@ -0,0 +1,105 @@ +# Port allocations for internal services + +Extends `generic.md` §9 (firewalld) and §3 (Binaries and Runtime). Every service that +listens on a TCP port needs a port *number*, and on a fleet where many services share +hosts — and where a runner, a proxy, and an app can land on the same box — a careless +choice collides. This doc is how we pick numbers that don't, and the **registry** of what's +already assigned. + +Decision rule (the whole thing in one line): + +> **Pick an uncommon port in the IANA *registered* range (1024–49151), never a crowded +> alt-HTTP port and never the ephemeral range, and record it in the registry below.** + +The registry — not cleverness in the number — is what actually prevents collisions. Check +it before you assign; add your service when you do. + +--- + +## 1. Why the default ports bite + +The ports frameworks and tools reach for by default form a dense cluster that half the +software world also uses: + +`3000` `3001` `4000` `5000` `5173` `8000` `8008` `8080` **`8081`** `8443` `9000` `9090` … + +On a dedicated single-service host these are fine. On our fleet they are not: hosts run +gongfoo runners, nginx, sometimes more than one app, plus whatever a container or a dev +shell brought up. Two things pick `8080`/`8081` and one fails to bind (`EADDRINUSE`) — or, +worse, *doesn't* fail: it binds a different interface, the health probe hits the **other** +service on that port, and you debug a green check that lies. Avoid the whole cluster. + +## 2. The three ranges + +| Range | IANA class | Use for a listener? | +| --- | --- | --- | +| `0–1023` | well-known / system | **No.** Requires privilege; reserved for standard services. TLS terminates at nginx (`:443`), so apps don't need these. | +| `1024–49151` | **registered / user** | **Yes.** This is where our services live. | +| `49152–65535` | dynamic / ephemeral | **No.** The kernel hands these out as *source* ports for outbound connections. A fixed listener here can collide with an ephemeral allocation and fail intermittently — the worst kind of bug. | + +So: **registered range, and specifically a sparse part of it.** The crowded cluster (§1) +sits at the bottom of the registered range; climb out of it. A good default band is +**`20000–29999`** — clear of the alt-HTTP mess below and the ephemeral range above, and +sparsely used. `30000–39999` is equally fine (cortex already lives there). + +## 3. Picking the number + +1. **Derive, don't guess.** Human-chosen ports cluster (everyone likes round numbers and + `x080`). Map the service name deterministically into the band instead: + + ```sh + name=newsfeed + python3 -c "import hashlib;print(20000 + int(hashlib.sha256('$name'.encode()).hexdigest(),16) % 10000)" + # newsfeed -> 22672 + ``` + + This is reproducible (anyone can re-derive it) and spreads services across the band. +2. **Check the registry (§5).** If the derived number is taken, bump by 1 until free, or + pick a memorable nearby number — then it's a deliberate, recorded choice. +3. **Record it** in the registry with the same change that introduces the service. +4. A memorable hand-picked number in the band is equally valid (`neuron` = `13131`, + `cortex` = `31313`). The rule is *uncommon + registered + recorded*, not *hashed*. + +## 4. Consequences elsewhere + +- **firewalld** (`generic.md` §9): ship the port in the app's named service XML. The + registry and the XML must agree. +- **SELinux** (`generic.md` §10): a non-standard port the daemon binds must be labelled, or + the bind is denied. `semanage port -a -t http_port_t -p tcp ` (guard with + `semanage port -l | grep` so re-runs are no-ops). Do this in the host-prep step + (`infra-setup.sh`), before the first service start. +- **Binding scope**: bind loopback (`127.0.0.1`) when nginx is on the same host, or the + mesh-routable address when the proxy is on a different host (e.g. an API on one host + fronted by the site's edge proxy on another). firewalld's named service bounds who may + reach it; the registry bounds what else may bind it. +- **Reverse proxy** (`reverse-proxies.md`): the nginx `upstream` points at + `:` — the same number as the registry and the service's bind config. + +## 5. The registry + +Canonical list of internal-service port assignments. **Append here when you allocate a +port**; treat a duplicate as a bug. (Seeded from the services documented across this repo; +extend as you encounter others — this table is authoritative going forward.) + +| Port | Service | Host(s) | Notes | +| --- | --- | --- | --- | +| `5432` | PostgreSQL | `magrathea`, `frankie` (standby) | standard well-known assignment; mTLS | +| `13131` | helexa **neuron** | each GPU host | OpenAI-compatible inference API | +| `13132` | helexa-bench read API | `bob` | benchmark telemetry | +| `18443` | gongfoo **controller** | `golgafrinchans` | agent-facing RPC (mTLS) | +| `22672` | **newsfeed-api** | `slartibartfast` | REST/JSON; fronted by oolon | +| `28443` | gongfoo **agent** | each runner host | controller-facing RPC (mTLS) | +| `31313` | helexa **cortex** API | `hanzalova` | unified OpenAI/Anthropic gateway | +| `31314` | helexa **cortex** metrics | `hanzalova` | Prometheus | + +`443` (nginx edge, every proxy host) and the ephemeral range are intentionally omitted — +they aren't allocations. + +## 6. Checklist for a new listening service + +1. Derive a candidate port in `20000–29999` from the service name (§3). +2. Check it's free in the registry (§5); bump if taken. +3. Record it in the registry in the same change. +4. Put the number in exactly these places, all agreeing: the service's bind config, its + firewalld service XML (§9), its SELinux `semanage port` label (§10, in host prep), and + any nginx `upstream` (`reverse-proxies.md`). diff --git a/readme.md b/readme.md index 5eb6c33..9be7422 100644 --- a/readme.md +++ b/readme.md @@ -13,6 +13,7 @@ The goal is boring consistency: the same crate layout, the same deploy flow, the - **`generic.md`** — the baseline. Applies to every project unless that project explicitly overrides a section. Covers workspace layout, separation of concerns, configuration, secrets, deployment, service accounts, firewalld, SELinux, and code quality. - **`deployment-gitea-actions.md`** — CI-driven deployment via a Gitea Actions workflow, as an alternative to the `deploy.sh` + `manifest.yml` flow in `generic.md` §7. The workflow is the source of infra truth; the runner deploys as a scoped `gitea_ci` user. - **`gitea-runners.md`** — the catalogue of `gongfoo`-managed CI runner images (what `runs-on:` label gives you which toolchain), how to pick the right one, and how to add or extend an image instead of `dnf`-installing at run time. Makes `deployment-gitea-actions.md` §5 concrete. +- **`port-allocations.md`** — how to choose a conflict-unlikely TCP port for an internal service (uncommon, registered range, not the alt-HTTP cluster or the ephemeral range) and the fleet registry of what's already assigned. Extends `generic.md` §9/§3. - **`internal-tls.md`** — provisioning and renewing per-service internal TLS certs (`.internal`) for mesh-only nginx vhosts, extending the PKI conventions in `generic.md` §11. - **`external-tls.md`** — publicly-trusted certs for WAN-facing vhosts via Let's Encrypt (certbot, Cloudflare DNS-01, ECDSA). The external counterpart to `internal-tls.md`. - **`reverse-proxies.md`** — the per-site nginx edge proxies (`oolon` for kosherinata, `hanzalova.internal` for the office), what sits behind each, the public-vs-mesh access paths, and the per-vhost cert choice. Names the topology behind `generic.md` §11 Ingress.