From 746c55fe94995f6d3a52477bd345fa42aa7544da Mon Sep 17 00:00:00 2001 From: rob thijssen Date: Sun, 14 Jun 2026 15:50:57 +0300 Subject: [PATCH] docs: add reverse-proxy topology + external-TLS conventions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Capture the cert + edge-proxy conventions worked through deploying the helexa-bench UI: - external-tls.md — publicly-trusted certs via Let's Encrypt (certbot, Cloudflare DNS-01, ECDSA, /root/.certbot-internal); the external counterpart to internal-tls.md. Decision rule: public name → LE, *.internal → internal CA. - reverse-proxies.md — names the per-site edge proxies (oolon for kosherinata, hanzalova.internal for the office) and what sits behind each, the public-vs-mesh access paths + the "public names don't hairpin from inside the mesh" gotcha, per-vhost cert choice, nginx conventions, and the bench (bench.helexa.ai + bench.internal) worked example. - readme + generic.md §11 cross-reference both. Co-Authored-By: Claude Opus 4.8 (1M context) --- external-tls.md | 113 +++++++++++++++++++++++++++++++++++++++++++++ generic.md | 2 +- readme.md | 2 + reverse-proxies.md | 87 ++++++++++++++++++++++++++++++++++ 4 files changed, 203 insertions(+), 1 deletion(-) create mode 100644 external-tls.md create mode 100644 reverse-proxies.md diff --git a/external-tls.md b/external-tls.md new file mode 100644 index 0000000..f08f96e --- /dev/null +++ b/external-tls.md @@ -0,0 +1,113 @@ +# External TLS: public certs for WAN-facing vhosts + +Extends `generic.md` §11 (TLS / PKI). That section and `internal-tls.md` cover the +**internal** PKI (Smallstep `step-ca`, `*.internal` names, mesh-only). This doc covers +the other half: **publicly-trusted certs for names served to the public internet** at a +site's WAN edge — e.g. `bench.helexa.ai`, `qapish.ai`, `*.zap.pics`. + +Decision rule (the whole strategy in one line): + +> **Public, internet-resolvable name → Let's Encrypt. Mesh-only `*.internal` name → +> internal CA (`internal-tls.md`).** A service reached both ways gets one vhost of each +> (see `reverse-proxies.md`). + +Public certs must chain to a publicly-trusted root (browsers off the mesh don't trust +the `lair` internal root), so these come from Let's Encrypt — never `step-ca`. + +--- + +## 1. Issuance: certbot + Cloudflare DNS-01, ECDSA + +Our public DNS zones are on Cloudflare, so we use the **DNS-01** challenge via the +`certbot-dns-cloudflare` plugin. DNS-01 is deliberate: + +- **No inbound :80 needed.** The challenge is a TXT record, not an HTTP hit — so a cert + can be issued (or renewed) even while nginx is stopped or the host isn't yet reachable + from the WAN. (This is why a dormant edge proxy doesn't block issuance.) +- **Wildcard-capable**, if a zone ever wants `*.example.com`. + +Keys are **ECDSA** (`--key-type ecdsa`), matching the rest of the fleet. + +```sh +sudo certbot certonly \ + -m ops@zap.pics --agree-tos --no-eff-email --noninteractive \ + --cert-name \ + --key-type ecdsa \ + --dns-cloudflare \ + --dns-cloudflare-credentials /root/.certbot-internal \ + --dns-cloudflare-propagation-seconds 60 \ + --keep-until-expiring \ + -d +``` + +- **`/root/.certbot-internal`** holds the Cloudflare API token. One token covers all the + zones we manage (`helexa.ai`, `zap.pics`, …), so new sub-domains under an existing zone + need no new credential — just run the command. +- **`--keep-until-expiring`** makes scripted/repeated runs idempotent (no-op if the cert + is still valid), so this is safe to call unconditionally from `infra-setup.sh`. +- `--cert-name ` pins the lineage name so the cert lands at a predictable path + regardless of `-d` ordering. + +## 2. Paths + +certbot's standard layout (do **not** relocate — the renew timer expects it): + +| Path | Contents | +| --- | --- | +| `/etc/letsencrypt/live//fullchain.pem` | cert + intermediate chain | +| `/etc/letsencrypt/live//privkey.pem` | private key | + +These live under root-only `/etc/letsencrypt/live` (`0700`). Scripts that check for an +existing cert must `sudo test -d /etc/letsencrypt/live/` — an unprivileged +`test` silently returns false and will wrongly conclude the cert is missing. + +## 3. Renewal + +Automatic via certbot's own `certbot-renew.timer` (systemd) — **no per-cert unit**, +unlike the internal `step@` template. certbot renews any lineage within 30 days of +expiry and runs the configured deploy hook. Ensure nginx reloads after renewal with a +deploy hook (once per host): + +```sh +# /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh (chmod +x) +#!/bin/sh +systemctl reload nginx 2>/dev/null || true +``` + +## 4. nginx wiring + +```nginx +server { + listen 443 ssl; + http2 on; + server_name ; + + ssl_certificate /etc/letsencrypt/live//fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live//privkey.pem; + ssl_protocols TLSv1.2 TLSv1.3; +} +``` + +Keep an `:80` server for the same name only if you want an HTTP→HTTPS redirect; the +cert itself needs no `:80` (DNS-01). Never reference a cert path before the cert exists — +`nginx -t` fails on a missing `ssl_certificate` file and blocks **all** of nginx from +(re)starting. Issue first, then install the TLS vhost (gate the vhost install on +`sudo test -d /etc/letsencrypt/live/`). + +## 5. Gotchas + +- **SAN, not CN.** Modern clients ignore CN; the served name must be in the SAN. certbot + sets SAN from `-d`, so this is automatic — but if `curl` reports *"no alternative + certificate subject name matches target hostname"*, the listener answering isn't the + one holding this cert (see next point). +- **Wrong cert on the public endpoint = a routing problem, not a cert problem.** If a + public name returns something like `CN=opnsense..internal`, the WAN `:443` + forward (or HAProxy SNI route) on OPNsense isn't landing on the site's nginx. Fix the + edge route (`reverse-proxies.md` §2), not the cert. + +## 6. Checklist for a new public vhost + +1. Add the public DNS record on Cloudflare (unproxied by default — `generic.md` §11). +2. Issue the cert (§1), from `infra-setup.sh`, idempotently. +3. Point the nginx vhost at the `live/` paths (§4); `nginx -t` && reload. +4. Confirm the site's OPNsense forwards WAN `:443` to this nginx (`reverse-proxies.md`). diff --git a/generic.md b/generic.md index d2637d4..a45b7e0 100644 --- a/generic.md +++ b/generic.md @@ -536,7 +536,7 @@ templated `step@` unit. That pattern is documented separately in **`internal-tls.md`**. ### Ingress -- Per-site nginx reverse proxy terminates all WAN inbound 443. +- Per-site nginx reverse proxy terminates all WAN inbound 443 (`oolon` for kosherinata, `hanzalova.internal` for the office). The named topology, the public-vs-mesh access paths (and the hairpin gotcha), and the per-vhost cert choice are in **`reverse-proxies.md`**; external (Let's Encrypt) cert provisioning in **`external-tls.md`**. - Public DNS via Cloudflare, **unproxied by default** (CF's mTLS origin-pull has been unreliable). Revisit if/when that changes. - nginx serves static frontends directly from `/var/www/` and reverse-proxies API traffic to the internal host:port from `manifest.yml`. diff --git a/readme.md b/readme.md index 1a796f9..6221622 100644 --- a/readme.md +++ b/readme.md @@ -13,6 +13,8 @@ The goal is boring consistency: the same crate layout, the same deploy flow, the - **`generic.md`** — the baseline. Applies to every project unless that project explicitly overrides a section. Covers workspace layout, separation of concerns, configuration, secrets, deployment, service accounts, firewalld, SELinux, and code quality. - **`deployment-gitea-actions.md`** — CI-driven deployment via a Gitea Actions workflow, as an alternative to the `deploy.sh` + `manifest.yml` flow in `generic.md` §7. The workflow is the source of infra truth; the runner deploys as a scoped `gitea_ci` user. - **`internal-tls.md`** — provisioning and renewing per-service internal TLS certs (`.internal`) for mesh-only nginx vhosts, extending the PKI conventions in `generic.md` §11. +- **`external-tls.md`** — publicly-trusted certs for WAN-facing vhosts via Let's Encrypt (certbot, Cloudflare DNS-01, ECDSA). The external counterpart to `internal-tls.md`. +- **`reverse-proxies.md`** — the per-site nginx edge proxies (`oolon` for kosherinata, `hanzalova.internal` for the office), what sits behind each, the public-vs-mesh access paths, and the per-vhost cert choice. Names the topology behind `generic.md` §11 Ingress. More files will appear here over time as guidance that's more specific than `generic.md` gets extracted — per-stack, per-deployment-target, or per-problem-domain documents. When a project needs guidance that isn't generic, it belongs in a new file here, not buried in one project's repo. diff --git a/reverse-proxies.md b/reverse-proxies.md new file mode 100644 index 0000000..baa9689 --- /dev/null +++ b/reverse-proxies.md @@ -0,0 +1,87 @@ +# Reverse proxies and edge ingress + +Extends `generic.md` §11 (Network / Ingress). That section says "per-site nginx reverse +proxy terminates all WAN inbound 443"; this doc names the proxies, maps what sits behind +each, and pins down the two access paths and the per-vhost cert choice — plus the one +gotcha that bites every time (a public name doesn't work from *inside* the mesh). + +--- + +## 1. The proxies (one per site) + +Each WireGuard site has a single nginx edge proxy. All WAN-inbound 443 for that site is +port-forwarded by the site's OPNsense router to its proxy, which terminates TLS and +fans out to internal upstreams. + +| Site | Edge proxy (nginx host) | Notable hosts behind it | +| --- | --- | --- | +| **kosherinata** (DC) | `oolon.kosherinata.internal` | `magrathea` (Postgres primary), `nikola`, `gramathea`, … | +| **hanzalova** (office) | `hanzalova.internal` | GPU/inference: `beast`, `benjy`, `quadbrat`; `bob` (helexa-bench API + Agent Zero); `frankie` (Postgres streaming standby); `trillian`; the workstation | + +Site octet encodes the mesh subnet (`10..0.0/16`); see `generic.md` §11. New +office services front on `hanzalova.internal`; new DC services on `oolon`. + +## 2. Two access paths — and the mesh hairpin gotcha + +A service can be reached two ways, and they are **not** interchangeable: + +- **Public (from the WAN):** public DNS (Cloudflare, unproxied by default) → site WAN IP + → OPNsense forwards `:443` → site nginx → upstream. Cert: **Let's Encrypt** + (`external-tls.md`). +- **Internal (from the mesh):** split-horizon `.internal` DNS → the host/proxy directly + over WireGuard → nginx. Cert: **internal CA** (`internal-tls.md`). + +> **Gotcha — public names don't hairpin.** From *inside* the mesh, a public name still +> resolves (via public DNS) to the site's **WAN** IP, so the packet hits the OPNsense +> **LAN** interface — which only forwards `:443` inbound from the **WAN**, not from the +> LAN. The connection dead-ends (or worse, gets OPNsense's own default cert). So a +> service that mesh clients also need must be published under a **`*.internal` name with +> its own internal-CA vhost**, in addition to its public vhost. + +This is why dual-audience services get **two vhosts** on the same proxy — one public +(LE), one internal (`lair` CA) — usually sharing one webroot and one upstream. + +## 3. Per-vhost cert choice + +| vhost audience | name | cert | doc | +| --- | --- | --- | --- | +| Public / WAN | `.` (e.g. `bench.helexa.ai`) | Let's Encrypt (certbot, Cloudflare DNS-01, ECDSA) | `external-tls.md` | +| Mesh-only | `.internal` | internal CA (`step ca`, `lair` provisioner, `step@` renewal) | `internal-tls.md` | + +Provisioner credentials for the internal CA (`~/.step/secrets/provisioner`, shipped to +the host transiently and removed) are covered in `internal-tls.md` §4. + +## 4. nginx conventions on the proxies + +- **`sites-available/` + `sites-enabled/` symlink**, included via + `/etc/nginx/conf.d/sites-enabled.conf` (`include /etc/nginx/sites-enabled/*.conf;`). + One file per `server_name`; enable with a relative symlink + (`ln -sf ../sites-available/.conf /etc/nginx/sites-enabled/`). +- **Static SPA** served from `/var/www/` with SPA fallback + (`try_files $uri $uri/ /index.html;`); **API** reverse-proxied to the internal + `host:port`. Internal vhosts add `ssl_trusted_certificate ` and pin + `ssl_protocols TLSv1.3`. +- **SELinux (enforcing):** webroots must be labelled `httpd_sys_content_t` or nginx + returns **403**. After creating/populating `/var/www/`, run + `restorecon -R /var/www/`; rsynced files inherit the dir's type. +- **Never reference a cert path before the cert exists** — `nginx -t` fails on a missing + `ssl_certificate` and blocks the whole server from (re)starting. Issue the cert, then + install the TLS vhost (gate on the cert's presence; serve an http-only bootstrap until + then if needed). +- Config + cert/renewal wiring is installed idempotently from each project's + `infra-setup.sh` (`deployment-gitea-actions.md` §2); the recurring artifact rsync + (e.g. built SPA `dist/`) rides in the deploy workflow. + +## 5. Worked example: helexa-bench UI + +The bench visualisation is reached both ways, fronted by `hanzalova.internal`: + +| vhost | cert | DNS | +| --- | --- | --- | +| `bench.helexa.ai` (public) | Let's Encrypt | Cloudflare A → office WAN IP; OPNsense forwards WAN `:443` → `hanzalova` | +| `bench.internal` (mesh) | internal `lair` CA, renewed by `step@bench.timer` | split-horizon `bench.internal → hanzalova` mesh IP | + +Both vhosts share one webroot (`/var/www/bench.helexa.ai`, the built SPA) and proxy +`/api` to the helexa-bench read API on `bob.hanzalova.internal:13132`. The internal vhost +exists precisely because of §2: from a workstation on the mesh, `bench.helexa.ai` +hairpins to the OPNsense LAN interface and fails, so mesh users hit `bench.internal`.