docs: add reverse-proxy topology + external-TLS conventions
Capture the cert + edge-proxy conventions worked through deploying the helexa-bench UI: - external-tls.md — publicly-trusted certs via Let's Encrypt (certbot, Cloudflare DNS-01, ECDSA, /root/.certbot-internal); the external counterpart to internal-tls.md. Decision rule: public name → LE, *.internal → internal CA. - reverse-proxies.md — names the per-site edge proxies (oolon for kosherinata, hanzalova.internal for the office) and what sits behind each, the public-vs-mesh access paths + the "public names don't hairpin from inside the mesh" gotcha, per-vhost cert choice, nginx conventions, and the bench (bench.helexa.ai + bench.internal) worked example. - readme + generic.md §11 cross-reference both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
113
external-tls.md
Normal file
113
external-tls.md
Normal file
@@ -0,0 +1,113 @@
|
||||
# External TLS: public certs for WAN-facing vhosts
|
||||
|
||||
Extends `generic.md` §11 (TLS / PKI). That section and `internal-tls.md` cover the
|
||||
**internal** PKI (Smallstep `step-ca`, `*.internal` names, mesh-only). This doc covers
|
||||
the other half: **publicly-trusted certs for names served to the public internet** at a
|
||||
site's WAN edge — e.g. `bench.helexa.ai`, `qapish.ai`, `*.zap.pics`.
|
||||
|
||||
Decision rule (the whole strategy in one line):
|
||||
|
||||
> **Public, internet-resolvable name → Let's Encrypt. Mesh-only `*.internal` name →
|
||||
> internal CA (`internal-tls.md`).** A service reached both ways gets one vhost of each
|
||||
> (see `reverse-proxies.md`).
|
||||
|
||||
Public certs must chain to a publicly-trusted root (browsers off the mesh don't trust
|
||||
the `lair` internal root), so these come from Let's Encrypt — never `step-ca`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Issuance: certbot + Cloudflare DNS-01, ECDSA
|
||||
|
||||
Our public DNS zones are on Cloudflare, so we use the **DNS-01** challenge via the
|
||||
`certbot-dns-cloudflare` plugin. DNS-01 is deliberate:
|
||||
|
||||
- **No inbound :80 needed.** The challenge is a TXT record, not an HTTP hit — so a cert
|
||||
can be issued (or renewed) even while nginx is stopped or the host isn't yet reachable
|
||||
from the WAN. (This is why a dormant edge proxy doesn't block issuance.)
|
||||
- **Wildcard-capable**, if a zone ever wants `*.example.com`.
|
||||
|
||||
Keys are **ECDSA** (`--key-type ecdsa`), matching the rest of the fleet.
|
||||
|
||||
```sh
|
||||
sudo certbot certonly \
|
||||
-m ops@zap.pics --agree-tos --no-eff-email --noninteractive \
|
||||
--cert-name <domain> \
|
||||
--key-type ecdsa \
|
||||
--dns-cloudflare \
|
||||
--dns-cloudflare-credentials /root/.certbot-internal \
|
||||
--dns-cloudflare-propagation-seconds 60 \
|
||||
--keep-until-expiring \
|
||||
-d <domain>
|
||||
```
|
||||
|
||||
- **`/root/.certbot-internal`** holds the Cloudflare API token. One token covers all the
|
||||
zones we manage (`helexa.ai`, `zap.pics`, …), so new sub-domains under an existing zone
|
||||
need no new credential — just run the command.
|
||||
- **`--keep-until-expiring`** makes scripted/repeated runs idempotent (no-op if the cert
|
||||
is still valid), so this is safe to call unconditionally from `infra-setup.sh`.
|
||||
- `--cert-name <domain>` pins the lineage name so the cert lands at a predictable path
|
||||
regardless of `-d` ordering.
|
||||
|
||||
## 2. Paths
|
||||
|
||||
certbot's standard layout (do **not** relocate — the renew timer expects it):
|
||||
|
||||
| Path | Contents |
|
||||
| --- | --- |
|
||||
| `/etc/letsencrypt/live/<domain>/fullchain.pem` | cert + intermediate chain |
|
||||
| `/etc/letsencrypt/live/<domain>/privkey.pem` | private key |
|
||||
|
||||
These live under root-only `/etc/letsencrypt/live` (`0700`). Scripts that check for an
|
||||
existing cert must `sudo test -d /etc/letsencrypt/live/<domain>` — an unprivileged
|
||||
`test` silently returns false and will wrongly conclude the cert is missing.
|
||||
|
||||
## 3. Renewal
|
||||
|
||||
Automatic via certbot's own `certbot-renew.timer` (systemd) — **no per-cert unit**,
|
||||
unlike the internal `step@<name>` template. certbot renews any lineage within 30 days of
|
||||
expiry and runs the configured deploy hook. Ensure nginx reloads after renewal with a
|
||||
deploy hook (once per host):
|
||||
|
||||
```sh
|
||||
# /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh (chmod +x)
|
||||
#!/bin/sh
|
||||
systemctl reload nginx 2>/dev/null || true
|
||||
```
|
||||
|
||||
## 4. nginx wiring
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 443 ssl;
|
||||
http2 on;
|
||||
server_name <domain>;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/<domain>/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/<domain>/privkey.pem;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
}
|
||||
```
|
||||
|
||||
Keep an `:80` server for the same name only if you want an HTTP→HTTPS redirect; the
|
||||
cert itself needs no `:80` (DNS-01). Never reference a cert path before the cert exists —
|
||||
`nginx -t` fails on a missing `ssl_certificate` file and blocks **all** of nginx from
|
||||
(re)starting. Issue first, then install the TLS vhost (gate the vhost install on
|
||||
`sudo test -d /etc/letsencrypt/live/<domain>`).
|
||||
|
||||
## 5. Gotchas
|
||||
|
||||
- **SAN, not CN.** Modern clients ignore CN; the served name must be in the SAN. certbot
|
||||
sets SAN from `-d`, so this is automatic — but if `curl` reports *"no alternative
|
||||
certificate subject name matches target hostname"*, the listener answering isn't the
|
||||
one holding this cert (see next point).
|
||||
- **Wrong cert on the public endpoint = a routing problem, not a cert problem.** If a
|
||||
public name returns something like `CN=opnsense.<site>.internal`, the WAN `:443`
|
||||
forward (or HAProxy SNI route) on OPNsense isn't landing on the site's nginx. Fix the
|
||||
edge route (`reverse-proxies.md` §2), not the cert.
|
||||
|
||||
## 6. Checklist for a new public vhost
|
||||
|
||||
1. Add the public DNS record on Cloudflare (unproxied by default — `generic.md` §11).
|
||||
2. Issue the cert (§1), from `infra-setup.sh`, idempotently.
|
||||
3. Point the nginx vhost at the `live/<domain>` paths (§4); `nginx -t` && reload.
|
||||
4. Confirm the site's OPNsense forwards WAN `:443` to this nginx (`reverse-proxies.md`).
|
||||
@@ -536,7 +536,7 @@ templated `step@<name>` unit. That pattern is documented separately in
|
||||
**`internal-tls.md`**.
|
||||
|
||||
### Ingress
|
||||
- Per-site nginx reverse proxy terminates all WAN inbound 443.
|
||||
- Per-site nginx reverse proxy terminates all WAN inbound 443 (`oolon` for kosherinata, `hanzalova.internal` for the office). The named topology, the public-vs-mesh access paths (and the hairpin gotcha), and the per-vhost cert choice are in **`reverse-proxies.md`**; external (Let's Encrypt) cert provisioning in **`external-tls.md`**.
|
||||
- Public DNS via Cloudflare, **unproxied by default** (CF's mTLS origin-pull has been unreliable). Revisit if/when that changes.
|
||||
- nginx serves static frontends directly from `/var/www/<app>` and reverse-proxies API traffic to the internal host:port from `manifest.yml`.
|
||||
|
||||
|
||||
@@ -13,6 +13,8 @@ The goal is boring consistency: the same crate layout, the same deploy flow, the
|
||||
- **`generic.md`** — the baseline. Applies to every project unless that project explicitly overrides a section. Covers workspace layout, separation of concerns, configuration, secrets, deployment, service accounts, firewalld, SELinux, and code quality.
|
||||
- **`deployment-gitea-actions.md`** — CI-driven deployment via a Gitea Actions workflow, as an alternative to the `deploy.sh` + `manifest.yml` flow in `generic.md` §7. The workflow is the source of infra truth; the runner deploys as a scoped `gitea_ci` user.
|
||||
- **`internal-tls.md`** — provisioning and renewing per-service internal TLS certs (`<service>.internal`) for mesh-only nginx vhosts, extending the PKI conventions in `generic.md` §11.
|
||||
- **`external-tls.md`** — publicly-trusted certs for WAN-facing vhosts via Let's Encrypt (certbot, Cloudflare DNS-01, ECDSA). The external counterpart to `internal-tls.md`.
|
||||
- **`reverse-proxies.md`** — the per-site nginx edge proxies (`oolon` for kosherinata, `hanzalova.internal` for the office), what sits behind each, the public-vs-mesh access paths, and the per-vhost cert choice. Names the topology behind `generic.md` §11 Ingress.
|
||||
|
||||
More files will appear here over time as guidance that's more specific than `generic.md` gets extracted — per-stack, per-deployment-target, or per-problem-domain documents. When a project needs guidance that isn't generic, it belongs in a new file here, not buried in one project's repo.
|
||||
|
||||
|
||||
87
reverse-proxies.md
Normal file
87
reverse-proxies.md
Normal file
@@ -0,0 +1,87 @@
|
||||
# Reverse proxies and edge ingress
|
||||
|
||||
Extends `generic.md` §11 (Network / Ingress). That section says "per-site nginx reverse
|
||||
proxy terminates all WAN inbound 443"; this doc names the proxies, maps what sits behind
|
||||
each, and pins down the two access paths and the per-vhost cert choice — plus the one
|
||||
gotcha that bites every time (a public name doesn't work from *inside* the mesh).
|
||||
|
||||
---
|
||||
|
||||
## 1. The proxies (one per site)
|
||||
|
||||
Each WireGuard site has a single nginx edge proxy. All WAN-inbound 443 for that site is
|
||||
port-forwarded by the site's OPNsense router to its proxy, which terminates TLS and
|
||||
fans out to internal upstreams.
|
||||
|
||||
| Site | Edge proxy (nginx host) | Notable hosts behind it |
|
||||
| --- | --- | --- |
|
||||
| **kosherinata** (DC) | `oolon.kosherinata.internal` | `magrathea` (Postgres primary), `nikola`, `gramathea`, … |
|
||||
| **hanzalova** (office) | `hanzalova.internal` | GPU/inference: `beast`, `benjy`, `quadbrat`; `bob` (helexa-bench API + Agent Zero); `frankie` (Postgres streaming standby); `trillian`; the workstation |
|
||||
|
||||
Site octet encodes the mesh subnet (`10.<site>.0.0/16`); see `generic.md` §11. New
|
||||
office services front on `hanzalova.internal`; new DC services on `oolon`.
|
||||
|
||||
## 2. Two access paths — and the mesh hairpin gotcha
|
||||
|
||||
A service can be reached two ways, and they are **not** interchangeable:
|
||||
|
||||
- **Public (from the WAN):** public DNS (Cloudflare, unproxied by default) → site WAN IP
|
||||
→ OPNsense forwards `:443` → site nginx → upstream. Cert: **Let's Encrypt**
|
||||
(`external-tls.md`).
|
||||
- **Internal (from the mesh):** split-horizon `.internal` DNS → the host/proxy directly
|
||||
over WireGuard → nginx. Cert: **internal CA** (`internal-tls.md`).
|
||||
|
||||
> **Gotcha — public names don't hairpin.** From *inside* the mesh, a public name still
|
||||
> resolves (via public DNS) to the site's **WAN** IP, so the packet hits the OPNsense
|
||||
> **LAN** interface — which only forwards `:443` inbound from the **WAN**, not from the
|
||||
> LAN. The connection dead-ends (or worse, gets OPNsense's own default cert). So a
|
||||
> service that mesh clients also need must be published under a **`*.internal` name with
|
||||
> its own internal-CA vhost**, in addition to its public vhost.
|
||||
|
||||
This is why dual-audience services get **two vhosts** on the same proxy — one public
|
||||
(LE), one internal (`lair` CA) — usually sharing one webroot and one upstream.
|
||||
|
||||
## 3. Per-vhost cert choice
|
||||
|
||||
| vhost audience | name | cert | doc |
|
||||
| --- | --- | --- | --- |
|
||||
| Public / WAN | `<svc>.<public-zone>` (e.g. `bench.helexa.ai`) | Let's Encrypt (certbot, Cloudflare DNS-01, ECDSA) | `external-tls.md` |
|
||||
| Mesh-only | `<svc>.internal` | internal CA (`step ca`, `lair` provisioner, `step@` renewal) | `internal-tls.md` |
|
||||
|
||||
Provisioner credentials for the internal CA (`~/.step/secrets/provisioner`, shipped to
|
||||
the host transiently and removed) are covered in `internal-tls.md` §4.
|
||||
|
||||
## 4. nginx conventions on the proxies
|
||||
|
||||
- **`sites-available/` + `sites-enabled/` symlink**, included via
|
||||
`/etc/nginx/conf.d/sites-enabled.conf` (`include /etc/nginx/sites-enabled/*.conf;`).
|
||||
One file per `server_name`; enable with a relative symlink
|
||||
(`ln -sf ../sites-available/<name>.conf /etc/nginx/sites-enabled/`).
|
||||
- **Static SPA** served from `/var/www/<name>` with SPA fallback
|
||||
(`try_files $uri $uri/ /index.html;`); **API** reverse-proxied to the internal
|
||||
`host:port`. Internal vhosts add `ssl_trusted_certificate <internal root>` and pin
|
||||
`ssl_protocols TLSv1.3`.
|
||||
- **SELinux (enforcing):** webroots must be labelled `httpd_sys_content_t` or nginx
|
||||
returns **403**. After creating/populating `/var/www/<name>`, run
|
||||
`restorecon -R /var/www/<name>`; rsynced files inherit the dir's type.
|
||||
- **Never reference a cert path before the cert exists** — `nginx -t` fails on a missing
|
||||
`ssl_certificate` and blocks the whole server from (re)starting. Issue the cert, then
|
||||
install the TLS vhost (gate on the cert's presence; serve an http-only bootstrap until
|
||||
then if needed).
|
||||
- Config + cert/renewal wiring is installed idempotently from each project's
|
||||
`infra-setup.sh` (`deployment-gitea-actions.md` §2); the recurring artifact rsync
|
||||
(e.g. built SPA `dist/`) rides in the deploy workflow.
|
||||
|
||||
## 5. Worked example: helexa-bench UI
|
||||
|
||||
The bench visualisation is reached both ways, fronted by `hanzalova.internal`:
|
||||
|
||||
| vhost | cert | DNS |
|
||||
| --- | --- | --- |
|
||||
| `bench.helexa.ai` (public) | Let's Encrypt | Cloudflare A → office WAN IP; OPNsense forwards WAN `:443` → `hanzalova` |
|
||||
| `bench.internal` (mesh) | internal `lair` CA, renewed by `step@bench.timer` | split-horizon `bench.internal → hanzalova` mesh IP |
|
||||
|
||||
Both vhosts share one webroot (`/var/www/bench.helexa.ai`, the built SPA) and proxy
|
||||
`/api` to the helexa-bench read API on `bob.hanzalova.internal:13132`. The internal vhost
|
||||
exists precisely because of §2: from a workstation on the mesh, `bench.helexa.ai`
|
||||
hairpins to the OPNsense LAN interface and fails, so mesh users hit `bench.internal`.
|
||||
Reference in New Issue
Block a user