architecture/reverse-proxies.md

# Reverse proxies and edge ingress

Extends `generic.md` §11 (Network / Ingress). That section says "per-site nginx reverse
proxy terminates all WAN inbound 443"; this doc names the proxies, maps what sits behind
each, and pins down the two access paths and the per-vhost cert choice — plus the one
gotcha that bites every time (a public name doesn't work from *inside* the mesh).

---

## 1. The proxies (one per site)

Each WireGuard site has a single nginx edge proxy. All WAN-inbound 443 for that site is
port-forwarded by the site's OPNsense router to its proxy, which terminates TLS and
fans out to internal upstreams.

| Site | Edge proxy (nginx host) | Notable hosts behind it |
| --- | --- | --- |
| **kosherinata** (DC) | `oolon.kosherinata.internal` | `magrathea` (Postgres primary), `nikola`, `gramathea`, … |
| **hanzalova** (office) | `hanzalova.internal` | GPU/inference: `beast`, `benjy`, `quadbrat`; `bob` (helexa-bench API + Agent Zero); `frankie` (Postgres streaming standby); `trillian`; the workstation |

Site octet encodes the mesh subnet (`10.<site>.0.0/16`); see `generic.md` §11. New
office services front on `hanzalova.internal`; new DC services on `oolon`.

## 2. Two access paths — and the mesh hairpin gotcha

A service can be reached two ways, and they are **not** interchangeable:

- **Public (from the WAN):** public DNS (Cloudflare, unproxied by default) → site WAN IP
  → OPNsense forwards `:443` → site nginx → upstream. Cert: **Let's Encrypt**
  (`external-tls.md`).
- **Internal (from the mesh):** split-horizon `.internal` DNS → the host/proxy directly
  over WireGuard → nginx. Cert: **internal CA** (`internal-tls.md`).

> **Gotcha — public names don't hairpin.** From *inside* the mesh, a public name still
> resolves (via public DNS) to the site's **WAN** IP, so the packet hits the OPNsense
> **LAN** interface — which only forwards `:443` inbound from the **WAN**, not from the
> LAN. The connection dead-ends (or worse, gets OPNsense's own default cert). So a
> service that mesh clients also need must be published under a **`*.internal` name with
> its own internal-CA vhost**, in addition to its public vhost.

This is why dual-audience services get **two vhosts** on the same proxy — one public
(LE), one internal (`lair` CA) — usually sharing one webroot and one upstream.

> **Debugging a wrong-host resolution.** When a mesh client reaches the wrong site (wrong
> cert / wrong content), the culprit is usually a split-horizon record in the site
> router's Unbound — a missing `*.internal` override, or a stale host-override sending a
> public name to the wrong proxy. Inspect and fix it with `opn-cli` (`generic.md` §11
> Network): `opn-cli --config ~/.opn-cli/<site>.yml unbound host list`.

## 3. Per-vhost cert choice

| vhost audience | name | cert | doc |
| --- | --- | --- | --- |
| Public / WAN | `<svc>.<public-zone>` (e.g. `bench.helexa.ai`) | Let's Encrypt (certbot, Cloudflare DNS-01, ECDSA) | `external-tls.md` |
| Mesh-only | `<svc>.internal` | internal CA (`step ca`, `lair` provisioner, `step@` renewal) | `internal-tls.md` |

Provisioner credentials for the internal CA (`~/.step/secrets/provisioner`, shipped to
the host transiently and removed) are covered in `internal-tls.md` §4.

## 4. nginx conventions on the proxies

- **`sites-available/` + `sites-enabled/` symlink**, included via
  `/etc/nginx/conf.d/sites-enabled.conf` (`include /etc/nginx/sites-enabled/*.conf;`).
  One file per `server_name`; enable with a relative symlink
  (`ln -sf ../sites-available/<name>.conf /etc/nginx/sites-enabled/`).
- **Static SPA** served from `/var/www/<name>` with SPA fallback
  (`try_files $uri $uri/ /index.html;`); **API** reverse-proxied to the internal
  `host:port`. Internal vhosts add `ssl_trusted_certificate <internal root>` and pin
  `ssl_protocols TLSv1.3`.
- **SELinux (enforcing):** webroots must be labelled `httpd_sys_content_t` or nginx
  returns **403**. After creating/populating `/var/www/<name>`, run
  `restorecon -R /var/www/<name>`; rsynced files inherit the dir's type.
- **Never reference a cert path before the cert exists** — `nginx -t` fails on a missing
  `ssl_certificate` and blocks the whole server from (re)starting. Issue the cert, then
  install the TLS vhost (gate on the cert's presence; serve an http-only bootstrap until
  then if needed).
- Config + cert/renewal wiring is installed idempotently from each project's
  `infra-setup.sh` (`deployment-gitea-actions.md` §2); the recurring artifact rsync
  (e.g. built SPA `dist/`) rides in the deploy workflow.

## 5. Worked example: helexa-bench UI

The bench visualisation is reached both ways, fronted by `hanzalova.internal`:

| vhost | cert | DNS |
| --- | --- | --- |
| `bench.helexa.ai` (public) | Let's Encrypt | Cloudflare A → office WAN IP; OPNsense forwards WAN `:443` → `hanzalova` |
| `bench.internal` (mesh) | internal `lair` CA, renewed by `step@bench.timer` | split-horizon `bench.internal → hanzalova` mesh IP |

Both vhosts share one webroot (`/var/www/bench.helexa.ai`, the built SPA) and proxy
`/api` to the helexa-bench read API on `bob.hanzalova.internal:13132`. The internal vhost
exists precisely because of §2: from a workstation on the mesh, `bench.helexa.ai`
hairpins to the OPNsense LAN interface and fails, so mesh users hit `bench.internal`.