Files

docs: document opn-cli for managing the OPNsense site routers

Add a "Managing the OPNsense routers" note to generic.md §11 (Network):
opn-cli talks to a site's OPNsense over its API, credentials in a per-site
~/.opn-cli/<site>.yml (api_key/secret + url + internal-CA ca:), --config is
required. The split-horizon .internal records and public-name host-overrides
live in each router's Unbound, so `unbound host list/create/update/delete` is
how you inspect and fix them:

  opn-cli --config ~/.opn-cli/hanzalova.yml   unbound host list
  opn-cli --config ~/.opn-cli/kosherinata.yml unbound host list

Cross-referenced from reverse-proxies.md §2 as the tool to reach for when a
mesh client resolves a name to the wrong host — exactly the rob.fyi case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP

2026-07-08 13:20:58 +03:00

5.2 KiB

Raw Blame History

Reverse proxies and edge ingress

Extends generic.md §11 (Network / Ingress). That section says "per-site nginx reverse proxy terminates all WAN inbound 443"; this doc names the proxies, maps what sits behind each, and pins down the two access paths and the per-vhost cert choice — plus the one gotcha that bites every time (a public name doesn't work from inside the mesh).

1. The proxies (one per site)

Each WireGuard site has a single nginx edge proxy. All WAN-inbound 443 for that site is port-forwarded by the site's OPNsense router to its proxy, which terminates TLS and fans out to internal upstreams.

Site	Edge proxy (nginx host)	Notable hosts behind it
kosherinata (DC)	`oolon.kosherinata.internal`	`magrathea` (Postgres primary), `nikola`, `gramathea`, …
hanzalova (office)	`hanzalova.internal`	GPU/inference: `beast`, `benjy`, `quadbrat`; `bob` (helexa-bench API + Agent Zero); `frankie` (Postgres streaming standby); `trillian`; the workstation

Site octet encodes the mesh subnet (10.<site>.0.0/16); see generic.md §11. New office services front on hanzalova.internal; new DC services on oolon.

2. Two access paths — and the mesh hairpin gotcha

A service can be reached two ways, and they are not interchangeable:

Public (from the WAN): public DNS (Cloudflare, unproxied by default) → site WAN IP → OPNsense forwards :443 → site nginx → upstream. Cert: Let's Encrypt (external-tls.md).
Internal (from the mesh): split-horizon .internal DNS → the host/proxy directly over WireGuard → nginx. Cert: internal CA (internal-tls.md).

Gotcha — public names don't hairpin. From inside the mesh, a public name still resolves (via public DNS) to the site's WAN IP, so the packet hits the OPNsense LAN interface — which only forwards :443 inbound from the WAN, not from the LAN. The connection dead-ends (or worse, gets OPNsense's own default cert). So a service that mesh clients also need must be published under a *.internal name with its own internal-CA vhost, in addition to its public vhost.

This is why dual-audience services get two vhosts on the same proxy — one public (LE), one internal (lair CA) — usually sharing one webroot and one upstream.

Debugging a wrong-host resolution. When a mesh client reaches the wrong site (wrong cert / wrong content), the culprit is usually a split-horizon record in the site router's Unbound — a missing *.internal override, or a stale host-override sending a public name to the wrong proxy. Inspect and fix it with opn-cli (generic.md §11 Network): opn-cli --config ~/.opn-cli/<site>.yml unbound host list.

3. Per-vhost cert choice

vhost audience	name	cert	doc
Public / WAN	`<svc>.<public-zone>` (e.g. `bench.helexa.ai`)	Let's Encrypt (certbot, Cloudflare DNS-01, ECDSA)	`external-tls.md`
Mesh-only	`<svc>.internal`	internal CA (`step ca`, `lair` provisioner, `step@` renewal)	`internal-tls.md`

Provisioner credentials for the internal CA (~/.step/secrets/provisioner, shipped to the host transiently and removed) are covered in internal-tls.md §4.

4. nginx conventions on the proxies

sites-available/ + sites-enabled/ symlink, included via /etc/nginx/conf.d/sites-enabled.conf (include /etc/nginx/sites-enabled/*.conf;). One file per server_name; enable with a relative symlink (ln -sf ../sites-available/<name>.conf /etc/nginx/sites-enabled/).
Static SPA served from /var/www/<name> with SPA fallback (try_files $uri $uri/ /index.html;); API reverse-proxied to the internal host:port. Internal vhosts add ssl_trusted_certificate <internal root> and pin ssl_protocols TLSv1.3.
SELinux (enforcing): webroots must be labelled httpd_sys_content_t or nginx returns 403. After creating/populating /var/www/<name>, run restorecon -R /var/www/<name>; rsynced files inherit the dir's type.
Never reference a cert path before the cert exists — nginx -t fails on a missing ssl_certificate and blocks the whole server from (re)starting. Issue the cert, then install the TLS vhost (gate on the cert's presence; serve an http-only bootstrap until then if needed).
Config + cert/renewal wiring is installed idempotently from each project's infra-setup.sh (deployment-gitea-actions.md §2); the recurring artifact rsync (e.g. built SPA dist/) rides in the deploy workflow.

5. Worked example: helexa-bench UI

The bench visualisation is reached both ways, fronted by hanzalova.internal:

vhost	cert	DNS
`bench.helexa.ai` (public)	Let's Encrypt	Cloudflare A → office WAN IP; OPNsense forwards WAN `:443` → `hanzalova`
`bench.internal` (mesh)	internal `lair` CA, renewed by `step@bench.timer`	split-horizon `bench.internal → hanzalova` mesh IP

Both vhosts share one webroot (/var/www/bench.helexa.ai, the built SPA) and proxy /api to the helexa-bench read API on bob.hanzalova.internal:13132. The internal vhost exists precisely because of §2: from a workstation on the mesh, bench.helexa.ai hairpins to the OPNsense LAN interface and fails, so mesh users hit bench.internal.

5.2 KiB Raw Blame History