Any route asked for without a trailing slash sent the visitor to a port
nothing answers on from outside:
$ curl -sSI https://rob.tn/activity
HTTP/2 301
location: https://rob.tn:14443/activity/
`try_files $uri $uri/` 301s a slash-less directory URL to add the slash,
and nginx builds that Location as an absolute URL from its own
$server_port. This vhost listens on WEB_LISTEN — 127.0.0.1:14443, behind
the edge's stream SNI router — so the redirect advertised 14443 instead of
the 443 the client used. The browser then sat on a TCP connect that never
completes and gave up only after its own timeout, 60s+, before showing an
error. It reads as the site hanging.
`absolute_redirect off` makes the Location relative, so the client keeps
whatever scheme, host and port it actually used. `port_in_redirect off`
would drop the port too, but this also stops nginx asserting a scheme and
host it cannot know from behind the router.
Every /activity, /blog, /cv and /project/... request without the trailing
slash was affected — external links, bookmarks, typed URLs, crawlers. In-app
navigation never round-trips to the server, and `/` needs no directory
redirect, which is why the homepage always loaded fine and this stayed
hidden. It is not a regression from any recent change; it follows from the
vhost listening on a shifted port.
Verified by reproducing the port leak in a container listening on 8081
published as 18081 — `Location: http://127.0.0.1:8081/activity/` before the
directive, `Location: /activity/` after — then rendering the real template
through script/render-site-conf.py (9 placeholders in, none surviving) and
passing `nginx -t` on the result.
Closes #9
The nightly refresh.yml and deploy.yml each substituted asset/nginx/
site.conf.tmpl with their own inline python. When bb2f5b1 templated the
listen line as {{WEB_LISTEN}} (moving the vhost behind oolon's stream SNI
router), it added the substitution to the template and deploy.yml but not
to refresh.yml. The daily refresh then rsynced a literal
`listen {{WEB_LISTEN}};` into /etc/nginx/conf.d/rob.tn.conf, `nginx -t`
failed for the whole edge, and — because the file is written into the live
conf.d before it is tested — every vhost's reload (including the step@
cert renewals) stayed frozen. Internal vhosts, cichlid.internal among
them, served certs that had expired days earlier while the renewed certs
sat unused on disk.
- Replace both inline renderers with script/render-site-conf.py, shared by
deploy.yml and refresh.yml so they cannot drift on what they substitute.
- Guard rails: the renderer fails if any {{PLACEHOLDER}} lacks an env value
or survives substitution, so a forgotten/misnamed variable is a red build
on the runner instead of a broken vhost on the edge.
- Add the missing WEB_LISTEN to refresh.yml's env (the immediate drift).
- Rename the template's {{DOCROOT}} to {{WEB_ROOT}} so every placeholder
maps to the env var of the same name.
- Remove script/deploy.sh: the third, unused renderer of the same template
(superseded by the Actions workflows) and a standing source of drift.
- Docs (readme, CLAUDE.md) updated to the Actions-only deploy path.
Known follow-up (needs a sudoers change + infra-setup re-run on oolon, so
out of scope here): the rendered vhost is still rsynced straight into the
live conf.d and only then `nginx -t`'d, so a valid-but-wrong config could
still wedge nginx. Stage-validate-swap with rollback would close that.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QsH1rcWQYtRVhvaftiKm22
oolon's TCP 443 belongs to the stream SNI router, which ssl_prereads the
handshake and forwards to the local https tier on 127.0.0.1:14443 with
PROXY protocol. site.conf.tmpl predates that and still bound 443 itself,
so every deploy and every daily refresh rsynced a vhost that collides
with the router.
Nothing in the pipeline caught it. `nginx -t` only detects duplicate
listeners within a context, not across http{} and stream{}, and
`systemctl reload` merely sends SIGHUP, so it exits 0 while nginx logs
"bind() to 0.0.0.0:443 failed (98: Address already in use) ... still
could not bind()", aborts the reconfiguration and keeps its old cycle.
The deploy went green while oolon's running config was frozen. It stayed
frozen for a day, stranding every cert the step@ timers renewed on disk
until eleven internal vhosts were serving expired certs. A cold start
would have failed outright, taking the whole public edge down.
Template the listen line from WEB_LISTEN (manifest web.config.listen for
script/deploy.sh, which renders the same template), and assert that the
reload landed by requiring a fresh worker generation, dumping the nginx
error log when it did not.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0182wzZE8DguMPWhxD21gfP2
The prerendered pages are large (the dashboard bakes the full all-time
activity dataset, ~900 KB), so compress text responses on the wire. text/html
is always compressed when gzip is on; the listed gzip_types also cover the JS
bundle, CSS, SVG icons, and (via gzip_proxied any) the JSON from the /api/
upstream. The home page drops to ~90 KB on the wire. woff2 is pre-compressed
and intentionally omitted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7zF7Kf4JqDwa6M8Qgge9M
The static asset regex matched .png in /api/v1/og/contributions.png
before the /api/ proxy block, returning 404. Add negative lookahead
to skip /api/ paths.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
deploy.sh:
- never rsync into /; stage to /tmp on the remote and install at final
paths via sudo bash heredoc, closing the parent-dir attribute leak
that broke three hosts in the earlier rsync incident
- shell-quote heredoc args via ${var@Q}
- drop -A -X on the remaining (web) rsyncs
- generic worker.secrets loop reads (env-var → pass path) from manifest;
GITEA_TOKEN now flows through automatically
- in-memory bash substitution for templates (secrets never on argv)
- simplify semanage port labelling: --add 2>/dev/null || --modify (the
old grep pre-check matched only the first listed port)
- restorecon back to short flags (Fedora policycoreutils has no long
forms; --recursive errored at deploy time)
- quieter health probe loop: curl diagnostics only on final failure
manifest as source of truth:
- api.config.bind drives BIND_ADDR, firewalld port, semanage label,
health-probe URL
- web.config.{server_name,root,api_upstream} drives nginx render,
rsync targets, restorecon scope
- nginx config renamed to site.conf.tmpl; firewalld svc to
moments-api.xml.tmpl; both rendered at deploy time
- topology flip: api → nikola, worker → frootmig (anjie freed)
new scripts:
- script/teardown.sh: idempotent component teardown, never rsyncs,
shared-state cleanup gated on absence of remaining env files,
--remove-docroot guard against shallow / system paths
- script/db-perms.sh: rewritten — fixes grep/append role mismatch that
appended duplicates on re-run, adds postgres reload, hits primary +
standby in a single invocation
readme: genericized; deployment topology no longer carries real host
or site names.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
nikola and frootmig are flagging power events and drive warnings on
the iLO interface and need drive replacement. Move both moments
components onto anjie.kosherinata.internal until those hosts are
back in service. Update the nginx upstream and the readme topology
table to match; the postgres pg_ident.conf on magrathea now needs
to map anjie's cert CN to both moments_ro and moments_rw (two lines
for the same cert_cn).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These ship in a public repo; topology narration in nginx, systemd,
firewalld, and env templates is gratuitous. Keep the config terse —
directives speak for themselves.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The per-site nginx ingress for rob.tn lives on oolon (the host the
external router forwards 443 traffic to), not on nikola. Adjust the
topology so:
- web (static ui + nginx) → oolon.hanzalova.internal
- api binds 0.0.0.0:42424 on nikola.kosherinata.internal so oolon
can reverse-proxy across the WG mesh
- new firewalld service moments-api opens 42424 in the default zone
on nikola
- oolon labels port 42424 http_port_t so httpd_t may name_connect
outbound to it (httpd_can_network_connect was already set)
- nginx ssl_certificate switched to oolon's host cert; upstream
rewritten to nikola.kosherinata.internal:42424
Plaintext between oolon and nikola for now — the WG mesh provides
the encryption layer and the data is already public. Documented
the deferral so a future move to per-hop mTLS is obvious.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires up the prod deployment per architecture-doc conventions:
- api → nikola.kosherinata.internal, loopback bind 127.0.0.1:42424
(less-common port, registered with SELinux as http_port_t).
- worker → frootmig.kosherinata.internal, no listening port.
- web (static ui/dist + nginx server_name rob.tn) → nikola, with
/api/* reverse-proxied to the loopback API.
- db → existing magrathea cluster via mTLS, hostname-baked DATABASE_URL
rendered into /etc/moments/{api,worker}.env at deploy time.
Cert rotation: step-ca renews host certs every 24h; .path units watch
/etc/pki/tls/misc/<host>.pem and trigger systemctl restart of the
relevant service. Both binaries hold cert state in rustls and read
once at startup, so restart is the right reload semantics.
deploy.sh contract matches the architecture doc: positional env arg,
component list (or `all` / `default`), --dry-run support. Renders
config templates from `pass`, rsyncs over ssh+sudo, runs sysusers /
restorecon / semanage / systemctl / nginx -t idempotently.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
