cichlid.internal (and ~a dozen other internal vhosts on oolon) have been serving certs that expired days ago, even though fresh certs sit renewed on disk. Root cause is a CI drift bug in this repo, not in the consuming services.
deploy.yml and refresh.yml each rendered asset/nginx/site.conf.tmpl with their own inline Python substitution. Commit bb2f5b1 templated the listen line as {{WEB_LISTEN}} (moving the vhost behind oolon's stream SNI router) and taught the template + deploy.yml + deploy.sh about it — but not refresh.yml. So the nightly refresh (cron 17 4 * * *) rsynced a literal listen {{WEB_LISTEN}}; into /etc/nginx/conf.d/rob.tn.conf.
Because the rendered file is written into the liveconf.dbeforenginx -t runs, that one unrendered placeholder failed the config test for the whole edge and left the wreckage in place — freezing every vhost reload on oolon, including the step@*.timer cert renewals. Certs lapsed on the wire while their renewals piled up unused on disk.
A live fix to rob.tn.conf would be undone by the next 04:17 UTC refresh, so the fix has to be in the pipeline.
What
One shared renderer — script/render-site-conf.py, called by both workflows, so they can no longer drift on what they substitute.
Guard rails — the renderer fails the build if any {{PLACEHOLDER}} lacks an env value or survives substitution. A forgotten/misnamed variable is now a red run on the runner instead of a broken vhost on the edge (this exact incident would have failed loudly before shipping).
Close the immediate drift — add the missing WEB_LISTEN to refresh.yml's env.
Template — rename {{DOCROOT}} → {{WEB_ROOT}} so every placeholder maps to the env var of the same name.
Remove script/deploy.sh — the third, unused renderer of the same template (superseded by the Actions workflows) and a standing source of drift.
Docs — readme.md / CLAUDE.md updated to the Actions-only deploy path.
Validation
render-site-conf.py renders the correct vhost from the full env (listen 127.0.0.1:14443 ssl proxy_protocol, no surviving {{ }}).
With WEB_LISTEN unset (the old refresh.yml state) it exits non-zero and writes no file.
Both workflows still parse as YAML; no {{DOCROOT}} or deploy.sh references remain.
Merging this heals oolon
A merge to main triggers deploy.yml, which re-renders the correct rob.tn.conf; nginx -t then passes (rob.tn.conf is the only broken file) and the reload lands, so cichlid.internal and the other frozen vhosts immediately pick up their current on-disk certs. The next nightly refresh stays correct too.
Known follow-up (out of scope — needs a sudoers change + infra-setup.sh re-run on oolon)
The rendered vhost is still rsynced straight into the live conf.d and only then nginx -t'd, so a valid-but-wrong config (e.g. a directive nginx -t can't catch across http{}/stream{}) could still wedge nginx. A stage → validate → swap with rollback would close that, but the gitea_ci scoped sudoers only permits rsync … rob.tn.conf (no cp/mv/install for the vhost), so it needs a provisioning change and is left for a separate PR.
## Why
`cichlid.internal` (and ~a dozen other internal vhosts on oolon) have been serving certs that expired days ago, even though fresh certs sit renewed on disk. Root cause is a **CI drift bug in this repo**, not in the consuming services.
`deploy.yml` and `refresh.yml` each rendered `asset/nginx/site.conf.tmpl` with their **own inline Python substitution**. Commit `bb2f5b1` templated the listen line as `{{WEB_LISTEN}}` (moving the vhost behind oolon's stream SNI router) and taught the template + `deploy.yml` + `deploy.sh` about it — **but not `refresh.yml`**. So the nightly refresh (cron `17 4 * * *`) rsynced a literal `listen {{WEB_LISTEN}};` into `/etc/nginx/conf.d/rob.tn.conf`.
Because the rendered file is written into the **live** `conf.d` *before* `nginx -t` runs, that one unrendered placeholder failed the config test for the whole edge and left the wreckage in place — freezing **every** vhost reload on oolon, including the `step@*.timer` cert renewals. Certs lapsed on the wire while their renewals piled up unused on disk.
A live fix to `rob.tn.conf` would be undone by the next 04:17 UTC refresh, so the fix has to be in the pipeline.
## What
- **One shared renderer** — `script/render-site-conf.py`, called by both workflows, so they can no longer drift on what they substitute.
- **Guard rails** — the renderer fails the build if any `{{PLACEHOLDER}}` lacks an env value or survives substitution. A forgotten/misnamed variable is now a red run on the runner instead of a broken vhost on the edge (this exact incident would have failed loudly before shipping).
- **Close the immediate drift** — add the missing `WEB_LISTEN` to `refresh.yml`'s `env`.
- **Template** — rename `{{DOCROOT}}` → `{{WEB_ROOT}}` so every placeholder maps to the env var of the same name.
- **Remove `script/deploy.sh`** — the third, unused renderer of the same template (superseded by the Actions workflows) and a standing source of drift.
- **Docs** — `readme.md` / `CLAUDE.md` updated to the Actions-only deploy path.
## Validation
- `render-site-conf.py` renders the correct vhost from the full env (`listen 127.0.0.1:14443 ssl proxy_protocol`, no surviving `{{ }}`).
- With `WEB_LISTEN` unset (the old `refresh.yml` state) it exits non-zero and writes **no** file.
- Both workflows still parse as YAML; no `{{DOCROOT}}` or `deploy.sh` references remain.
## Merging this heals oolon
A merge to `main` triggers `deploy.yml`, which re-renders the correct `rob.tn.conf`; `nginx -t` then passes (rob.tn.conf is the only broken file) and the reload lands, so cichlid.internal and the other frozen vhosts immediately pick up their current on-disk certs. The next nightly refresh stays correct too.
## Known follow-up (out of scope — needs a sudoers change + `infra-setup.sh` re-run on oolon)
The rendered vhost is still rsynced straight into the live `conf.d` and only then `nginx -t`'d, so a *valid-but-wrong* config (e.g. a directive `nginx -t` can't catch across `http{}`/`stream{}`) could still wedge nginx. A stage → validate → swap with rollback would close that, but the `gitea_ci` scoped sudoers only permits `rsync … rob.tn.conf` (no `cp`/`mv`/`install` for the vhost), so it needs a provisioning change and is left for a separate PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The nightly refresh.yml and deploy.yml each substituted asset/nginx/
site.conf.tmpl with their own inline python. When bb2f5b1 templated the
listen line as {{WEB_LISTEN}} (moving the vhost behind oolon's stream SNI
router), it added the substitution to the template and deploy.yml but not
to refresh.yml. The daily refresh then rsynced a literal
`listen {{WEB_LISTEN}};` into /etc/nginx/conf.d/rob.tn.conf, `nginx -t`
failed for the whole edge, and — because the file is written into the live
conf.d before it is tested — every vhost's reload (including the step@
cert renewals) stayed frozen. Internal vhosts, cichlid.internal among
them, served certs that had expired days earlier while the renewed certs
sat unused on disk.
- Replace both inline renderers with script/render-site-conf.py, shared by
deploy.yml and refresh.yml so they cannot drift on what they substitute.
- Guard rails: the renderer fails if any {{PLACEHOLDER}} lacks an env value
or survives substitution, so a forgotten/misnamed variable is a red build
on the runner instead of a broken vhost on the edge.
- Add the missing WEB_LISTEN to refresh.yml's env (the immediate drift).
- Rename the template's {{DOCROOT}} to {{WEB_ROOT}} so every placeholder
maps to the env var of the same name.
- Remove script/deploy.sh: the third, unused renderer of the same template
(superseded by the Actions workflows) and a standing source of drift.
- Docs (readme, CLAUDE.md) updated to the Actions-only deploy path.
Known follow-up (needs a sudoers change + infra-setup re-run on oolon, so
out of scope here): the rendered vhost is still rsynced straight into the
live conf.d and only then `nginx -t`'d, so a valid-but-wrong config could
still wedge nginx. Stage-validate-swap with rollback would close that.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QsH1rcWQYtRVhvaftiKm22
grenade
merged commit 9905cef1d2 into main2026-07-26 12:54:48 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Why
cichlid.internal(and ~a dozen other internal vhosts on oolon) have been serving certs that expired days ago, even though fresh certs sit renewed on disk. Root cause is a CI drift bug in this repo, not in the consuming services.deploy.ymlandrefresh.ymleach renderedasset/nginx/site.conf.tmplwith their own inline Python substitution. Commitbb2f5b1templated the listen line as{{WEB_LISTEN}}(moving the vhost behind oolon's stream SNI router) and taught the template +deploy.yml+deploy.shabout it — but notrefresh.yml. So the nightly refresh (cron17 4 * * *) rsynced a literallisten {{WEB_LISTEN}};into/etc/nginx/conf.d/rob.tn.conf.Because the rendered file is written into the live
conf.dbeforenginx -truns, that one unrendered placeholder failed the config test for the whole edge and left the wreckage in place — freezing every vhost reload on oolon, including thestep@*.timercert renewals. Certs lapsed on the wire while their renewals piled up unused on disk.A live fix to
rob.tn.confwould be undone by the next 04:17 UTC refresh, so the fix has to be in the pipeline.What
script/render-site-conf.py, called by both workflows, so they can no longer drift on what they substitute.{{PLACEHOLDER}}lacks an env value or survives substitution. A forgotten/misnamed variable is now a red run on the runner instead of a broken vhost on the edge (this exact incident would have failed loudly before shipping).WEB_LISTENtorefresh.yml'senv.{{DOCROOT}}→{{WEB_ROOT}}so every placeholder maps to the env var of the same name.script/deploy.sh— the third, unused renderer of the same template (superseded by the Actions workflows) and a standing source of drift.readme.md/CLAUDE.mdupdated to the Actions-only deploy path.Validation
render-site-conf.pyrenders the correct vhost from the full env (listen 127.0.0.1:14443 ssl proxy_protocol, no surviving{{ }}).WEB_LISTENunset (the oldrefresh.ymlstate) it exits non-zero and writes no file.{{DOCROOT}}ordeploy.shreferences remain.Merging this heals oolon
A merge to
maintriggersdeploy.yml, which re-renders the correctrob.tn.conf;nginx -tthen passes (rob.tn.conf is the only broken file) and the reload lands, so cichlid.internal and the other frozen vhosts immediately pick up their current on-disk certs. The next nightly refresh stays correct too.Known follow-up (out of scope — needs a sudoers change +
infra-setup.shre-run on oolon)The rendered vhost is still rsynced straight into the live
conf.dand only thennginx -t'd, so a valid-but-wrong config (e.g. a directivenginx -tcan't catch acrosshttp{}/stream{}) could still wedge nginx. A stage → validate → swap with rollback would close that, but thegitea_ciscoped sudoers only permitsrsync … rob.tn.conf(nocp/mv/installfor the vhost), so it needs a provisioning change and is left for a separate PR.🤖 Generated with Claude Code