diff --git a/.gitea/workflows/deploy.yml b/.gitea/workflows/deploy.yml index 366813d..83e1eb2 100644 --- a/.gitea/workflows/deploy.yml +++ b/.gitea/workflows/deploy.yml @@ -106,11 +106,11 @@ jobs: # raced the api/worker deploys and the crawler snapshot silently disagreed # with the live api for a whole day, until the nightly refresh: it won the # race once (a visibility fix reached both together) and lost it once (an - # api field the snapshot then lacked). deploy-worker is in the list too, - # not just deploy-api — the worker owns migrations, so a schema change - # isn't applied until it has restarted. + # api field the snapshot then lacked). deploy-worker is named as well as + # deploy-api even though deploy-api already implies it — stating it keeps + # this job correct on its own terms if that chain is ever shortened. # - # The cost is a serial deploy: binaries -> api/worker -> web. Worth it; + # The cost is a serial deploy: binaries -> worker -> api -> web. Worth it; # the alternative is a snapshot whose correctness depends on which runner # finished first. needs: [deploy-api, deploy-worker] @@ -133,7 +133,13 @@ jobs: deploy-api: name: Deploy moments-api to nikola - needs: build-binaries + # The worker owns migrations (it connects as moments_rw; the api is + # SELECT-only and cannot run them), so the schema has to be current before + # a new api starts answering with it. These are separate hosts — nikola and + # frootmig — so the systemd ordering the comment in moments-api/src/main.rs + # refers to cannot reach across them; the dependency has to live here. + # deploy-worker holds itself open until migrations have actually completed. + needs: [build-binaries, deploy-worker] runs-on: fedora-44 steps: - uses: actions/checkout@v4 @@ -328,6 +334,45 @@ jobs: sudo /usr/bin/systemctl restart moments-worker.service sudo /usr/bin/systemctl is-active --quiet moments-worker.service' + # The unit is Type=simple, so `restart` returns once the process is + # exec'd and `is-active` is true immediately — neither says anything + # about migrations. deploy-api waits on this job precisely so that the + # schema is current before the new api answers a request, so the job + # has to stay open until migrations are actually done. + # + # moments-worker awaits store.migrate() before it logs "worker started", + # so that line is the signal. Scoping the search to the unit's current + # InvocationID means a previous start's line can't satisfy the wait. + - name: Wait for migrations to finish + run: | + ssh gitea_ci@"${WORKER_HOST}" ' + set -euo pipefail + invocation="$(systemctl show --property=InvocationID --value moments-worker.service)" + if [ -z "${invocation}" ]; then + echo "no InvocationID for moments-worker.service" >&2 + exit 1 + fi + for attempt in $(seq 1 60); do + # Deliberately no `| grep -q`: a consumer that exits on first + # match closes the pipe, journalctl dies of SIGPIPE, and under + # `pipefail` the match reads as a failure. Let journalctl filter + # itself and test whether anything came back. `|| true` because + # --grep exits 1 when nothing matched yet. + started="$(journalctl _SYSTEMD_INVOCATION_ID="${invocation}" \ + --no-pager --grep "worker started" || true)" + if [ -n "${started}" ]; then + echo "migrations complete after ${attempt} check(s)" + exit 0 + fi + if ! systemctl is-active --quiet moments-worker.service; then + echo "worker exited before reporting startup" >&2 + exit 1 + fi + sleep 2 + done + echo "timed out after 120s waiting for worker startup" >&2 + exit 1' + - name: Capture startup journal if: always() run: | diff --git a/CLAUDE.md b/CLAUDE.md index b81f296..9fb4f5b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -119,13 +119,21 @@ CI-driven via **Gitea Actions** (`.gitea/workflows/`), the source of infra truth prerendered web bundle, then deploy each component over SSH as the `gitea_ci` user with scoped sudo (`asset/sudoers.d/`). Services run under systemd with hardened units; the api/worker reach postgres over mTLS using the host cert. - The job graph is deliberately serial — binaries → api/worker → web — because - the prerender fetches `VITE_API_BASE` at build time and so bakes whatever the - api is serving when `build-web` runs. `build-web` therefore `needs:` both - deploy jobs (the worker owns migrations, so a schema change isn't live until - it restarts). When it ran in parallel instead, whether the crawler snapshot - matched the api came down to which runner finished first, and a lost race - meant a stale snapshot until the nightly refresh. + The job graph is deliberately serial — binaries → worker → api → web — and + each link is load-bearing: + - `deploy-api` needs `deploy-worker` because the worker owns migrations (it + connects as `moments_rw`; the api is SELECT-only and cannot run them). The + two live on different hosts, so the systemd ordering the comment in + `moments-api/src/main.rs` describes cannot reach across them. + - `deploy-worker` doesn't finish at `systemctl restart` — the unit is + `Type=simple`, so that returns before migrations do. It polls the journal, + scoped to the unit's current `InvocationID`, for the `worker started` line + the worker only logs after `store.migrate()` returns. + - `build-web` needs both deploy jobs because the prerender fetches + `VITE_API_BASE` at build time and bakes whatever the api is serving when it + runs. In parallel, whether the crawler snapshot matched the api came down to + which runner finished first; a lost race meant a stale published snapshot + until the nightly refresh, with nothing detecting it. - `refresh.yml` — daily `schedule:` (+ manual): rebuilds and redeploys only the web tier, re-baking the prerendered crawler snapshot from the current gist (CV) and activity API without bouncing the api/worker.