fix(ci): restart the api only after the worker has finished migrating

The worker owns migrations — it connects as moments_rw, while the api is
SELECT-only and would fail with `permission denied for schema public` if
it tried. Nothing ordered the two deploy jobs, though, and they run
against different hosts (nikola, frootmig), so the systemd ordering the
comment in moments-api/src/main.rs appeals to cannot reach between them.

On the run that shipped the `events.repo` migration, the worker happened
to restart 5 seconds before the api. Reversed, the api would have come up
querying a column that did not exist yet and errored on every request
touching it — and `/v1/healthz` returns a static "ok" without touching
the database, so the deploy's own health probe would have passed it as
fine.

deploy-api now needs deploy-worker. That ordering is only worth anything
if the worker job stays open until migrations are actually done, and it
didn't: the unit is Type=simple, so `systemctl restart` returns once the
process is exec'd and `is-active` is true immediately, neither of which
says anything about migrations. So the job now waits for the worker to
log "worker started", which it does only after awaiting store.migrate().
The search is scoped to the unit's current InvocationID so a previous
start's line cannot satisfy the wait, and it gives up loudly after 120s
or as soon as the process exits.

The wait deliberately avoids `journalctl | grep -q`: a consumer that
exits on first match closes the pipe, journalctl dies of SIGPIPE, and
under `pipefail` the match reads as a failure — which is exactly what the
first draft did, and it would have failed every deploy on a worker that
started perfectly. journalctl --grep does the filtering instead and the
result is tested for emptiness, with no pipe in the pipeline.

Verified by extracting the gate out of the workflow and running it
against fake systemctl/journalctl: started -> exit 0 on the first check;
process gone -> exit 1 immediately with "worker exited before reporting
startup"; no InvocationID -> exit 1 immediately; never logs but stays
active -> loops and fails on timeout. Job graph re-checked acyclic with
every `needs:` resolving to a real job (binaries -> worker -> api -> web).

Closes #8

.gitea/workflows/deploy.yml

@@ -106,11 +106,11 @@ jobs:
     # raced the api/worker deploys and the crawler snapshot silently disagreed
     # with the live api for a whole day, until the nightly refresh: it won the
     # race once (a visibility fix reached both together) and lost it once (an
-    # api field the snapshot then lacked). deploy-worker is in the list too,
-    # not just deploy-api — the worker owns migrations, so a schema change
-    # isn't applied until it has restarted.
+    # api field the snapshot then lacked). deploy-worker is named as well as
+    # deploy-api even though deploy-api already implies it — stating it keeps
+    # this job correct on its own terms if that chain is ever shortened.
     #
-    # The cost is a serial deploy: binaries -> api/worker -> web. Worth it;
+    # The cost is a serial deploy: binaries -> worker -> api -> web. Worth it;
     # the alternative is a snapshot whose correctness depends on which runner
     # finished first.
     needs: [deploy-api, deploy-worker]
@@ -133,7 +133,13 @@ jobs:
 
   deploy-api:
     name: Deploy moments-api to nikola
-    needs: build-binaries
+    # The worker owns migrations (it connects as moments_rw; the api is
+    # SELECT-only and cannot run them), so the schema has to be current before
+    # a new api starts answering with it. These are separate hosts — nikola and
+    # frootmig — so the systemd ordering the comment in moments-api/src/main.rs
+    # refers to cannot reach across them; the dependency has to live here.
+    # deploy-worker holds itself open until migrations have actually completed.
+    needs: [build-binaries, deploy-worker]
     runs-on: fedora-44
     steps:
       - uses: actions/checkout@v4
@@ -328,6 +334,45 @@ jobs:
             sudo /usr/bin/systemctl restart moments-worker.service
             sudo /usr/bin/systemctl is-active --quiet moments-worker.service'
 
+      # The unit is Type=simple, so `restart` returns once the process is
+      # exec'd and `is-active` is true immediately — neither says anything
+      # about migrations. deploy-api waits on this job precisely so that the
+      # schema is current before the new api answers a request, so the job
+      # has to stay open until migrations are actually done.
+      #
+      # moments-worker awaits store.migrate() before it logs "worker started",
+      # so that line is the signal. Scoping the search to the unit's current
+      # InvocationID means a previous start's line can't satisfy the wait.
+      - name: Wait for migrations to finish
+        run: |
+          ssh gitea_ci@"${WORKER_HOST}" '
+            set -euo pipefail
+            invocation="$(systemctl show --property=InvocationID --value moments-worker.service)"
+            if [ -z "${invocation}" ]; then
+              echo "no InvocationID for moments-worker.service" >&2
+              exit 1
+            fi
+            for attempt in $(seq 1 60); do
+              # Deliberately no `| grep -q`: a consumer that exits on first
+              # match closes the pipe, journalctl dies of SIGPIPE, and under
+              # `pipefail` the match reads as a failure. Let journalctl filter
+              # itself and test whether anything came back. `|| true` because
+              # --grep exits 1 when nothing matched yet.
+              started="$(journalctl _SYSTEMD_INVOCATION_ID="${invocation}" \
+                           --no-pager --grep "worker started" || true)"
+              if [ -n "${started}" ]; then
+                echo "migrations complete after ${attempt} check(s)"
+                exit 0
+              fi
+              if ! systemctl is-active --quiet moments-worker.service; then
+                echo "worker exited before reporting startup" >&2
+                exit 1
+              fi
+              sleep 2
+            done
+            echo "timed out after 120s waiting for worker startup" >&2
+            exit 1'
+
       - name: Capture startup journal
         if: always()
         run: |

CLAUDE.md

@@ -119,13 +119,21 @@ CI-driven via **Gitea Actions** (`.gitea/workflows/`), the source of infra truth
   prerendered web bundle, then deploy each component over SSH as the `gitea_ci`
   user with scoped sudo (`asset/sudoers.d/`). Services run under systemd with
   hardened units; the api/worker reach postgres over mTLS using the host cert.
-  The job graph is deliberately serial — binaries → api/worker → web — because
-  the prerender fetches `VITE_API_BASE` at build time and so bakes whatever the
-  api is serving when `build-web` runs. `build-web` therefore `needs:` both
-  deploy jobs (the worker owns migrations, so a schema change isn't live until
-  it restarts). When it ran in parallel instead, whether the crawler snapshot
-  matched the api came down to which runner finished first, and a lost race
-  meant a stale snapshot until the nightly refresh.
+  The job graph is deliberately serial — binaries → worker → api → web — and
+  each link is load-bearing:
+  - `deploy-api` needs `deploy-worker` because the worker owns migrations (it
+    connects as `moments_rw`; the api is SELECT-only and cannot run them). The
+    two live on different hosts, so the systemd ordering the comment in
+    `moments-api/src/main.rs` describes cannot reach across them.
+  - `deploy-worker` doesn't finish at `systemctl restart` — the unit is
+    `Type=simple`, so that returns before migrations do. It polls the journal,
+    scoped to the unit's current `InvocationID`, for the `worker started` line
+    the worker only logs after `store.migrate()` returns.
+  - `build-web` needs both deploy jobs because the prerender fetches
+    `VITE_API_BASE` at build time and bakes whatever the api is serving when it
+    runs. In parallel, whether the crawler snapshot matched the api came down to
+    which runner finished first; a lost race meant a stale published snapshot
+    until the nightly refresh, with nothing detecting it.
 - `refresh.yml` — daily `schedule:` (+ manual): rebuilds and redeploys only the
   web tier, re-baking the prerendered crawler snapshot from the current gist (CV)
   and activity API without bouncing the api/worker.