rob thijssen 7e186f76ae fix(ci): restart the api only after the worker has finished migrating ...

The worker owns migrations — it connects as moments_rw, while the api is
SELECT-only and would fail with `permission denied for schema public` if
it tried. Nothing ordered the two deploy jobs, though, and they run
against different hosts (nikola, frootmig), so the systemd ordering the
comment in moments-api/src/main.rs appeals to cannot reach between them.

On the run that shipped the `events.repo` migration, the worker happened
to restart 5 seconds before the api. Reversed, the api would have come up
querying a column that did not exist yet and errored on every request
touching it — and `/v1/healthz` returns a static "ok" without touching
the database, so the deploy's own health probe would have passed it as
fine.

deploy-api now needs deploy-worker. That ordering is only worth anything
if the worker job stays open until migrations are actually done, and it
didn't: the unit is Type=simple, so `systemctl restart` returns once the
process is exec'd and `is-active` is true immediately, neither of which
says anything about migrations. So the job now waits for the worker to
log "worker started", which it does only after awaiting store.migrate().
The search is scoped to the unit's current InvocationID so a previous
start's line cannot satisfy the wait, and it gives up loudly after 120s
or as soon as the process exits.

The wait deliberately avoids `journalctl | grep -q`: a consumer that
exits on first match closes the pipe, journalctl dies of SIGPIPE, and
under `pipefail` the match reads as a failure — which is exactly what the
first draft did, and it would have failed every deploy on a worker that
started perfectly. journalctl --grep does the filtering instead and the
result is tested for emptiness, with no pipe in the pipeline.

Verified by extracting the gate out of the workflow and running it
against fake systemctl/journalctl: started -> exit 0 on the first check;
process gone -> exit 1 immediately with "worker exited before reporting
startup"; no InvocationID -> exit 1 immediately; never logs but stays
active -> loops and fails on timeout. Job graph re-checked acyclic with
every `needs:` resolving to a real job (binaries -> worker -> api -> web).

Closes #8

2026-08-17 13:04:19 +03:00

21 KiB

Raw Blame History

View Raw