build-web races the api/worker deploys, so the crawler snapshot may not match the api #8

New Issue

grenade · 2026-08-17T09:58:03Z

grenade commented

2026-08-17 09:58:03 +00:00

build-web in deploy.yml has no needs:, so it runs in parallel with build-binaries, deploy-api and deploy-worker. The prerender fetches VITE_API_BASE at build time, which means the job bakes whatever the api happens to be serving at the moment it runs — the old binary or the new one, depending on which runner finishes first.

Observed both outcomes on consecutive runs of the same day:

run 68 — build-web started 16:47:26, deploy-api finished 16:47:51. The prerender's fetches landed after the new api was live, so a repo-visibility fix reached the api and the snapshot together. Won by ~25 seconds.
run 69 — lost the race. The api served the new activity/summary private-aggregate rows while the baked /activity/ snapshot had none: the 15 August card read 47 changes in 3 repositories where the api said 54 with 7 private. Browsers hydrate and refetch so a visitor saw the right thing, but curl — crawlers, AI screeners — got the stale page until refresh.yml was dispatched manually.

Nothing in the pipeline detects this. The snapshot is simply wrong for up to a day, and silently.

Fix

needs: [deploy-api, deploy-worker] on build-web, making the graph serial: binaries → api/worker → web. deploy-worker belongs in the list as well as deploy-api, because the worker owns migrations, so a schema change is not live until it has restarted.

Cost is wall-clock: the web build no longer overlaps the binary build.

Nothing orders deploy-api against deploy-worker either, and they are on different hosts (nikola / frootmig) so the systemd dependency the comment in moments-api/src/main.rs refers to cannot apply across them. /v1/healthz returns a static "ok" without touching the database, so the deploy's health probe passes regardless of whether the schema the new api expects has been migrated yet. On the migration in run 68 the worker happened to restart 5 seconds before the api; the reverse order would have left the api erroring on every query referencing the new column until the worker caught up.

Two candidate fixes, both worth considering separately:

deploy-api needs: deploy-worker, so migrations always precede the api restart
make healthz schema-aware (cheap query against the columns the binary needs) so the probe fails loudly instead of passing into a broken state

`build-web` in `deploy.yml` has no `needs:`, so it runs in parallel with `build-binaries`, `deploy-api` and `deploy-worker`. The prerender fetches `VITE_API_BASE` at build time, which means the job bakes whatever the api happens to be serving at the moment it runs — the old binary or the new one, depending on which runner finishes first. Observed both outcomes on consecutive runs of the same day: - **run 68** — `build-web` started 16:47:26, `deploy-api` finished 16:47:51. The prerender's fetches landed after the new api was live, so a repo-visibility fix reached the api and the snapshot together. Won by ~25 seconds. - **run 69** — lost the race. The api served the new `activity/summary` private-aggregate rows while the baked `/activity/` snapshot had none: the 15 August card read `47 changes in 3 repositories` where the api said 54 with 7 private. Browsers hydrate and refetch so a visitor saw the right thing, but `curl` — crawlers, AI screeners — got the stale page until `refresh.yml` was dispatched manually. Nothing in the pipeline detects this. The snapshot is simply wrong for up to a day, and silently. ### Fix `needs: [deploy-api, deploy-worker]` on `build-web`, making the graph serial: binaries → api/worker → web. `deploy-worker` belongs in the list as well as `deploy-api`, because the worker owns migrations, so a schema change is not live until it has restarted. Cost is wall-clock: the web build no longer overlaps the binary build. ### Related, not fixed here Nothing orders `deploy-api` against `deploy-worker` either, and they are on different hosts (nikola / frootmig) so the systemd dependency the comment in `moments-api/src/main.rs` refers to cannot apply across them. `/v1/healthz` returns a static `"ok"` without touching the database, so the deploy's health probe passes regardless of whether the schema the new api expects has been migrated yet. On the migration in run 68 the worker happened to restart 5 seconds before the api; the reverse order would have left the api erroring on every query referencing the new column until the worker caught up. Two candidate fixes, both worth considering separately: - `deploy-api` `needs: deploy-worker`, so migrations always precede the api restart - make `healthz` schema-aware (cheap query against the columns the binary needs) so the probe fails loudly instead of passing into a broken state

grenade referenced this issue from a commit

2026-08-17 09:59:06 +00:00

fix(ci): prerender the web bundle after the api/worker deploy, not beside it

grenade closed this issue

2026-08-17 09:59:06 +00:00

grenade referenced this issue from a commit

2026-08-17 10:04:27 +00:00

fix(ci): restart the api only after the worker has finished migrating

grenade referenced this issue from a commit

2026-08-17 10:14:08 +00:00

feat(api): make healthz verify the schema, not just that the process is up

Sign in to join this conversation.