build-web in deploy.yml has no needs:, so it runs in parallel with build-binaries, deploy-api and deploy-worker. The prerender fetches VITE_API_BASE at build time, which means the job bakes whatever the api happens to be serving at the moment it runs — the old binary or the new one, depending on which runner finishes first.
Observed both outcomes on consecutive runs of the same day:
run 68 — build-web started 16:47:26, deploy-api finished 16:47:51. The prerender's fetches landed after the new api was live, so a repo-visibility fix reached the api and the snapshot together. Won by ~25 seconds.
run 69 — lost the race. The api served the new activity/summary private-aggregate rows while the baked /activity/ snapshot had none: the 15 August card read 47 changes in 3 repositories where the api said 54 with 7 private. Browsers hydrate and refetch so a visitor saw the right thing, but curl — crawlers, AI screeners — got the stale page until refresh.yml was dispatched manually.
Nothing in the pipeline detects this. The snapshot is simply wrong for up to a day, and silently.
Fix
needs: [deploy-api, deploy-worker] on build-web, making the graph serial: binaries → api/worker → web. deploy-worker belongs in the list as well as deploy-api, because the worker owns migrations, so a schema change is not live until it has restarted.
Cost is wall-clock: the web build no longer overlaps the binary build.
Related, not fixed here
Nothing orders deploy-api against deploy-worker either, and they are on different hosts (nikola / frootmig) so the systemd dependency the comment in moments-api/src/main.rs refers to cannot apply across them. /v1/healthz returns a static "ok" without touching the database, so the deploy's health probe passes regardless of whether the schema the new api expects has been migrated yet. On the migration in run 68 the worker happened to restart 5 seconds before the api; the reverse order would have left the api erroring on every query referencing the new column until the worker caught up.
Two candidate fixes, both worth considering separately:
deploy-apineeds: deploy-worker, so migrations always precede the api restart
make healthz schema-aware (cheap query against the columns the binary needs) so the probe fails loudly instead of passing into a broken state
`build-web` in `deploy.yml` has no `needs:`, so it runs in parallel with `build-binaries`, `deploy-api` and `deploy-worker`. The prerender fetches `VITE_API_BASE` at build time, which means the job bakes whatever the api happens to be serving at the moment it runs — the old binary or the new one, depending on which runner finishes first.
Observed both outcomes on consecutive runs of the same day:
- **run 68** — `build-web` started 16:47:26, `deploy-api` finished 16:47:51. The prerender's fetches landed after the new api was live, so a repo-visibility fix reached the api and the snapshot together. Won by ~25 seconds.
- **run 69** — lost the race. The api served the new `activity/summary` private-aggregate rows while the baked `/activity/` snapshot had none: the 15 August card read `47 changes in 3 repositories` where the api said 54 with 7 private. Browsers hydrate and refetch so a visitor saw the right thing, but `curl` — crawlers, AI screeners — got the stale page until `refresh.yml` was dispatched manually.
Nothing in the pipeline detects this. The snapshot is simply wrong for up to a day, and silently.
### Fix
`needs: [deploy-api, deploy-worker]` on `build-web`, making the graph serial: binaries → api/worker → web. `deploy-worker` belongs in the list as well as `deploy-api`, because the worker owns migrations, so a schema change is not live until it has restarted.
Cost is wall-clock: the web build no longer overlaps the binary build.
### Related, not fixed here
Nothing orders `deploy-api` against `deploy-worker` either, and they are on different hosts (nikola / frootmig) so the systemd dependency the comment in `moments-api/src/main.rs` refers to cannot apply across them. `/v1/healthz` returns a static `"ok"` without touching the database, so the deploy's health probe passes regardless of whether the schema the new api expects has been migrated yet. On the migration in run 68 the worker happened to restart 5 seconds before the api; the reverse order would have left the api erroring on every query referencing the new column until the worker caught up.
Two candidate fixes, both worth considering separately:
- `deploy-api` `needs: deploy-worker`, so migrations always precede the api restart
- make `healthz` schema-aware (cheap query against the columns the binary needs) so the probe fails loudly instead of passing into a broken state
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
build-webindeploy.ymlhas noneeds:, so it runs in parallel withbuild-binaries,deploy-apianddeploy-worker. The prerender fetchesVITE_API_BASEat build time, which means the job bakes whatever the api happens to be serving at the moment it runs — the old binary or the new one, depending on which runner finishes first.Observed both outcomes on consecutive runs of the same day:
build-webstarted 16:47:26,deploy-apifinished 16:47:51. The prerender's fetches landed after the new api was live, so a repo-visibility fix reached the api and the snapshot together. Won by ~25 seconds.activity/summaryprivate-aggregate rows while the baked/activity/snapshot had none: the 15 August card read47 changes in 3 repositorieswhere the api said 54 with 7 private. Browsers hydrate and refetch so a visitor saw the right thing, butcurl— crawlers, AI screeners — got the stale page untilrefresh.ymlwas dispatched manually.Nothing in the pipeline detects this. The snapshot is simply wrong for up to a day, and silently.
Fix
needs: [deploy-api, deploy-worker]onbuild-web, making the graph serial: binaries → api/worker → web.deploy-workerbelongs in the list as well asdeploy-api, because the worker owns migrations, so a schema change is not live until it has restarted.Cost is wall-clock: the web build no longer overlaps the binary build.
Related, not fixed here
Nothing orders
deploy-apiagainstdeploy-workereither, and they are on different hosts (nikola / frootmig) so the systemd dependency the comment inmoments-api/src/main.rsrefers to cannot apply across them./v1/healthzreturns a static"ok"without touching the database, so the deploy's health probe passes regardless of whether the schema the new api expects has been migrated yet. On the migration in run 68 the worker happened to restart 5 seconds before the api; the reverse order would have left the api erroring on every query referencing the new column until the worker caught up.Two candidate fixes, both worth considering separately:
deploy-apineeds: deploy-worker, so migrations always precede the api restarthealthzschema-aware (cheap query against the columns the binary needs) so the probe fails loudly instead of passing into a broken state