rob thijssen
7e186f76ae
The worker owns migrations — it connects as moments_rw, while the api is SELECT-only and would fail with `permission denied for schema public` if it tried. Nothing ordered the two deploy jobs, though, and they run against different hosts (nikola, frootmig), so the systemd ordering the comment in moments-api/src/main.rs appeals to cannot reach between them. On the run that shipped the `events.repo` migration, the worker happened to restart 5 seconds before the api. Reversed, the api would have come up querying a column that did not exist yet and errored on every request touching it — and `/v1/healthz` returns a static "ok" without touching the database, so the deploy's own health probe would have passed it as fine. deploy-api now needs deploy-worker. That ordering is only worth anything if the worker job stays open until migrations are actually done, and it didn't: the unit is Type=simple, so `systemctl restart` returns once the process is exec'd and `is-active` is true immediately, neither of which says anything about migrations. So the job now waits for the worker to log "worker started", which it does only after awaiting store.migrate(). The search is scoped to the unit's current InvocationID so a previous start's line cannot satisfy the wait, and it gives up loudly after 120s or as soon as the process exits. The wait deliberately avoids `journalctl | grep -q`: a consumer that exits on first match closes the pipe, journalctl dies of SIGPIPE, and under `pipefail` the match reads as a failure — which is exactly what the first draft did, and it would have failed every deploy on a worker that started perfectly. journalctl --grep does the filtering instead and the result is tested for emptiness, with no pipe in the pipeline. Verified by extracting the gate out of the workflow and running it against fake systemctl/journalctl: started -> exit 0 on the first check; process gone -> exit 1 immediately with "worker exited before reporting startup"; no InvocationID -> exit 1 immediately; never logs but stays active -> loops and fails on timeout. Job graph re-checked acyclic with every `needs:` resolving to a real job (binaries -> worker -> api -> web). Closes #8
moments
personal activity timeline and portfolio site. polls public sources (github, gitea, mercurial, bugzilla), stores raw payloads in postgres, and serves a dashboard + project detail views to a react frontend.
successor to the now-defunct grenade-events-react, which depended on mongodb stitch (retired by mongodb in september 2022).
layout
crates/
moments-entities/ # types and dtos (event, source, project/daily summaries)
moments-core/ # ingestion traits, presentation reshape, poller loop
moments-data/ # postgres adapter, migrations, all event-source impls
moments-api/ # axum read-only http api + forge proxy + og image (binary)
moments-worker/ # ingestion daemon (binary)
ui/ # vite + react + swc + typescript frontend
asset/ # systemd, nginx, firewalld, manifest.yml
script/
render-site-conf.py # render the nginx vhost from env (shared by both workflows)
hg-ingest.sh # one-shot local hg clone + psql ingest
certify.sh # letsencrypt cert management
teardown.sh # service removal
db-perms.sh # postgres role + ident setup
architectural conventions follow grenade/architecture/generic.md.
data sources
| source | impl | endpoint | notes |
|---|---|---|---|
| github events | github.rs |
/users/{user}/events |
last 90 days, etag-optimised polling |
| github search | github_search.rs |
/search/commits + /search/issues |
historical backfill, 1000-result cap |
| github repo | github_repo.rs |
/user/repos + /repos/{o}/{r}/commits |
full commit history, no cap, weekly poll |
| gitea | gitea.rs |
user + org activity feeds | auto-discovers orgs, filters by user |
| mercurial | hg.rs |
json-log?rev=author() |
revset-based, one-shot backfill then skip |
| bugzilla | bugzilla.rs |
/rest/bug?creator= |
mozilla bugzilla |
hg repos are archived (mozilla retired hg). the worker skips hg after the first successful scan. for bulk ingestion, script/hg-ingest.sh clones repos locally and inserts via psql, avoiding rate limits on hg-edge.mozilla.org.
frontend routes
| path | page | description |
|---|---|---|
/ or /dash |
dashboard | contribution graphs (daily + all-time weekly) + ranked project cards with forge icons and language info |
/activity |
timeline | filterable activity feed with source toggles, date range slider, and event limit |
/activity/:timespan |
timeline | pre-filtered by date (YYYY-MM-DD) or range (YYYY-MM-DD..YYYY-MM-DD) |
/project/:source/* |
project detail | repo readme, language breakdown bar, per-repo activity timeline |
/cv |
resume | loaded from github gist, markdown-rendered |
shared layout provides nav header (dash, activity, cv + external links) and footer across all routes.
api endpoints
| method | path | description |
|---|---|---|
| GET | /v1/healthz |
liveness probe |
| GET | /v1/events?from=&to=&source=&repo=&limit= |
reshaped timeline items |
| GET | /v1/sources |
per-source summary (count, earliest, latest) |
| GET | /v1/projects |
per-repo aggregated stats (commits, issues, prs, date range) |
| GET | /v1/activity/daily?from=&to= |
per-day event counts for contribution graphs |
| GET | /v1/forge/{source}/*?host= |
proxy to github/gitea apis (avoids cors) |
| GET | /v1/og/contributions.png |
server-rendered contribution graph as png (resvg) |
the og image endpoint renders the all-time weekly contribution graph as svg, rasterizes to png via resvg, and serves it with a 1-hour cache. used as the og:image meta tag for social media previews.
local development
cargo build --workspace
cargo run -p moments-api # serves on 127.0.0.1:8080
cargo run -p moments-worker # starts all pollers
cd ui && npm install && npm run dev # vite dev server on :5173
the api expects a postgres reachable at DATABASE_URL. in production this is an mtls connection using the host cert. for local dev against a throwaway database:
DATABASE_URL=postgres://localhost/moments cargo run -p moments-api
migrations live in crates/moments-data/migrations/ and run automatically on worker startup. the api connects as moments_ro and never runs migrations — the worker (as moments_rw) is the schema owner.
private work is counted but never described: events.public gates the detail
endpoints (events, projects, activity/summary, languages/repos) while the
count endpoints (activity/daily, activity/hourly, sources, languages/daily)
include everything, so a private repo shows up as volume on the contribution graph
without leaking its name or commit messages. because that flag is stamped at ingest
and the pollers are all incremental, the github and gitea sources re-read current
repo visibility on each poll and rewrite events.public across the affected repo's
whole history — a repo flipped to private upstream stops being described, one
flipped back to public reappears.
deployment
deployment is driven by Gitea Actions, not an operator workstation:
.gitea/workflows/deploy.yml— on push tomain(or manual dispatch): lint/test gate, build the api + worker as static musl binaries and the prerendered web bundle, then deploy each component over SSH as thegitea_ciuser with scoped sudo (asset/sudoers.d/)..gitea/workflows/refresh.yml— dailyschedule:(or manual): rebuilds and redeploys only the web tier, re-baking the prerendered crawler snapshot without bouncing the api/worker.
both workflows carry the infra truth (hosts, ports, paths) in their env: blocks and render the nginx vhost through the shared script/render-site-conf.py, which fails the build if any template placeholder is unset rather than shipping it. one-time per-host provisioning (the gitea_ci user, its authorized_keys, the scoped sudoers drop-in) is script/infra-setup.sh.
the shape of the deployment:
| component | notes |
|---|---|
| api | binds the port from api.config.bind; firewalld service moments-api |
| worker | no listening port; pollers only |
| web | per-site nginx ingress; /api/* reverse-proxies to the api host |
| db | postgres mtls, passwordless |
postgres roles moments_rw and moments_ro must exist on the primary, with pg_ident.conf.d/<host>.conf mapping the api host's fqdn to moments_ro and the worker host's fqdn to moments_rw. see asset/sql/bootstrap-moments.sql, asset/postgres/ident.conf.tmpl, and script/db-perms.sh.
the worker's poller tokens are Gitea repo Actions secrets (QUERY_GITHUB_TOKEN, QUERY_GITEA_TOKEN — the bare GITHUB_TOKEN/GITEA_TOKEN names are reserved by Actions). deploy.yml's deploy-worker job substitutes them into the matching {{NAME}} placeholders in worker.env.tmpl at deploy time; secrets come from the runner environment and never touch a command line.
environment variables
worker
| variable | default | description |
|---|---|---|
DATABASE_URL |
required | postgres connection string |
GITHUB_USER |
grenade |
github username |
GITHUB_TOKEN |
optional | github pat for higher rate limits + private events |
POLL_INTERVAL_SECS |
600 |
github events api poll interval |
SEARCH_POLL_INTERVAL_SECS |
86400 |
github search backfill interval |
REPO_POLL_INTERVAL_SECS |
604800 |
github per-repo commit enumeration (weekly) |
GITEA_HOST |
git.lair.cafe |
gitea instance hostname |
GITEA_USER |
grenade |
gitea username |
GITEA_TOKEN |
optional | gitea token for org discovery |
GITEA_POLL_INTERVAL_SECS |
600 |
gitea activity feed poll interval |
HG_HOST |
hg-edge.mozilla.org |
mercurial host |
HG_GROUPS |
build,integration |
hg repo groups to discover |
HG_REPOS |
mozilla-central |
individual hg repos |
HG_AUTHOR_TERMS |
rthijssen,grenade |
author substrings for revset queries |
HG_POLL_INTERVAL_SECS |
86400 |
hg poll interval (skips after first scan) |
BUGZILLA_HOST |
bugzilla.mozilla.org |
bugzilla instance |
BUGZILLA_EMAIL |
rthijssen@mozilla.com |
bugzilla creator email filter |
BUGZILLA_POLL_INTERVAL_SECS |
86400 |
bugzilla poll interval |
api
| variable | default | description |
|---|---|---|
DATABASE_URL |
required | postgres connection string (read-only role) |
BIND_ADDR |
127.0.0.1:8080 |
api listen address |