rob thijssen
815bfa7deb
`events.public` was decided once, when a row was ingested, from whatever the forge reported at that moment — and every poller is incremental (the github events feed caps at 90 days, search at its top-1000 window, the per-repo scanner at a `since` cursor, the gitea feed at page 1 after the first run). Nothing ever revisited a repo, so flipping one to private upstream only relabelled whatever activity happened afterwards: its history kept serving commit messages, issue titles and the repo name indefinitely. The reverse flip was equally frozen. The github and gitea sources now run a reconciliation pass before ingesting. github reuses its existing repo discovery, which already re-reads `private`/`isPrivate` for everything reachable, and only spends a request on repos we're still exposing that discovery didn't return; gitea has no equivalent bulk endpoint, so it asks per repo, once a day rather than once a tick. Repos already hidden are skipped — they can't leak, and staying hidden is the safe direction to err in. A 404 counts as private (with the user's own token, a repo still in reach answers 200 even when private, so 404 means gone or transferred), while rate limits and transient errors flip nothing; the pass runs in both directions, so a spurious hide is undone by the next successful poll. That needs a repo key the worker can UPDATE against, hence `events.repo` — a stored generated column, and now the single definition of the payload -> repo mapping that list_events, list_projects, activity_summary and language_daily_counts each carried their own copy of. Consolidating them fixes an attribution gap along the way: /search/issues items carry neither `repo.name` nor `repository.full_name`, only `repository_url`, so every issue and PR backfilled through search resolved to NULL in all four queries. Those events now attach to their repo, which both makes them reconcilable and means they show up in /projects and /activity/summary. Also closes a leak that predates the flip problem: `/v1/languages/repos` had no visibility gate at all, and repo_languages is populated for every repo the worker discovers, private ones included. Repo names were on the wire (and baked into the prerendered HTML via the dehydrated query cache) regardless of what `events.public` said. Rather than a second visibility column to keep in sync, the response now derives it — a repo's languages are exposed exactly when at least one of its events is. Verified against postgres 16: the generated column extracts every payload shape the four sources produce (and NULLs a non-github `repository_url`), the reconciliation UPDATE flips all three github event shapes for a repo in one statement and is a no-op on re-run, the gitea host filter excludes other hosts while treating rows predating the `_host` stamp as local, and /v1/languages/repos drops a repo once its events go private. Closes #6
8.4 KiB
moments
personal activity timeline and portfolio site. polls public sources (github, gitea, mercurial, bugzilla), stores raw payloads in postgres, and serves a dashboard + project detail views to a react frontend.
successor to the now-defunct grenade-events-react, which depended on mongodb stitch (retired by mongodb in september 2022).
layout
crates/
moments-entities/ # types and dtos (event, source, project/daily summaries)
moments-core/ # ingestion traits, presentation reshape, poller loop
moments-data/ # postgres adapter, migrations, all event-source impls
moments-api/ # axum read-only http api + forge proxy + og image (binary)
moments-worker/ # ingestion daemon (binary)
ui/ # vite + react + swc + typescript frontend
asset/ # systemd, nginx, firewalld, manifest.yml
script/
render-site-conf.py # render the nginx vhost from env (shared by both workflows)
hg-ingest.sh # one-shot local hg clone + psql ingest
certify.sh # letsencrypt cert management
teardown.sh # service removal
db-perms.sh # postgres role + ident setup
architectural conventions follow grenade/architecture/generic.md.
data sources
| source | impl | endpoint | notes |
|---|---|---|---|
| github events | github.rs |
/users/{user}/events |
last 90 days, etag-optimised polling |
| github search | github_search.rs |
/search/commits + /search/issues |
historical backfill, 1000-result cap |
| github repo | github_repo.rs |
/user/repos + /repos/{o}/{r}/commits |
full commit history, no cap, weekly poll |
| gitea | gitea.rs |
user + org activity feeds | auto-discovers orgs, filters by user |
| mercurial | hg.rs |
json-log?rev=author() |
revset-based, one-shot backfill then skip |
| bugzilla | bugzilla.rs |
/rest/bug?creator= |
mozilla bugzilla |
hg repos are archived (mozilla retired hg). the worker skips hg after the first successful scan. for bulk ingestion, script/hg-ingest.sh clones repos locally and inserts via psql, avoiding rate limits on hg-edge.mozilla.org.
frontend routes
| path | page | description |
|---|---|---|
/ or /dash |
dashboard | contribution graphs (daily + all-time weekly) + ranked project cards with forge icons and language info |
/activity |
timeline | filterable activity feed with source toggles, date range slider, and event limit |
/activity/:timespan |
timeline | pre-filtered by date (YYYY-MM-DD) or range (YYYY-MM-DD..YYYY-MM-DD) |
/project/:source/* |
project detail | repo readme, language breakdown bar, per-repo activity timeline |
/cv |
resume | loaded from github gist, markdown-rendered |
shared layout provides nav header (dash, activity, cv + external links) and footer across all routes.
api endpoints
| method | path | description |
|---|---|---|
| GET | /v1/healthz |
liveness probe |
| GET | /v1/events?from=&to=&source=&repo=&limit= |
reshaped timeline items |
| GET | /v1/sources |
per-source summary (count, earliest, latest) |
| GET | /v1/projects |
per-repo aggregated stats (commits, issues, prs, date range) |
| GET | /v1/activity/daily?from=&to= |
per-day event counts for contribution graphs |
| GET | /v1/forge/{source}/*?host= |
proxy to github/gitea apis (avoids cors) |
| GET | /v1/og/contributions.png |
server-rendered contribution graph as png (resvg) |
the og image endpoint renders the all-time weekly contribution graph as svg, rasterizes to png via resvg, and serves it with a 1-hour cache. used as the og:image meta tag for social media previews.
local development
cargo build --workspace
cargo run -p moments-api # serves on 127.0.0.1:8080
cargo run -p moments-worker # starts all pollers
cd ui && npm install && npm run dev # vite dev server on :5173
the api expects a postgres reachable at DATABASE_URL. in production this is an mtls connection using the host cert. for local dev against a throwaway database:
DATABASE_URL=postgres://localhost/moments cargo run -p moments-api
migrations live in crates/moments-data/migrations/ and run automatically on worker startup. the api connects as moments_ro and never runs migrations — the worker (as moments_rw) is the schema owner.
private work is counted but never described: events.public gates the detail
endpoints (events, projects, activity/summary, languages/repos) while the
count endpoints (activity/daily, activity/hourly, sources, languages/daily)
include everything, so a private repo shows up as volume on the contribution graph
without leaking its name or commit messages. because that flag is stamped at ingest
and the pollers are all incremental, the github and gitea sources re-read current
repo visibility on each poll and rewrite events.public across the affected repo's
whole history — a repo flipped to private upstream stops being described, one
flipped back to public reappears.
deployment
deployment is driven by Gitea Actions, not an operator workstation:
.gitea/workflows/deploy.yml— on push tomain(or manual dispatch): lint/test gate, build the api + worker as static musl binaries and the prerendered web bundle, then deploy each component over SSH as thegitea_ciuser with scoped sudo (asset/sudoers.d/)..gitea/workflows/refresh.yml— dailyschedule:(or manual): rebuilds and redeploys only the web tier, re-baking the prerendered crawler snapshot without bouncing the api/worker.
both workflows carry the infra truth (hosts, ports, paths) in their env: blocks and render the nginx vhost through the shared script/render-site-conf.py, which fails the build if any template placeholder is unset rather than shipping it. one-time per-host provisioning (the gitea_ci user, its authorized_keys, the scoped sudoers drop-in) is script/infra-setup.sh.
the shape of the deployment:
| component | notes |
|---|---|
| api | binds the port from api.config.bind; firewalld service moments-api |
| worker | no listening port; pollers only |
| web | per-site nginx ingress; /api/* reverse-proxies to the api host |
| db | postgres mtls, passwordless |
postgres roles moments_rw and moments_ro must exist on the primary, with pg_ident.conf.d/<host>.conf mapping the api host's fqdn to moments_ro and the worker host's fqdn to moments_rw. see asset/sql/bootstrap-moments.sql, asset/postgres/ident.conf.tmpl, and script/db-perms.sh.
the worker's poller tokens are Gitea repo Actions secrets (QUERY_GITHUB_TOKEN, QUERY_GITEA_TOKEN — the bare GITHUB_TOKEN/GITEA_TOKEN names are reserved by Actions). deploy.yml's deploy-worker job substitutes them into the matching {{NAME}} placeholders in worker.env.tmpl at deploy time; secrets come from the runner environment and never touch a command line.
environment variables
worker
| variable | default | description |
|---|---|---|
DATABASE_URL |
required | postgres connection string |
GITHUB_USER |
grenade |
github username |
GITHUB_TOKEN |
optional | github pat for higher rate limits + private events |
POLL_INTERVAL_SECS |
600 |
github events api poll interval |
SEARCH_POLL_INTERVAL_SECS |
86400 |
github search backfill interval |
REPO_POLL_INTERVAL_SECS |
604800 |
github per-repo commit enumeration (weekly) |
GITEA_HOST |
git.lair.cafe |
gitea instance hostname |
GITEA_USER |
grenade |
gitea username |
GITEA_TOKEN |
optional | gitea token for org discovery |
GITEA_POLL_INTERVAL_SECS |
600 |
gitea activity feed poll interval |
HG_HOST |
hg-edge.mozilla.org |
mercurial host |
HG_GROUPS |
build,integration |
hg repo groups to discover |
HG_REPOS |
mozilla-central |
individual hg repos |
HG_AUTHOR_TERMS |
rthijssen,grenade |
author substrings for revset queries |
HG_POLL_INTERVAL_SECS |
86400 |
hg poll interval (skips after first scan) |
BUGZILLA_HOST |
bugzilla.mozilla.org |
bugzilla instance |
BUGZILLA_EMAIL |
rthijssen@mozilla.com |
bugzilla creator email filter |
BUGZILLA_POLL_INTERVAL_SECS |
86400 |
bugzilla poll interval |
api
| variable | default | description |
|---|---|---|
DATABASE_URL |
required | postgres connection string (read-only role) |
BIND_ADDR |
127.0.0.1:8080 |
api listen address |