rob thijssen
815bfa7deb
`events.public` was decided once, when a row was ingested, from whatever the forge reported at that moment — and every poller is incremental (the github events feed caps at 90 days, search at its top-1000 window, the per-repo scanner at a `since` cursor, the gitea feed at page 1 after the first run). Nothing ever revisited a repo, so flipping one to private upstream only relabelled whatever activity happened afterwards: its history kept serving commit messages, issue titles and the repo name indefinitely. The reverse flip was equally frozen. The github and gitea sources now run a reconciliation pass before ingesting. github reuses its existing repo discovery, which already re-reads `private`/`isPrivate` for everything reachable, and only spends a request on repos we're still exposing that discovery didn't return; gitea has no equivalent bulk endpoint, so it asks per repo, once a day rather than once a tick. Repos already hidden are skipped — they can't leak, and staying hidden is the safe direction to err in. A 404 counts as private (with the user's own token, a repo still in reach answers 200 even when private, so 404 means gone or transferred), while rate limits and transient errors flip nothing; the pass runs in both directions, so a spurious hide is undone by the next successful poll. That needs a repo key the worker can UPDATE against, hence `events.repo` — a stored generated column, and now the single definition of the payload -> repo mapping that list_events, list_projects, activity_summary and language_daily_counts each carried their own copy of. Consolidating them fixes an attribution gap along the way: /search/issues items carry neither `repo.name` nor `repository.full_name`, only `repository_url`, so every issue and PR backfilled through search resolved to NULL in all four queries. Those events now attach to their repo, which both makes them reconcilable and means they show up in /projects and /activity/summary. Also closes a leak that predates the flip problem: `/v1/languages/repos` had no visibility gate at all, and repo_languages is populated for every repo the worker discovers, private ones included. Repo names were on the wire (and baked into the prerendered HTML via the dehydrated query cache) regardless of what `events.public` said. Rather than a second visibility column to keep in sync, the response now derives it — a repo's languages are exposed exactly when at least one of its events is. Verified against postgres 16: the generated column extracts every payload shape the four sources produce (and NULLs a non-github `repository_url`), the reconciliation UPDATE flips all three github event shapes for a repo in one statement and is a no-op on re-run, the gitea host filter excludes other hosts while treating rows predating the `_host` stamp as local, and /v1/languages/repos drops a repo once its events go private. Closes #6
7.4 KiB
CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
Project Overview
moments is a personal activity timeline and portfolio site. It ingests developer activity from multiple forges (GitHub, Gitea, Mercurial, Bugzilla), stores raw JSON payloads in PostgreSQL, and serves a React frontend showing contribution graphs, a ranked project dashboard, and a filterable activity timeline.
Architecture
Hexagonal (ports & adapters) Rust backend with a React/TypeScript frontend.
Crate Dependency Graph
moments-entities — pure types/DTOs, no DB or HTTP deps
^
moments-core — port traits (EventReader, EventWriter, EventSource, PollerStateStore)
+ presentation reshape + poller loop
^
moments-data — sole adapter: PgStore implements all core traits
+ EventSource impls (github, gitea, hg, bugzilla)
+ SQL migrations
^
moments-api — axum HTTP API binary (read-only, connects as moments_ro)
moments-worker — ingestion daemon binary (runs migrations, connects as moments_rw)
Key Design Decisions
- Raw payload storage: upstream JSON is stored verbatim in
events.payload(JSONB). Thereshape()function inmoments-core/src/presentation.rstransforms payloads intoTimelineItemat request time — no re-ingestion needed to change presentation. - Public/private gate:
events.publicboolean controls API visibility. Onlypublic = truerows are served on the detail endpoints (events,projects,activity/summary,languages/repos); the count endpoints (activity/daily,activity/hourly,sources,languages/daily) passinclude_private = true, so private work shows up as volume without leaking repo names or messages. - Visibility reconciliation:
publicis stamped at ingest from whatever the forge reported then, and every poller is incremental — so nothing would ever revisit a repo that later flipped public ↔ private. The github and gitea sources therefore run a reconciliation pass before ingesting (reconcile_visibilityingithub_repo.rs/gitea.rs): they re-read current visibility andUPDATE events SET publicfor the whole history of each repo. It keys offevents.repo, a stored generated column (migration 0006) that derives the repo from the payload — the same expression four read queries used to each carry their own copy of. Repos already hidden are skipped (they can't leak, and staying hidden is the safe direction), and a 404 counts as private: with the user's own token, a repo still in reach answers 200 even when private. Rate limits and transient errors never flip anything. - Wire types are hand-maintained:
ui/src/api/client.tsmirrors Rust entity types manually. - Migrations: run automatically on worker startup via
sqlx::migrate!. The API binary never runs migrations.
Frontend
React 19 + Vite 6 (SWC) + TypeScript + Bootstrap 5. State/data via @tanstack/react-query. Package manager is pnpm.
Routes: / (dashboard), /activity (timeline), /project/:source/* (project detail), /blog + /blog/:slug (blog), /cv (resume).
Build & Dev Commands
Rust
cargo build --workspace # build all crates
cargo build --workspace --release # release build
cargo clippy --workspace # lint
cargo fmt --check # format check
cargo test --workspace # run tests
# Run binaries (need DATABASE_URL)
DATABASE_URL=postgres://localhost/moments cargo run -p moments-api
DATABASE_URL=postgres://localhost/moments cargo run -p moments-worker
Frontend
cd ui
pnpm install # install deps
pnpm dev # dev server on :5173 (proxies /api/* to localhost:8080)
pnpm lint # tsc --noEmit type-check
pnpm build # production build: client bundle, then prerender
The build is three steps (see ui/package.json): tsc -b → vite build (client
SPA) → pnpm run prerender (an SSR build of src/entry-server.tsx, driven by
run-prerender.mjs, that bakes one static index.html per route into ui/dist/).
The prerender fetches data at build time from VITE_API_BASE (default
https://rob.tn/api/v1) and inlines the dehydrated react-query cache as
window.__RQ_STATE__; the client hydrates it and refetches live. So a plain
curl of any route returns full content (for crawlers / AI screeners), while the
browser keeps full interactivity. Date formatting in the shared tree is pinned to
UTC + explicit field widths so SSR and client hydration match byte-for-byte.
Database
PostgreSQL with migrations in crates/moments-data/migrations/. Two roles: moments_rw (worker, full access) and moments_ro (API, SELECT-only).
API Endpoints
All under /v1/: healthz, events, sources, projects, blog, blog/{slug}, activity/daily, forge/{source}/*, og/contributions.png.
Blog posts are markdown files with YAML frontmatter (title, slug, date; optional draft/public) in the grenade/blog Gitea repo. The worker's BlogSource polls the repo (branch-tip sha as change detection) and upserts posts into events with source='blog' and occurred_at from the frontmatter date, so imported posts keep their original publish dates. The repo is the source of truth for the full set of posts: publishing, editing, renaming, and deleting are all just pushes — each poll upserts the current tree and prunes source='blog' rows that are no longer in it.
Deployment
CI-driven via Gitea Actions (.gitea/workflows/), the source of infra truth
(hosts/ports/paths live in the workflow env, not a manifest):
deploy.yml— on push tomain(or manual dispatch): lint/test gate, build the api + worker as static musl binaries (pure-rustls, so no glibc skew) and the prerendered web bundle, then deploy each component over SSH as thegitea_ciuser with scoped sudo (asset/sudoers.d/). Services run under systemd with hardened units; the api/worker reach postgres over mTLS using the host cert.refresh.yml— dailyschedule:(+ manual): rebuilds and redeploys only the web tier, re-baking the prerendered crawler snapshot from the current gist (CV) and activity API without bouncing the api/worker.
One-time per-host provisioning (the gitea_ci user, its authorized_keys, the
scoped sudoers drop-in) is script/infra-setup.sh, run once per host by an
operator. Gitea repo secrets: RSYNC_SSH_KEY, QUERY_GITHUB_TOKEN,
QUERY_GITEA_TOKEN (the bare GITHUB_TOKEN/GITEA_TOKEN names are reserved by
Actions, so the worker poller's tokens use the QUERY_ prefix).
Nginx reverse-proxies /api/ to the API host and serves the per-route static
files via try_files $uri $uri/ /index.html.
Both workflows render the nginx vhost through the shared script/render-site-conf.py
rather than an inline substitution per workflow. It requires every {{PLACEHOLDER}}
in asset/nginx/site.conf.tmpl to have a matching env var and refuses to emit a
file with any placeholder left unrendered — so a variable added to the template
but forgotten in one workflow's env: fails that build instead of shipping a
broken vhost to the edge. (The former per-workflow renderers drifted exactly this
way once: WEB_LISTEN reached the template and deploy.yml but not refresh.yml,
and the nightly refresh froze every reload on oolon.)