rob thijssen
45ceec2ec7
Adds the first ingestion source. Page-1 polling is ETag-conditional
(304s don't count against rate limit); the very first run paginates
back through Link "next" pages up to a 10-page safety cap so the
table starts populated rather than waiting for new activity.
Hits /users/{user}/events/public — works without auth, returns the
right scope for a public timeline. Token (GITHUB_TOKEN) is optional;
when present it raises the rate limit from 60 to 5000/hr.
New plumbing:
moments-core::sources
- EventSource trait (poll() -> count)
- PollerStateStore trait (etag persistence port)
- run_poller driver: tokio interval + jittered exponential backoff
moments-data::github
- GithubSource impl, raw payload preserved as JSONB
- parse_link_next for pagination
- 4 unit tests covering parser + Link parsing
migration 0002_poller_state.sql
- one row per source: source, etag, last_modified, last_fetched
Worker binary spawns one tokio task per source (just github for now)
and aborts on SIGINT. Verified by smoke-curling the upstream endpoint:
ETag and Link headers are present; payload shape matches the parser.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
moments
Personal activity timeline for rob.tn. Polls public sources (GitHub, Gitea, hg-edge.mozilla.org, bugzilla.mozilla.org), stores raw payloads in Postgres, and serves a reshaped timeline to a static React frontend.
Successor to the now-defunct grenade-events-react, which depended on MongoDB Stitch (retired by MongoDB in September 2022).
Layout
crates/
moments-entities/ # types and DTOs
moments-core/ # ingestion + reshape logic
moments-data/ # postgres adapter + migrations
moments-api/ # axum read-only HTTP API (binary)
moments-worker/ # ingestion daemon (binary)
ui/ # vite + react + swc + ts frontend
asset/ # systemd, nginx, firewalld, manifest.yml
script/deploy.sh
Architectural conventions follow grenade/architecture/generic.md.
Local development
cargo build --workspace
cargo run -p moments-api # serves on 127.0.0.1:8080
cargo run -p moments-worker # one-shot ingest tick (until --interval is wired up)
The API expects a Postgres reachable at DATABASE_URL. For magrathea, that's an mTLS connection using the host cert. For local dev against a throwaway database:
DATABASE_URL=postgres://localhost/moments cargo run -p moments-api
Migrations live in crates/moments-data/migrations/ and run automatically on API startup.
Deployment
See asset/manifest.yml and script/deploy.sh.