diff --git a/readme.md b/readme.md index 1f3e09b..5bba4e8 100644 --- a/readme.md +++ b/readme.md @@ -1,42 +1,87 @@ # moments -Personal activity timeline. Polls public sources (GitHub, Gitea, Mercurial, Bugzilla), stores raw payloads in Postgres, and serves a reshaped timeline to a static React frontend. +personal activity timeline and portfolio site. polls public sources (github, gitea, mercurial, bugzilla), stores raw payloads in postgres, and serves a dashboard + project detail views to a react frontend. -Successor to the now-defunct [grenade-events-react](https://github.com/grenade/grenade-events-react), which depended on MongoDB Stitch (retired by MongoDB in September 2022). +successor to the now-defunct [grenade-events-react](https://github.com/grenade/grenade-events-react), which depended on mongodb stitch (retired by mongodb in september 2022). -## Layout +## layout ``` crates/ - moments-entities/ # types and DTOs - moments-core/ # ingestion + reshape logic - moments-data/ # postgres adapter + migrations - moments-api/ # axum read-only HTTP API (binary) + moments-entities/ # types and dtos (event, source, project/daily summaries) + moments-core/ # ingestion traits, presentation reshape, poller loop + moments-data/ # postgres adapter, migrations, all event-source impls + moments-api/ # axum read-only http api + forge proxy + og image (binary) moments-worker/ # ingestion daemon (binary) -ui/ # vite + react + swc + ts frontend +ui/ # vite + react + swc + typescript frontend asset/ # systemd, nginx, firewalld, manifest.yml -script/deploy.sh +script/ + deploy.sh # manifest-driven deploy to prod + hg-ingest.sh # one-shot local hg clone + psql ingest + certify.sh # letsencrypt cert management + teardown.sh # service removal + db-perms.sh # postgres role + ident setup ``` -Architectural conventions follow [grenade/architecture/generic.md](https://git.lair.cafe/grenade/architecture/src/branch/main/generic.md). +architectural conventions follow [grenade/architecture/generic.md](https://git.lair.cafe/grenade/architecture/src/branch/main/generic.md). -## Local development +## data sources + +| source | impl | endpoint | notes | +|--------|------|----------|-------| +| github events | `github.rs` | `/users/{user}/events` | last 90 days, etag-optimised polling | +| github search | `github_search.rs` | `/search/commits` + `/search/issues` | historical backfill, 1000-result cap | +| github repo | `github_repo.rs` | `/user/repos` + `/repos/{o}/{r}/commits` | full commit history, no cap, weekly poll | +| gitea | `gitea.rs` | user + org activity feeds | auto-discovers orgs, filters by user | +| mercurial | `hg.rs` | `json-log?rev=author()` | revset-based, one-shot backfill then skip | +| bugzilla | `bugzilla.rs` | `/rest/bug?creator=` | mozilla bugzilla | + +hg repos are archived (mozilla retired hg). the worker skips hg after the first successful scan. for bulk ingestion, `script/hg-ingest.sh` clones repos locally and inserts via psql, avoiding rate limits on hg-edge.mozilla.org. + +## frontend routes + +| path | page | description | +|------|------|-------------| +| `/` or `/dash` | dashboard | contribution graphs (daily + all-time weekly) + ranked project cards with forge icons and language info | +| `/activity` | timeline | filterable activity feed with source toggles, date range slider, and event limit | +| `/activity/:timespan` | timeline | pre-filtered by date (`YYYY-MM-DD`) or range (`YYYY-MM-DD..YYYY-MM-DD`) | +| `/project/:source/*` | project detail | repo readme, language breakdown bar, per-repo activity timeline | +| `/cv` | resume | loaded from github gist, markdown-rendered | + +shared layout provides nav header (dash, activity, cv + external links) and footer across all routes. + +## api endpoints + +| method | path | description | +|--------|------|-------------| +| GET | `/v1/healthz` | liveness probe | +| GET | `/v1/events?from=&to=&source=&repo=&limit=` | reshaped timeline items | +| GET | `/v1/sources` | per-source summary (count, earliest, latest) | +| GET | `/v1/projects` | per-repo aggregated stats (commits, issues, prs, date range) | +| GET | `/v1/activity/daily?from=&to=` | per-day event counts for contribution graphs | +| GET | `/v1/forge/{source}/*?host=` | proxy to github/gitea apis (avoids cors) | +| GET | `/v1/og/contributions.png` | server-rendered contribution graph as png (resvg) | + +the og image endpoint renders the all-time weekly contribution graph as svg, rasterizes to png via resvg, and serves it with a 1-hour cache. used as the `og:image` meta tag for social media previews. + +## local development ```sh cargo build --workspace cargo run -p moments-api # serves on 127.0.0.1:8080 -cargo run -p moments-worker # one-shot ingest tick (until --interval is wired up) +cargo run -p moments-worker # starts all pollers +cd ui && npm install && npm run dev # vite dev server on :5173 ``` -The API expects a Postgres reachable at `DATABASE_URL`. In production this is an mTLS connection using the host cert. For local dev against a throwaway database: +the api expects a postgres reachable at `DATABASE_URL`. in production this is an mtls connection using the host cert. for local dev against a throwaway database: ```sh DATABASE_URL=postgres://localhost/moments cargo run -p moments-api ``` -Migrations live in `crates/moments-data/migrations/` and run automatically on worker startup. The API connects as `moments_ro` and never runs migrations — the worker (as `moments_rw`) is the schema owner. +migrations live in `crates/moments-data/migrations/` and run automatically on worker startup. the api connects as `moments_ro` and never runs migrations — the worker (as `moments_rw`) is the schema owner. -## Deployment +## deployment ```sh ./script/deploy.sh all # api + worker + web @@ -45,25 +90,47 @@ Migrations live in `crates/moments-data/migrations/` and run automatically on wo ./script/deploy.sh all --dry-run ``` -Concrete hosts, ports, and the site's `server_name` live in `asset/manifest.yml`. The shape of the deployment: +concrete hosts, ports, and the site's `server_name` live in `asset/manifest.yml`. the shape of the deployment: -| Component | Notes | -| --------- | ---------------------------------------------------------------------- | -| api | binds the port from `api.config.bind`; firewalld service `moments-api` | -| worker | no listening port; pollers only | -| web | per-site nginx ingress; `/api/*` reverse-proxies to the api host | -| db | postgres mTLS, passwordless | +| component | notes | +|-----------|-------| +| api | binds the port from `api.config.bind`; firewalld service `moments-api` | +| worker | no listening port; pollers only | +| web | per-site nginx ingress; `/api/*` reverse-proxies to the api host | +| db | postgres mtls, passwordless | -Postgres roles `moments_rw` and `moments_ro` must exist on the primary, with `pg_ident.conf.d/.conf` mapping the api host's FQDN → `moments_ro` and the worker host's FQDN → `moments_rw`. See `asset/sql/bootstrap-moments.sql`, `asset/postgres/ident.conf.tmpl`, and `script/db-perms.sh` (idempotently adds the cert_cn lines on the postgres primary + standby and reloads postgres). +postgres roles `moments_rw` and `moments_ro` must exist on the primary, with `pg_ident.conf.d/.conf` mapping the api host's fqdn to `moments_ro` and the worker host's fqdn to `moments_rw`. see `asset/sql/bootstrap-moments.sql`, `asset/postgres/ident.conf.tmpl`, and `script/db-perms.sh`. -Inter-host traffic over the WG mesh: web's nginx connects to the api host in plaintext. The mesh provides the encryption layer; per-hop TLS for an internal HTTP read-only API on already-public data is deferred. If that changes, swap the api binary to rustls + the host cert pair, and update the nginx upstream to `https://`. +secrets are resolved at deploy time via `pass`. the mapping of env-var name to pass-store path lives under `worker.secrets` in `manifest.yml`; `deploy.sh` iterates the map, fetches each secret, and substitutes the matching `{{NAME}}` placeholder in `worker.env.tmpl`. -Secrets are resolved at deploy time via `pass`. The mapping of env-var name → pass-store path lives under `worker.secrets` in `manifest.yml`; `deploy.sh` iterates the map, fetches each secret, and substitutes the matching `{{NAME}}` placeholder in `worker.env.tmpl`. To add a secret: add a `worker.secrets` entry, add `NAME={{NAME}}` to `worker.env.tmpl`, and ensure `pass show ` returns the value on the deploying machine. +## environment variables -### First-time setup +### worker -After the first successful prod deploy: +| variable | default | description | +|----------|---------|-------------| +| `DATABASE_URL` | required | postgres connection string | +| `GITHUB_USER` | `grenade` | github username | +| `GITHUB_TOKEN` | optional | github pat for higher rate limits + private events | +| `POLL_INTERVAL_SECS` | `600` | github events api poll interval | +| `SEARCH_POLL_INTERVAL_SECS` | `86400` | github search backfill interval | +| `REPO_POLL_INTERVAL_SECS` | `604800` | github per-repo commit enumeration (weekly) | +| `GITEA_HOST` | `git.lair.cafe` | gitea instance hostname | +| `GITEA_USER` | `grenade` | gitea username | +| `GITEA_TOKEN` | optional | gitea token for org discovery | +| `GITEA_POLL_INTERVAL_SECS` | `600` | gitea activity feed poll interval | +| `HG_HOST` | `hg-edge.mozilla.org` | mercurial host | +| `HG_GROUPS` | `build,integration` | hg repo groups to discover | +| `HG_REPOS` | `mozilla-central` | individual hg repos | +| `HG_AUTHOR_TERMS` | `rthijssen,grenade` | author substrings for revset queries | +| `HG_POLL_INTERVAL_SECS` | `86400` | hg poll interval (skips after first scan) | +| `BUGZILLA_HOST` | `bugzilla.mozilla.org` | bugzilla instance | +| `BUGZILLA_EMAIL` | `rthijssen@mozilla.com` | bugzilla creator email filter | +| `BUGZILLA_POLL_INTERVAL_SECS` | `86400` | bugzilla poll interval | -1. Point public DNS for the site at the web host's public IP (unproxied). -2. Confirm `curl --fail --silent --show-error https:///api/v1/healthz` returns `ok`. -3. If migrating from a predecessor, archive the old repo with a pointer to this one. +### api + +| variable | default | description | +|----------|---------|-------------| +| `DATABASE_URL` | required | postgres connection string (read-only role) | +| `BIND_ADDR` | `127.0.0.1:8080` | api listen address |