grenade/moments
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2821548e6e2f5802e9b6955d8bbcc6b03c698285
moments/readme.md
rob thijssen 0aa53d30db docs: rewrite readme to reflect current architecture 
Cover all data sources (github events/search/repo, gitea with org
discovery, hg revset queries, bugzilla), frontend routes (dash with
contribution graphs, activity timeline with timespan filtering,
project detail with readme/languages, cv), api endpoints including
forge proxy and og image, environment variables, and deployment.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-05 18:52:25 +03:00

137 lines
7.0 KiB
Markdown
Raw Blame History

# moments
personal activity timeline and portfolio site. polls public sources (github, gitea, mercurial, bugzilla), stores raw payloads in postgres, and serves a dashboard + project detail views to a react frontend.
successor to the now-defunct [grenade-events-react](https://github.com/grenade/grenade-events-react), which depended on mongodb stitch (retired by mongodb in september 2022).
## layout
```
crates/
moments-entities/ # types and dtos (event, source, project/daily summaries)
moments-core/ # ingestion traits, presentation reshape, poller loop
moments-data/ # postgres adapter, migrations, all event-source impls
moments-api/ # axum read-only http api + forge proxy + og image (binary)
moments-worker/ # ingestion daemon (binary)
ui/ # vite + react + swc + typescript frontend
asset/ # systemd, nginx, firewalld, manifest.yml
script/
deploy.sh # manifest-driven deploy to prod
hg-ingest.sh # one-shot local hg clone + psql ingest
certify.sh # letsencrypt cert management
teardown.sh # service removal
db-perms.sh # postgres role + ident setup
```
architectural conventions follow [grenade/architecture/generic.md](https://git.lair.cafe/grenade/architecture/src/branch/main/generic.md).
## data sources
| source | impl | endpoint | notes |
|--------|------|----------|-------|
| github events | `github.rs` | `/users/{user}/events` | last 90 days, etag-optimised polling |
| github search | `github_search.rs` | `/search/commits` + `/search/issues` | historical backfill, 1000-result cap |
| github repo | `github_repo.rs` | `/user/repos` + `/repos/{o}/{r}/commits` | full commit history, no cap, weekly poll |
| gitea | `gitea.rs` | user + org activity feeds | auto-discovers orgs, filters by user |
| mercurial | `hg.rs` | `json-log?rev=author()` | revset-based, one-shot backfill then skip |
| bugzilla | `bugzilla.rs` | `/rest/bug?creator=` | mozilla bugzilla |
hg repos are archived (mozilla retired hg). the worker skips hg after the first successful scan. for bulk ingestion, `script/hg-ingest.sh` clones repos locally and inserts via psql, avoiding rate limits on hg-edge.mozilla.org.
## frontend routes
| path | page | description |
|------|------|-------------|
| `/` or `/dash` | dashboard | contribution graphs (daily + all-time weekly) + ranked project cards with forge icons and language info |
| `/activity` | timeline | filterable activity feed with source toggles, date range slider, and event limit |
| `/activity/:timespan` | timeline | pre-filtered by date (`YYYY-MM-DD`) or range (`YYYY-MM-DD..YYYY-MM-DD`) |
| `/project/:source/*` | project detail | repo readme, language breakdown bar, per-repo activity timeline |
| `/cv` | resume | loaded from github gist, markdown-rendered |
shared layout provides nav header (dash, activity, cv + external links) and footer across all routes.
## api endpoints
| method | path | description |
|--------|------|-------------|
| GET | `/v1/healthz` | liveness probe |
| GET | `/v1/events?from=&to=&source=&repo=&limit=` | reshaped timeline items |
| GET | `/v1/sources` | per-source summary (count, earliest, latest) |
| GET | `/v1/projects` | per-repo aggregated stats (commits, issues, prs, date range) |
| GET | `/v1/activity/daily?from=&to=` | per-day event counts for contribution graphs |
| GET | `/v1/forge/{source}/*?host=` | proxy to github/gitea apis (avoids cors) |
| GET | `/v1/og/contributions.png` | server-rendered contribution graph as png (resvg) |
the og image endpoint renders the all-time weekly contribution graph as svg, rasterizes to png via resvg, and serves it with a 1-hour cache. used as the `og:image` meta tag for social media previews.
## local development
```sh
cargo build --workspace
cargo run -p moments-api # serves on 127.0.0.1:8080
cargo run -p moments-worker # starts all pollers
cd ui && npm install && npm run dev # vite dev server on :5173
```
the api expects a postgres reachable at `DATABASE_URL`. in production this is an mtls connection using the host cert. for local dev against a throwaway database:
```sh
DATABASE_URL=postgres://localhost/moments cargo run -p moments-api
```
migrations live in `crates/moments-data/migrations/` and run automatically on worker startup. the api connects as `moments_ro` and never runs migrations — the worker (as `moments_rw`) is the schema owner.
## deployment
```sh
./script/deploy.sh <env> all # api + worker + web
./script/deploy.sh <env> api worker # subset
./script/deploy.sh <env> default # api + web only (worker untouched)
./script/deploy.sh <env> all --dry-run
```
concrete hosts, ports, and the site's `server_name` live in `asset/manifest.yml`. the shape of the deployment:
| component | notes |
|-----------|-------|
| api | binds the port from `api.config.bind`; firewalld service `moments-api` |
| worker | no listening port; pollers only |
| web | per-site nginx ingress; `/api/*` reverse-proxies to the api host |
| db | postgres mtls, passwordless |
postgres roles `moments_rw` and `moments_ro` must exist on the primary, with `pg_ident.conf.d/<host>.conf` mapping the api host's fqdn to `moments_ro` and the worker host's fqdn to `moments_rw`. see `asset/sql/bootstrap-moments.sql`, `asset/postgres/ident.conf.tmpl`, and `script/db-perms.sh`.
secrets are resolved at deploy time via `pass`. the mapping of env-var name to pass-store path lives under `worker.secrets` in `manifest.yml`; `deploy.sh` iterates the map, fetches each secret, and substitutes the matching `{{NAME}}` placeholder in `worker.env.tmpl`.
## environment variables
### worker
| variable | default | description |
|----------|---------|-------------|
| `DATABASE_URL` | required | postgres connection string |
| `GITHUB_USER` | `grenade` | github username |
| `GITHUB_TOKEN` | optional | github pat for higher rate limits + private events |
| `POLL_INTERVAL_SECS` | `600` | github events api poll interval |
| `SEARCH_POLL_INTERVAL_SECS` | `86400` | github search backfill interval |
| `REPO_POLL_INTERVAL_SECS` | `604800` | github per-repo commit enumeration (weekly) |
| `GITEA_HOST` | `git.lair.cafe` | gitea instance hostname |
| `GITEA_USER` | `grenade` | gitea username |
| `GITEA_TOKEN` | optional | gitea token for org discovery |
| `GITEA_POLL_INTERVAL_SECS` | `600` | gitea activity feed poll interval |
| `HG_HOST` | `hg-edge.mozilla.org` | mercurial host |
| `HG_GROUPS` | `build,integration` | hg repo groups to discover |
| `HG_REPOS` | `mozilla-central` | individual hg repos |
| `HG_AUTHOR_TERMS` | `rthijssen,grenade` | author substrings for revset queries |
| `HG_POLL_INTERVAL_SECS` | `86400` | hg poll interval (skips after first scan) |
| `BUGZILLA_HOST` | `bugzilla.mozilla.org` | bugzilla instance |
| `BUGZILLA_EMAIL` | `rthijssen@mozilla.com` | bugzilla creator email filter |
| `BUGZILLA_POLL_INTERVAL_SECS` | `86400` | bugzilla poll interval |
### api
| variable | default | description |
|----------|---------|-------------|
| `DATABASE_URL` | required | postgres connection string (read-only role) |
| `BIND_ADDR` | `127.0.0.1:8080` | api listen address |
Reference in New Issue View Git Blame Copy Permalink