grenade/moments
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: rewrite readme to reflect current architecture

Browse Source 
Cover all data sources (github events/search/repo, gitea with org
discovery, hg revset queries, bugzilla), frontend routes (dash with
contribution graphs, activity timeline with timespan filtering,
project detail with readme/languages, cv), api endpoints including
forge proxy and og image, environment variables, and deployment.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
rob thijssen
2026-05-05 18:52:25 +03:00
parent cd833b18f1
commit 0aa53d30db
1 changed files with 97 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

121
readme.md
View File

@@ -1,42 +1,87 @@
# moments
Personal activity timeline. Polls public sources (GitHub, Gitea, Mercurial, Bugzilla), stores raw payloads in Postgres, and serves a reshaped timeline to a static React frontend.
personal activity timeline and portfolio site. polls public sources (github, gitea, mercurial, bugzilla), stores raw payloads in postgres, and serves a dashboard + project detail views to a react frontend.
Successor to the now-defunct [grenade-events-react](https://github.com/grenade/grenade-events-react), which depended on MongoDB Stitch (retired by MongoDB in September 2022).
successor to the now-defunct [grenade-events-react](https://github.com/grenade/grenade-events-react), which depended on mongodb stitch (retired by mongodb in september 2022).
## Layout
## layout
```
crates/
moments-entities/ # types and DTOs
moments-core/ # ingestion + reshape logic
moments-data/ # postgres adapter + migrations
moments-api/ # axum read-only HTTP API (binary)
moments-entities/ # types and dtos (event, source, project/daily summaries)
moments-core/ # ingestion traits, presentation reshape, poller loop
moments-data/ # postgres adapter, migrations, all event-source impls
moments-api/ # axum read-only http api + forge proxy + og image (binary)
moments-worker/ # ingestion daemon (binary)
ui/ # vite + react + swc + ts frontend
ui/ # vite + react + swc + typescript frontend
asset/ # systemd, nginx, firewalld, manifest.yml
script/deploy.sh
script/
deploy.sh # manifest-driven deploy to prod
hg-ingest.sh # one-shot local hg clone + psql ingest
certify.sh # letsencrypt cert management
teardown.sh # service removal
db-perms.sh # postgres role + ident setup
```
Architectural conventions follow [grenade/architecture/generic.md](https://git.lair.cafe/grenade/architecture/src/branch/main/generic.md).
architectural conventions follow [grenade/architecture/generic.md](https://git.lair.cafe/grenade/architecture/src/branch/main/generic.md).
## Local development
## data sources
| source | impl | endpoint | notes |
|--------|------|----------|-------|
| github events | `github.rs` | `/users/{user}/events` | last 90 days, etag-optimised polling |
| github search | `github_search.rs` | `/search/commits` + `/search/issues` | historical backfill, 1000-result cap |
| github repo | `github_repo.rs` | `/user/repos` + `/repos/{o}/{r}/commits` | full commit history, no cap, weekly poll |
| gitea | `gitea.rs` | user + org activity feeds | auto-discovers orgs, filters by user |
| mercurial | `hg.rs` | `json-log?rev=author()` | revset-based, one-shot backfill then skip |
| bugzilla | `bugzilla.rs` | `/rest/bug?creator=` | mozilla bugzilla |
hg repos are archived (mozilla retired hg). the worker skips hg after the first successful scan. for bulk ingestion, `script/hg-ingest.sh` clones repos locally and inserts via psql, avoiding rate limits on hg-edge.mozilla.org.
## frontend routes
| path | page | description |
|------|------|-------------|
| `/` or `/dash` | dashboard | contribution graphs (daily + all-time weekly) + ranked project cards with forge icons and language info |
| `/activity` | timeline | filterable activity feed with source toggles, date range slider, and event limit |
| `/activity/:timespan` | timeline | pre-filtered by date (`YYYY-MM-DD`) or range (`YYYY-MM-DD..YYYY-MM-DD`) |
| `/project/:source/*` | project detail | repo readme, language breakdown bar, per-repo activity timeline |
| `/cv` | resume | loaded from github gist, markdown-rendered |
shared layout provides nav header (dash, activity, cv + external links) and footer across all routes.
## api endpoints
| method | path | description |
|--------|------|-------------|
| GET | `/v1/healthz` | liveness probe |
| GET | `/v1/events?from=&to=&source=&repo=&limit=` | reshaped timeline items |
| GET | `/v1/sources` | per-source summary (count, earliest, latest) |
| GET | `/v1/projects` | per-repo aggregated stats (commits, issues, prs, date range) |
| GET | `/v1/activity/daily?from=&to=` | per-day event counts for contribution graphs |
| GET | `/v1/forge/{source}/*?host=` | proxy to github/gitea apis (avoids cors) |
| GET | `/v1/og/contributions.png` | server-rendered contribution graph as png (resvg) |
the og image endpoint renders the all-time weekly contribution graph as svg, rasterizes to png via resvg, and serves it with a 1-hour cache. used as the `og:image` meta tag for social media previews.
## local development
```sh
cargo build --workspace
cargo run -p moments-api # serves on 127.0.0.1:8080
cargo run -p moments-worker # one-shot ingest tick (until --interval is wired up)
cargo run -p moments-worker # starts all pollers
cd ui && npm install && npm run dev # vite dev server on :5173
```
The API expects a Postgres reachable at `DATABASE_URL`. In production this is an mTLS connection using the host cert. For local dev against a throwaway database:
the api expects a postgres reachable at `DATABASE_URL`. in production this is an mtls connection using the host cert. for local dev against a throwaway database:
```sh
DATABASE_URL=postgres://localhost/moments cargo run -p moments-api
```
Migrations live in `crates/moments-data/migrations/` and run automatically on worker startup. The API connects as `moments_ro` and never runs migrations — the worker (as `moments_rw`) is the schema owner.
migrations live in `crates/moments-data/migrations/` and run automatically on worker startup. the api connects as `moments_ro` and never runs migrations — the worker (as `moments_rw`) is the schema owner.
## Deployment
## deployment
```sh
./script/deploy.sh <env> all # api + worker + web
@@ -45,25 +90,47 @@ Migrations live in `crates/moments-data/migrations/` and run automatically on wo
./script/deploy.sh <env> all --dry-run
```
Concrete hosts, ports, and the site's `server_name` live in `asset/manifest.yml`. The shape of the deployment:
concrete hosts, ports, and the site's `server_name` live in `asset/manifest.yml`. the shape of the deployment:
| Component | Notes |
| --------- | ---------------------------------------------------------------------- |
| component | notes |
|-----------|-------|
| api | binds the port from `api.config.bind`; firewalld service `moments-api` |
| worker | no listening port; pollers only |
| web | per-site nginx ingress; `/api/*` reverse-proxies to the api host |
| db | postgres mTLS, passwordless |
| db | postgres mtls, passwordless |
Postgres roles `moments_rw` and `moments_ro` must exist on the primary, with `pg_ident.conf.d/<host>.conf` mapping the api host's FQDN → `moments_ro` and the worker host's FQDN → `moments_rw`. See `asset/sql/bootstrap-moments.sql`, `asset/postgres/ident.conf.tmpl`, and `script/db-perms.sh` (idempotently adds the cert_cn lines on the postgres primary + standby and reloads postgres).
postgres roles `moments_rw` and `moments_ro` must exist on the primary, with `pg_ident.conf.d/<host>.conf` mapping the api host's fqdn to `moments_ro` and the worker host's fqdn to `moments_rw`. see `asset/sql/bootstrap-moments.sql`, `asset/postgres/ident.conf.tmpl`, and `script/db-perms.sh`.
Inter-host traffic over the WG mesh: web's nginx connects to the api host in plaintext. The mesh provides the encryption layer; per-hop TLS for an internal HTTP read-only API on already-public data is deferred. If that changes, swap the api binary to rustls + the host cert pair, and update the nginx upstream to `https://`.
secrets are resolved at deploy time via `pass`. the mapping of env-var name to pass-store path lives under `worker.secrets` in `manifest.yml`; `deploy.sh` iterates the map, fetches each secret, and substitutes the matching `{{NAME}}` placeholder in `worker.env.tmpl`.
Secrets are resolved at deploy time via `pass`. The mapping of env-var name → pass-store path lives under `worker.secrets` in `manifest.yml`; `deploy.sh` iterates the map, fetches each secret, and substitutes the matching `{{NAME}}` placeholder in `worker.env.tmpl`. To add a secret: add a `worker.secrets` entry, add `NAME={{NAME}}` to `worker.env.tmpl`, and ensure `pass show <path>` returns the value on the deploying machine.
## environment variables
### First-time setup
### worker
After the first successful prod deploy:
| variable | default | description |
|----------|---------|-------------|
| `DATABASE_URL` | required | postgres connection string |
| `GITHUB_USER` | `grenade` | github username |
| `GITHUB_TOKEN` | optional | github pat for higher rate limits + private events |
| `POLL_INTERVAL_SECS` | `600` | github events api poll interval |
| `SEARCH_POLL_INTERVAL_SECS` | `86400` | github search backfill interval |
| `REPO_POLL_INTERVAL_SECS` | `604800` | github per-repo commit enumeration (weekly) |
| `GITEA_HOST` | `git.lair.cafe` | gitea instance hostname |
| `GITEA_USER` | `grenade` | gitea username |
| `GITEA_TOKEN` | optional | gitea token for org discovery |
| `GITEA_POLL_INTERVAL_SECS` | `600` | gitea activity feed poll interval |
| `HG_HOST` | `hg-edge.mozilla.org` | mercurial host |
| `HG_GROUPS` | `build,integration` | hg repo groups to discover |
| `HG_REPOS` | `mozilla-central` | individual hg repos |
| `HG_AUTHOR_TERMS` | `rthijssen,grenade` | author substrings for revset queries |
| `HG_POLL_INTERVAL_SECS` | `86400` | hg poll interval (skips after first scan) |
| `BUGZILLA_HOST` | `bugzilla.mozilla.org` | bugzilla instance |
| `BUGZILLA_EMAIL` | `rthijssen@mozilla.com` | bugzilla creator email filter |
| `BUGZILLA_POLL_INTERVAL_SECS` | `86400` | bugzilla poll interval |
1. Point public DNS for the site at the web host's public IP (unproxied).
2. Confirm `curl --fail --silent --show-error https://<site>/api/v1/healthz` returns `ok`.
3. If migrating from a predecessor, archive the old repo with a pointer to this one.
### api
| variable | default | description |
|----------|---------|-------------|
| `DATABASE_URL` | required | postgres connection string (read-only role) |
| `BIND_ADDR` | `127.0.0.1:8080` | api listen address |