grenade/moments
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3260bfb35b8a6320c1201100853467f3aab66719
moments/readme.md
rob thijssen 3260bfb35b fix(ci): share one nginx-vhost renderer; drop unused deploy.sh 
The nightly refresh.yml and deploy.yml each substituted asset/nginx/
site.conf.tmpl with their own inline python. When bb2f5b1 templated the
listen line as {{WEB_LISTEN}} (moving the vhost behind oolon's stream SNI
router), it added the substitution to the template and deploy.yml but not
to refresh.yml. The daily refresh then rsynced a literal
`listen {{WEB_LISTEN}};` into /etc/nginx/conf.d/rob.tn.conf, `nginx -t`
failed for the whole edge, and — because the file is written into the live
conf.d before it is tested — every vhost's reload (including the step@
cert renewals) stayed frozen. Internal vhosts, cichlid.internal among
them, served certs that had expired days earlier while the renewed certs
sat unused on disk.

- Replace both inline renderers with script/render-site-conf.py, shared by
  deploy.yml and refresh.yml so they cannot drift on what they substitute.
- Guard rails: the renderer fails if any {{PLACEHOLDER}} lacks an env value
  or survives substitution, so a forgotten/misnamed variable is a red build
  on the runner instead of a broken vhost on the edge.
- Add the missing WEB_LISTEN to refresh.yml's env (the immediate drift).
- Rename the template's {{DOCROOT}} to {{WEB_ROOT}} so every placeholder
  maps to the env var of the same name.
- Remove script/deploy.sh: the third, unused renderer of the same template
  (superseded by the Actions workflows) and a standing source of drift.
- Docs (readme, CLAUDE.md) updated to the Actions-only deploy path.

Known follow-up (needs a sudoers change + infra-setup re-run on oolon, so
out of scope here): the rendered vhost is still rsynced straight into the
live conf.d and only then `nginx -t`'d, so a valid-but-wrong config could
still wedge nginx. Stage-validate-swap with rollback would close that.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QsH1rcWQYtRVhvaftiKm22
2026-07-26 15:52:18 +03:00

137 lines
7.7 KiB
Markdown
Raw Blame History

# moments
personal activity timeline and portfolio site. polls public sources (github, gitea, mercurial, bugzilla), stores raw payloads in postgres, and serves a dashboard + project detail views to a react frontend.
successor to the now-defunct [grenade-events-react](https://github.com/grenade/grenade-events-react), which depended on mongodb stitch (retired by mongodb in september 2022).
## layout
```
crates/
moments-entities/ # types and dtos (event, source, project/daily summaries)
moments-core/ # ingestion traits, presentation reshape, poller loop
moments-data/ # postgres adapter, migrations, all event-source impls
moments-api/ # axum read-only http api + forge proxy + og image (binary)
moments-worker/ # ingestion daemon (binary)
ui/ # vite + react + swc + typescript frontend
asset/ # systemd, nginx, firewalld, manifest.yml
script/
render-site-conf.py # render the nginx vhost from env (shared by both workflows)
hg-ingest.sh # one-shot local hg clone + psql ingest
certify.sh # letsencrypt cert management
teardown.sh # service removal
db-perms.sh # postgres role + ident setup
```
architectural conventions follow [grenade/architecture/generic.md](https://git.lair.cafe/grenade/architecture/src/branch/main/generic.md).
## data sources
| source | impl | endpoint | notes |
|--------|------|----------|-------|
| github events | `github.rs` | `/users/{user}/events` | last 90 days, etag-optimised polling |
| github search | `github_search.rs` | `/search/commits` + `/search/issues` | historical backfill, 1000-result cap |
| github repo | `github_repo.rs` | `/user/repos` + `/repos/{o}/{r}/commits` | full commit history, no cap, weekly poll |
| gitea | `gitea.rs` | user + org activity feeds | auto-discovers orgs, filters by user |
| mercurial | `hg.rs` | `json-log?rev=author()` | revset-based, one-shot backfill then skip |
| bugzilla | `bugzilla.rs` | `/rest/bug?creator=` | mozilla bugzilla |
hg repos are archived (mozilla retired hg). the worker skips hg after the first successful scan. for bulk ingestion, `script/hg-ingest.sh` clones repos locally and inserts via psql, avoiding rate limits on hg-edge.mozilla.org.
## frontend routes
| path | page | description |
|------|------|-------------|
| `/` or `/dash` | dashboard | contribution graphs (daily + all-time weekly) + ranked project cards with forge icons and language info |
| `/activity` | timeline | filterable activity feed with source toggles, date range slider, and event limit |
| `/activity/:timespan` | timeline | pre-filtered by date (`YYYY-MM-DD`) or range (`YYYY-MM-DD..YYYY-MM-DD`) |
| `/project/:source/*` | project detail | repo readme, language breakdown bar, per-repo activity timeline |
| `/cv` | resume | loaded from github gist, markdown-rendered |
shared layout provides nav header (dash, activity, cv + external links) and footer across all routes.
## api endpoints
| method | path | description |
|--------|------|-------------|
| GET | `/v1/healthz` | liveness probe |
| GET | `/v1/events?from=&to=&source=&repo=&limit=` | reshaped timeline items |
| GET | `/v1/sources` | per-source summary (count, earliest, latest) |
| GET | `/v1/projects` | per-repo aggregated stats (commits, issues, prs, date range) |
| GET | `/v1/activity/daily?from=&to=` | per-day event counts for contribution graphs |
| GET | `/v1/forge/{source}/*?host=` | proxy to github/gitea apis (avoids cors) |
| GET | `/v1/og/contributions.png` | server-rendered contribution graph as png (resvg) |
the og image endpoint renders the all-time weekly contribution graph as svg, rasterizes to png via resvg, and serves it with a 1-hour cache. used as the `og:image` meta tag for social media previews.
## local development
```sh
cargo build --workspace
cargo run -p moments-api # serves on 127.0.0.1:8080
cargo run -p moments-worker # starts all pollers
cd ui && npm install && npm run dev # vite dev server on :5173
```
the api expects a postgres reachable at `DATABASE_URL`. in production this is an mtls connection using the host cert. for local dev against a throwaway database:
```sh
DATABASE_URL=postgres://localhost/moments cargo run -p moments-api
```
migrations live in `crates/moments-data/migrations/` and run automatically on worker startup. the api connects as `moments_ro` and never runs migrations — the worker (as `moments_rw`) is the schema owner.
## deployment
deployment is driven by Gitea Actions, not an operator workstation:
- `.gitea/workflows/deploy.yml` — on push to `main` (or manual dispatch): lint/test gate, build the api + worker as static musl binaries and the prerendered web bundle, then deploy each component over SSH as the `gitea_ci` user with scoped sudo (`asset/sudoers.d/`).
- `.gitea/workflows/refresh.yml` — daily `schedule:` (or manual): rebuilds and redeploys only the web tier, re-baking the prerendered crawler snapshot without bouncing the api/worker.
both workflows carry the infra truth (hosts, ports, paths) in their `env:` blocks and render the nginx vhost through the shared `script/render-site-conf.py`, which fails the build if any template placeholder is unset rather than shipping it. one-time per-host provisioning (the `gitea_ci` user, its `authorized_keys`, the scoped sudoers drop-in) is `script/infra-setup.sh`.
the shape of the deployment:
| component | notes |
|-----------|-------|
| api | binds the port from `api.config.bind`; firewalld service `moments-api` |
| worker | no listening port; pollers only |
| web | per-site nginx ingress; `/api/*` reverse-proxies to the api host |
| db | postgres mtls, passwordless |
postgres roles `moments_rw` and `moments_ro` must exist on the primary, with `pg_ident.conf.d/<host>.conf` mapping the api host's fqdn to `moments_ro` and the worker host's fqdn to `moments_rw`. see `asset/sql/bootstrap-moments.sql`, `asset/postgres/ident.conf.tmpl`, and `script/db-perms.sh`.
the worker's poller tokens are Gitea repo Actions secrets (`QUERY_GITHUB_TOKEN`, `QUERY_GITEA_TOKEN` — the bare `GITHUB_TOKEN`/`GITEA_TOKEN` names are reserved by Actions). `deploy.yml`'s deploy-worker job substitutes them into the matching `{{NAME}}` placeholders in `worker.env.tmpl` at deploy time; secrets come from the runner environment and never touch a command line.
## environment variables
### worker
| variable | default | description |
|----------|---------|-------------|
| `DATABASE_URL` | required | postgres connection string |
| `GITHUB_USER` | `grenade` | github username |
| `GITHUB_TOKEN` | optional | github pat for higher rate limits + private events |
| `POLL_INTERVAL_SECS` | `600` | github events api poll interval |
| `SEARCH_POLL_INTERVAL_SECS` | `86400` | github search backfill interval |
| `REPO_POLL_INTERVAL_SECS` | `604800` | github per-repo commit enumeration (weekly) |
| `GITEA_HOST` | `git.lair.cafe` | gitea instance hostname |
| `GITEA_USER` | `grenade` | gitea username |
| `GITEA_TOKEN` | optional | gitea token for org discovery |
| `GITEA_POLL_INTERVAL_SECS` | `600` | gitea activity feed poll interval |
| `HG_HOST` | `hg-edge.mozilla.org` | mercurial host |
| `HG_GROUPS` | `build,integration` | hg repo groups to discover |
| `HG_REPOS` | `mozilla-central` | individual hg repos |
| `HG_AUTHOR_TERMS` | `rthijssen,grenade` | author substrings for revset queries |
| `HG_POLL_INTERVAL_SECS` | `86400` | hg poll interval (skips after first scan) |
| `BUGZILLA_HOST` | `bugzilla.mozilla.org` | bugzilla instance |
| `BUGZILLA_EMAIL` | `rthijssen@mozilla.com` | bugzilla creator email filter |
| `BUGZILLA_POLL_INTERVAL_SECS` | `86400` | bugzilla poll interval |
### api
| variable | default | description |
|----------|---------|-------------|
| `DATABASE_URL` | required | postgres connection string (read-only role) |
| `BIND_ADDR` | `127.0.0.1:8080` | api listen address |
Reference in New Issue View Git Blame Copy Permalink