Adds a new github-repo EventSource that enumerates all repos via
/user/repos and walks each repo's /commits?author= endpoint, which
has no 1000-result cap unlike the Search API. Events use the same
github-commit:{sha} ID scheme as github_search for dedup. Per-repo
poller state enables full backfill on first run, page-1-only on
subsequent polls. Weekly poll interval by default.
Closes #1
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove /dev/null redirects in hg-ingest.sh so errors are visible.
cd to work dir before loop to prevent getcwd failures after rm.
Use $HOME instead of ~ for proper expansion in default values.
Reduce timeline entry title, subtitle, and body font sizes for a
more compact activity feed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rewrites the hg worker to use json-log?rev=author() which matches the
changeset author (not the pusher), capturing commits landed by sheriffs.
Repos are discovered within configured groups plus individually listed
repos. The worker skips entirely after the first successful backfill.
Adds script/hg-ingest.sh for offline ingestion via local hg clones —
clones one repo at a time, caches extracted changesets to .tsv, inserts
via psql, and sets poller_state when done.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The user activity feed only returns events from the user's own namespace.
This adds org discovery via /api/v1/user/orgs and polls each org's
activity feed, filtering for events by the configured user. Per-org
poller state keys enable independent backfill. Org feed errors are
non-fatal to avoid disrupting the user feed poll.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
reproduces the legacy cv (previously at grenade.github.io/cv) as a
react-router /cv route, fetched at runtime from the same gist. moves
the lowercase aesthetic from per-element overrides to a single body-
level rule so a future toggle can flip it from one place. adds a small
site-wide footer noting why no cookie consent banner is shown.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
deploy.sh:
- never rsync into /; stage to /tmp on the remote and install at final
paths via sudo bash heredoc, closing the parent-dir attribute leak
that broke three hosts in the earlier rsync incident
- shell-quote heredoc args via ${var@Q}
- drop -A -X on the remaining (web) rsyncs
- generic worker.secrets loop reads (env-var → pass path) from manifest;
GITEA_TOKEN now flows through automatically
- in-memory bash substitution for templates (secrets never on argv)
- simplify semanage port labelling: --add 2>/dev/null || --modify (the
old grep pre-check matched only the first listed port)
- restorecon back to short flags (Fedora policycoreutils has no long
forms; --recursive errored at deploy time)
- quieter health probe loop: curl diagnostics only on final failure
manifest as source of truth:
- api.config.bind drives BIND_ADDR, firewalld port, semanage label,
health-probe URL
- web.config.{server_name,root,api_upstream} drives nginx render,
rsync targets, restorecon scope
- nginx config renamed to site.conf.tmpl; firewalld svc to
moments-api.xml.tmpl; both rendered at deploy time
- topology flip: api → nikola, worker → frootmig (anjie freed)
new scripts:
- script/teardown.sh: idempotent component teardown, never rsyncs,
shared-state cleanup gated on absence of remaining env files,
--remove-docroot guard against shallow / system paths
- script/db-perms.sh: rewritten — fixes grep/append role mismatch that
appended duplicates on re-run, adds postgres reload, hits primary +
standby in a single invocation
readme: genericized; deployment topology no longer carries real host
or site names.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
nikola and frootmig are flagging power events and drive warnings on
the iLO interface and need drive replacement. Move both moments
components onto anjie.kosherinata.internal until those hosts are
back in service. Update the nginx upstream and the readme topology
table to match; the postgres pg_ident.conf on magrathea now needs
to map anjie's cert CN to both moments_ro and moments_rw (two lines
for the same cert_cn).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These ship in a public repo; topology narration in nginx, systemd,
firewalld, and env templates is gratuitous. Keep the config terse —
directives speak for themselves.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The per-site nginx ingress for rob.tn lives on oolon (the host the
external router forwards 443 traffic to), not on nikola. Adjust the
topology so:
- web (static ui + nginx) → oolon.hanzalova.internal
- api binds 0.0.0.0:42424 on nikola.kosherinata.internal so oolon
can reverse-proxy across the WG mesh
- new firewalld service moments-api opens 42424 in the default zone
on nikola
- oolon labels port 42424 http_port_t so httpd_t may name_connect
outbound to it (httpd_can_network_connect was already set)
- nginx ssl_certificate switched to oolon's host cert; upstream
rewritten to nikola.kosherinata.internal:42424
Plaintext between oolon and nikola for now — the WG mesh provides
the encryption layer and the data is already public. Documented
the deferral so a future move to per-hop mTLS is obvious.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires up the prod deployment per architecture-doc conventions:
- api → nikola.kosherinata.internal, loopback bind 127.0.0.1:42424
(less-common port, registered with SELinux as http_port_t).
- worker → frootmig.kosherinata.internal, no listening port.
- web (static ui/dist + nginx server_name rob.tn) → nikola, with
/api/* reverse-proxied to the loopback API.
- db → existing magrathea cluster via mTLS, hostname-baked DATABASE_URL
rendered into /etc/moments/{api,worker}.env at deploy time.
Cert rotation: step-ca renews host certs every 24h; .path units watch
/etc/pki/tls/misc/<host>.pem and trigger systemctl restart of the
relevant service. Both binaries hold cert state in rustls and read
once at startup, so restart is the right reload semantics.
deploy.sh contract matches the architecture doc: positional env arg,
component list (or `all` / `default`), --dry-run support. Renders
config templates from `pass`, rsyncs over ssh+sudo, runs sysusers /
restorecon / semanage / systemctl / nginx -t idempotently.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires two historical sources for completeness with the 2019 timeline:
- hg-edge.mozilla.org: scans json-pushes for a configured set of
build/* repos and matches changeset author client-side, since the
pushlog `user=` filter targets the pusher (sheriffs/reviewers in
this case) rather than the author. Daily poll cadence — mozilla
retired hg, no new events expected.
- bugzilla.mozilla.org: queries /rest/bug?creator=<email>. Without
an api key the unauthenticated endpoint only returns public bugs,
which is what the public timeline wants anyway.
Reshape renders "<author> committed <short_node> in <repo>" for hg
and "filed bug #<id> in <product>" for bugzilla, both linking back
to the canonical upstream URL via a stamped `_host` payload field.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hits /api/v1/users/{user}/activities/feeds?only-performed-by=true
on the configured gitea host (default git.lair.cafe). Page-1 polling
on a 10-min cadence; first run paginates back through up to 20
pages (1000 items) to seed history.
Gitea has no ETag support on this endpoint, so each tick is a fresh
fetch — relying on idempotent upsert by `gitea:<id>` for dedup.
Reshape covers the gitea op_type set:
commit_repo → "pushed N commits to repo:branch" + commits body,
parsing the JSON-encoded `content` field
push_tag → "tagged X in repo"
create_repo → "created repo"
rename/transfer/delete_branch/delete_tag/star/fork — straightforward
create/close/reopen_issue → "{verb} issue #N in repo: title"
create/close/reopen_pull_request → "{verb} pull request #N"
merge_pull_request → GitMerge icon
comment_issue, comment_pull → markdown body from comment.body
approve/reject_pull_request, publish_release
fallback for anything else (mirror_sync_*, future op_types)
Issue / PR / release events use gitea's pipe-separated
`<index>|<title>` content field; pushes have JSON-encoded content.
Host stamping: parse_gitea_event injects `_host` into each row's
payload so the reshape layer can construct web URLs without a
config dependency. Multi-host gitea would still work as long as
each source instance has its own host configured.
Worker config:
GITEA_HOST default git.lair.cafe
GITEA_USER default grenade
GITEA_TOKEN optional (raises rate limit; required
for private repo activity to surface)
GITEA_POLL_INTERVAL_SECS default 600
Tests: +2 in moments-data (commit_repo parses, private flag
captured), +4 in moments-core (commit_repo with body, create_issue
pipe-content, merge icon swap, fallback) — 27 total green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PushEvent payloads carry `created`, `forced`, `distinct_size`, and
`ref` flags that I wasn't consulting — the result on the timeline
was "pushed 0 commits" for what were actually branch creations
(distinct_size 0 because the commits already existed elsewhere)
and force-pushes that didn't change the resulting tree.
* created=true → "created branch X in repo" + GitBranchCreate icon
* forced + size>0 → "force-pushed N commits to repo:branch"
* forced + size==0 → "force-pushed repo:branch"
* normal + size>0 → "pushed N commits to repo:branch" (unchanged)
* normal + size==0 → "pushed to repo:branch" (no awkward "0 commits")
Also: drop the instagram, facebook, and steel-horse-adventures
links from the UI header — those represent personae the user no
longer wants to surface from rob.tn.
Tests: +3 in presentation/github.rs covering the new push
branches — 21 total green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tower-http's TraceLayer logged the failure status code but not the
underlying error, leaving 500s opaque without curling the response
body. Log the error from the internal() helper so server logs carry
the actual cause (permission denied, query error, etc.).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The api connects as the read-only role and was failing on startup
with `permission denied for schema public` because moments_ro lacks
CREATE rights — moments_rw owns the database and runs migrations.
Migrations are now owned exclusively by moments-worker. In deploy
(step 7) systemd ordering ensures the worker runs at least once
before the api unit starts, so the schema is in place by the time
the api accepts traffic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the CRA + React 16 + class-component frontend with the
shape from architecture/generic.md §4: vite + react + swc + ts,
served as static from nginx in prod, vite dev server in dev with
/api proxied to localhost:8080.
Layout:
ui/
package.json, vite.config.ts, tsconfig.{json,app,node}.json
index.html
src/
main.tsx — react root + react-query provider
App.tsx — header, filters, vertical timeline
App.css — dark backdrop, hot-pink links
api/client.ts — TS types mirroring moments-entities;
fetchEvents, fetchSources via /api/v1
components/
Filters.tsx — source toggles, count slider, date range
TimelineEntry.tsx — renders one TimelineItem with body
support for markdown, commits, links
lib/icon.tsx — TimelineIcon → react-bootstrap-icons map
+ colour per icon
Stack: react 19, @tanstack/react-query 5, react-bootstrap 2 (on
bootstrap 5), react-vertical-timeline-component 3, rc-slider 11
(<Slider range /> replaces the removed v8 Range), react-markdown 9.
Dev proxy: /api/* → http://localhost:8080/* (rewrite strips /api).
Backend stays location-agnostic at /v1; ingress prefix is added
by nginx (and the dev proxy) so the same fetch shape works in
both environments.
Verified: tsc -b clean, vite build clean (417 KB js / 245 KB css
gzip 128 / 33), vite dev server serves the index. NOT verified
visually in a browser — that's a `pnpm run dev` away on roosta
once the api is up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Walk back the earlier decision to skip /search/commits. The fork
inflation that worried me isn't misattribution — those commits
really were authored by the user; they just persist in forks after
the original repo went away. Skipping them dropped legitimate
historical work from the timeline.
The duplicate-SHA-across-forks issue is a pure dedup concern:
* keyed `github-commit:<sha>` (SHA only — globally unique by Git's
content addressing; same commit in two forks lands in one row);
* within a single page, dedup by id before INSERT (postgres ON
CONFLICT errors when the conflict target appears twice in one
statement);
* across pages and runs, last-write-wins via upsert. The repo
association may flip between forks but the commit content is
identical.
Visibility is read inline from `repository.private` on the search
item, no extra lookup needed. Also opportunistically populates the
shared visibility cache so the issue loop in the same poll skips
/repos/{full_name} GETs for any repo it already saw via commits.
Reshape: presentation/github.rs gains a Commit path — short SHA
linked, repo linked, first line of the commit message as subtitle.
GitCommit icon.
Tests: +3 in github_search (parse uses sha as id, marks private,
rejects non-github URL), +1 in presentation (commit reshape uses
short sha + first message line) — 18 total green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Events API is hard-capped at 90 days (15 events for grenade
right now). The Search API has its own 1000-result-per-query cap
but reaches the start of the user's GitHub history — for grenade,
430 issues/PRs going back to 2012-08-08.
GET /search/issues?q=author:<user>&sort=created&order=desc
Polled hourly by default but defaults to 24h interval since this is
backfill, not a live feed. After the first run most upserts are
no-ops. Stored as Source::Github with action "Issue" or "PullRequest"
(distinguished by the .pull_request field on the search item),
keyed `github-issue:<owner>/<repo>#<n>`.
/search/commits is deliberately not used: GitHub matches the same
commit across every fork that contains it, so 275k of grenade's
"commits" are mostly duplicated fork hits in repos he never authored
to. If commit history becomes valuable we should enumerate his repos
and walk per-repo /commits?author= instead.
Visibility: search/issues items don't carry .private, so we lookup
/repos/{full_name} once per unique repo encountered (cached for the
duration of the poll). Failure to resolve is treated as private —
better to under-expose than over-expose on the public timeline.
Reshape: presentation/github.rs gains an Issue/PullRequest path that
extracts from the search item shape (html_url, number, title, state,
.pull_request.merged_at) rather than the events-API wrapper. Merged
PRs use the GitMerge icon, mirroring the events-API path.
Worker now spawns two tokio tasks (events + search), aborts both
on SIGINT. New env: SEARCH_POLL_INTERVAL_SECS (default 86400).
Tests: +2 in moments-data (URL parsing), +2 in moments-core
(search Issue + merged-PR reshape) — 14 total green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The DB now stores everything GitHub will give us, the API only ever
returns public events (for now).
Endpoint switch in the github poller: when GITHUB_TOKEN is set we
hit /users/{u}/events (public + private), otherwise fall back to
/users/{u}/events/public. Either way each event's top-level `public`
boolean is captured into a new column.
Schema:
migration 0003_event_public.sql adds events.public BOOLEAN NOT NULL
DEFAULT true, plus an index on (public, occurred_at DESC).
Wire:
Event gains a `public: bool` field.
EventQuery gains `include_private: bool` (default false).
list_events and source_summaries gate on it.
moments-api pins include_private = false at every call site —
threading it as a query param is a future-auth concern, not now.
The default-true on the column keeps existing rows correct: the 11
events already in the DB came from /events/public and are genuinely
public.
After this change, clear poller_state so the next worker run does a
fresh backfill via /events:
DELETE FROM poller_state WHERE source = 'github';
Tests: +2 in github poller (private flag captured, default-public
on missing field) — 10 total green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GET /v1/events now returns the presentation form rather than raw
upstream payloads. The frontend stays dumb: it renders title /
subtitle / body segments and picks an icon from a small kebab-case
enum. Title and subtitle are arrays of {text} | {text, url} segments
so the UI can interleave plain copy with anchors without parsing.
New entities (in moments-entities):
TimelineItem — id, source, action, occurred_at, icon, title, subtitle, body
TitleSegment — Text | Link
TimelineBody — Markdown | Commits | Links
CommitSummary — sha, short_sha, message, url, author
TimelineIcon — kebab-case enum; UI falls back to Generic on unknowns
Reshape lives in moments-core::presentation, dispatched by source.
github.rs covers the event types observed on grenade's feed:
PushEvent, PullRequest{,Review,ReviewComment}Event, Issues{,Comment}Event,
Create/Delete/Fork/Watch/Release/CommitComment/PublicEvent.
Anything else falls back to a generic "<action> on <repo>" line.
Other sources (gitea, hg, bugzilla) currently use a stub fallback;
they get their own reshape modules in steps 5 and 6.
4 unit tests cover the load-bearing cases (push commit list, merged
PR icon swap, issue-comment markdown body, unknown-event fallback).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous bootstrap docs implied a `-U postgres` connection that
won't work over the network — postgres peer auth is local-socket
only. Document the two paths that actually work on this infra:
(a) mTLS as the network superuser `grenade` using the host cert
via PGSSL* env vars (cert paths from /etc/pki/tls per §11).
(b) ssh to the db host and sudo to the local postgres peer.
No script changes — only comments in bootstrap.sql and
bootstrap-moments.sql.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the first ingestion source. Page-1 polling is ETag-conditional
(304s don't count against rate limit); the very first run paginates
back through Link "next" pages up to a 10-page safety cap so the
table starts populated rather than waiting for new activity.
Hits /users/{user}/events/public — works without auth, returns the
right scope for a public timeline. Token (GITHUB_TOKEN) is optional;
when present it raises the rate limit from 60 to 5000/hr.
New plumbing:
moments-core::sources
- EventSource trait (poll() -> count)
- PollerStateStore trait (etag persistence port)
- run_poller driver: tokio interval + jittered exponential backoff
moments-data::github
- GithubSource impl, raw payload preserved as JSONB
- parse_link_next for pagination
- 4 unit tests covering parser + Link parsing
migration 0002_poller_state.sql
- one row per source: source, etag, last_modified, last_fetched
Worker binary spawns one tokio task per source (just github for now)
and aborts on SIGINT. Verified by smoke-curling the upstream endpoint:
ETag and Link headers are present; payload shape matches the parser.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Idempotent SQL for role and database creation, split between the
postgres-database scope (bootstrap.sql) and the moments-database
scope (bootstrap-moments.sql), since CREATE DATABASE can't run
inside a DO block or transaction.
Roles:
moments_rw — owner of the moments database; runs migrations
and writes events from moments-worker.
moments_ro — read-only; consumed by moments-api.
The pg_ident template is rendered per-host by deploy.sh once it
lands; one (host, role) mapping per file. Reload required on both
magrathea and frankie after install — pg_ident is not replicated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
