grenade/moments
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: grenade:45ceec2ec72fa7f19a14391ae68c0aa38025bbfc
Branches Tags
grenade:main
..
compare: grenade:main
Branches Tags
grenade:main

80 Commits
45ceec2ec7 ... main

Author SHA1 Message Date
rob thijssen
2821548e6e feat(ui): add avg-by-hour panel to dashboard stats 
Complements the existing avg-by-weekday chart with its orthogonal
partner: which hour of the day the user typically commits. The api
buckets events by EXTRACT(hour FROM occurred_at AT TIME ZONE $tz) so
the chart matches the clock the user sees rather than UTC; the UI
passes the browser's resolved IANA timezone. Renders as 24 mini-bars
below the weekday chart with labels every 4 hours and per-bar
tooltips showing the average events/day at that hour.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 16:34:17 +03:00
rob thijssen
72eeb547af chore(deploy): self-heal /tmp perms before staging 
frootmig periodically has its /tmp reset from the standard sticky-
world-writable 1777 to root-owned 0755 (cause not yet pinned down),
which breaks the unprivileged rsync of the deploy stage dir and
surfaces as a cryptic "Permission denied" plus a follow-on install
failure. Stat /tmp before each rsync and, if the mode is off, sudo
chmod it back to 1777 — visible in the deploy log so it's obvious
which host keeps drifting.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 07:59:06 +03:00
rob thijssen
86411bb88e fix(worker): dedup gitea events from overlapping user and org feeds 
Gitea writes one Action row per interested user-context. A push to an
org repo by user U produces two rows — one with user_id=U, one with
user_id=org — differing only in `id` and `user_id`. Polling both the
user feed and org feeds (which we do, and need to, since neither alone
catches every cross-namespace event) surfaced both rows; the
`gitea:{action_row_id}` id gave them distinct ids, so the upsert dedup
never fired and ~38% of events on org-repo project pages rendered
twice. Switch to a content-derived id keyed on (op_type, act_user_id,
repo_id, ref_name, comment_id, created) so the two rows collide on
upsert, and add a migration that re-keys existing rows to the same
formula while collapsing the duplicates already in the table.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 07:53:43 +03:00
rob thijssen
acb061baca chore(deploy): build rust binaries in a podman container 
Workstation runs Fedora 44 (glibc 2.43); servers are still on F42 and
F43. A native release build produces ELFs the older glibc can't load
(GLIBC_2.43 not found), and the api/worker units fail-loop on every
deploy. Build inside docker.io/library/rust:1-bookworm (glibc 2.36)
so the artifacts are forward-compatible with every Fedora target.
Output goes to target/deploy/ to keep separate from native dev
builds, and the cargo registry/git index are cached in named podman
volumes so subsequent builds are incremental. podman is a hard
requirement; no docker fallback.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 16:05:13 +03:00
rob thijssen
8a7177a54a feat(ui): render GFM and embedded HTML in project READMEs 
ReactMarkdown was running with no plugins, so README headers full of
raw <div align=center>, tables, <details>/<summary>, and other GFM
markup rendered as escaped text. Wire in remark-gfm for tables and
GFM features, rehype-raw for embedded HTML, and rehype-sanitize with
an extended schema that permits README-typical tags and attributes
(align, target, width/height, picture/source, etc.) while still
blocking script/iframe/object — READMEs come from external repos so
they need adversarial-input handling.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 16:05:05 +03:00
rob thijssen
818a535903 feat(worker): capture commits on non-default branches and forks 
The ingestion paths each had a gap that let non-default-branch work
slip through: /search/commits silently excludes forks, the per-repo
REST commit scan only walked the default branch, and the user events
feed ages out after 90 days. Catch them by enumerating branches per
repo and scanning each (with per-branch state cursors so a brand-new
branch isn't cut off by the default branch's cursor), pre-filtering
branches via a GraphQL HEAD-author check so big upstream forks like
azure-docs don't trigger hundreds of wasted REST calls, treating
GitHub's HTTP 500 on author-filtered empty branches as "no commits"
rather than a server error, and adding fork:true to the search query.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 16:04:58 +03:00
rob thijssen
9a8c0955b5 chore: phrasing 2026-05-12 13:20:11 +03:00
rob thijssen
25eab2d795 feat: add robots.txt allowing all crawlers including social bots 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 16:49:55 +03:00
rob thijssen
2130032d46 chore: update Cargo.lock for fontdb dependency 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 16:35:08 +03:00
rob thijssen
92a66422ab feat(ui): add meta description, og:locale, and og:site_name 
Adds the standard HTML meta description (for SEO), og:locale, and
og:site_name tags flagged by Open Graph validators.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 16:32:33 +03:00
rob thijssen
94b6fbe42d feat(ui): add og:logo meta tag pointing to 512px icon 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 16:29:40 +03:00
rob thijssen
048646a7c1 feat(ui): add og:url meta tag for canonical URL 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 16:27:47 +03:00
rob thijssen
1f2fea3427 fix: load system fonts for OG image text rendering 
usvg's default Options creates an empty fontdb, so no fonts are found
for text rendering regardless of what's installed. Load system fonts
into a fontdb::Database and set the default font family to Noto Sans.

Also picks up a formatting change to index.html from a linter.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 16:17:17 +03:00
rob thijssen
d539892b70 fix: scale OG contribution graph to fill 1200x630 canvas 
Compute cell size from available width so the graph fills the canvas
instead of rendering at a fixed small size. Scale year labels
proportionally. Position headline and subtitle at the top with the
graph centered in the remaining space.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 16:11:05 +03:00
rob thijssen
a57682e610 feat: improve OG image and meta tags for social sharing 
- Resize OG image from ~676x216 to 1200x630 (recommended size)
- Add "rob thijssen" headline text overlay to the OG image
- Center the contribution graph within the canvas
- Expand og:title to 55 chars and og:description to 148 chars
  to meet social platform optimal lengths

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 16:08:29 +03:00
rob thijssen
22c80fd7af feat(ui): transpose weekday averages to vertical bar chart 
Show days on the X axis and volume on the Y axis, replacing the
horizontal bar layout with vertical bars for a more natural
time-series reading direction.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 16:02:30 +03:00
rob thijssen
8b5656ef26 fix: specify sans-serif font in OG image SVG text elements 
The SVG text elements had no font-family, causing usvg to default to
Times New Roman which isn't installed on the server. Specifying
sans-serif uses the system default and silences the warnings.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 15:56:24 +03:00
rob thijssen
dd1de38b2f feat(ui): show more languages in top languages chart 
Increase from 10 to 14 rows so languages visible in the contribution
graph (e.g. Svelte, C++) also appear in the legend.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 15:52:33 +03:00
rob thijssen
283b2126c0 feat(ui): color contribution graph circles by dominant language 
Replace fixed green palette with per-period dominant language colors.
Each circle's hue reflects the language with the most commits for that
day (last-year graph) or month (all-time graph), with opacity scaled
by volume quartile. Language data comes from the existing language
daily counts endpoint.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 15:50:19 +03:00
rob thijssen
e8dcb5fcaf feat(ui): show private activity count on timeline when no public events 
When viewing a date range with zero public activities, the status line
now shows the count of private contributions (derived from daily counts
which include private repos). Helps explain gaps in the public timeline.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 15:41:22 +03:00
rob thijssen
b41e8c330a feat: include private repo contributions in graph metrics 
Aggregate graph endpoints (daily counts, language daily counts, source
summaries, OG image) now include private repository activity. These
endpoints only expose numeric counts — no commit messages, repo names,
or other metadata — so private details remain hidden. The activity
timeline continues to serve only public events.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 15:35:22 +03:00
rob thijssen
f386e0b574 feat(ui): reshape all-time graph and add dashboard stats panels 
Transpose AllTimeGraph to show years on X axis and months on Y axis
instead of year-per-row with weekly columns. Add TopLanguages bar chart
(all-time code volume by language) and ContributionStats panel (current
and longest streaks, busiest day, active days, weekday averages) in a
three-column row matching the project card grid.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-11 15:27:39 +03:00
rob thijssen
111a2af573 feat(ui): language distribution bar on project cards 
Extract LanguageBar into a shared component used by both DashPage
(compact, bar only) and ProjectPage (full, with percentage labels).
Remove redundant forge source text from project cards since the
forge icon already indicates it.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-06 07:13:41 +03:00
rob thijssen
6f30a61184 feat(ui): smooth language stream graph with Catmull-Rom splines 
Replace straight line segments with cubic bezier curves using
Catmull-Rom to bezier conversion (tension 0.5) for a smoother
stream graph visualization.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-06 07:02:50 +03:00
rob thijssen
14643273c0 fix: weight language graph by repo language proportions 
Each commit was counted once per language in the repo regardless of
that language's share, so Shell (present in many repos as small
deploy scripts) appeared larger than Rust. Now weights each commit
by the language's byte proportion in the repo (e.g. a commit to a
95% Rust / 5% Shell repo contributes 0.95 to Rust, 0.05 to Shell).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-06 06:59:47 +03:00
rob thijssen
ee93429317 feat: language stream graph on dashboard 
Full-stack feature showing programming languages by commit activity
as a stream graph on the dashboard.

Backend:
- migration: repo_languages table (source, repo, language, bytes, color)
- worker: fetch language breakdowns via GitHub GraphQL (batched,
  20 repos/request) and Gitea REST API during poll cycles
- API: GET /v1/languages/daily (daily commit counts per language),
  GET /v1/languages/repos (all stored repo language data)
- fix timezone bug in daily_counts and language_daily_counts: the
  PostgreSQL server timezone (Europe/Sofia, UTC+3) shifted day
  boundaries, miscounting events near midnight. Now uses explicit
  UTC boundaries in generate_series JOINs.
- use per-source CASE for repo name extraction in language query
  to match gitea payload structure (repo.full_name vs repo.name)
- Gitea languages use GitHub colors via COALESCE fallback

Frontend:
- LanguageStreamGraph component: pure SVG stream graph, weekly
  buckets, centered baseline, top 8 languages + Other, GitHub
  canonical language colors, legend with color dots
- DashPage/ProjectPage: fetch repo languages once via new endpoint
  instead of per-repo forge proxy calls (eliminates 200+ GitHub
  API calls and 403 rate limit errors)
- removed fetchLanguages forge proxy wrapper (dead code)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-06 06:27:59 +03:00
rob thijssen
c66aaeb268 feat: discover contributed repos via GitHub GraphQL API 
The REST /user/repos endpoint only returns repos where the user is
owner, collaborator, or org member. Repos contributed to via PRs
(e.g. polkadot-js/api, zed-industries/zed) were never discovered
and their commits were missing from moments.

Now supplements /user/repos with a GraphQL
repositoriesContributedTo query, which returns all repos the user
has committed to, opened issues/PRs on, or reviewed — with cursor-
based pagination and no result cap.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-06 05:38:57 +03:00
rob thijssen
2a20b47a29 fix: resolve clippy redundant_closure warning in moments-api 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-06 05:04:18 +03:00
rob thijssen
f77a8ab48f fix: use since cursor in github-repo polls to prevent missed commits 
After initial backfill, scan_repo was fetching only page 1 (100 most
recent commits) per repo. If more than 100 commits landed between
7-day polls, older ones in that window were permanently missed.

Now stores the newest commit date in poller_state.last_modified and
passes it as &since= on subsequent polls, with full pagination, so
only genuinely new commits are fetched but none are skipped.

On first poll after deploy, last_modified is NULL so no since filter
is applied — triggering a full re-backfill that catches any
previously missed commits.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-06 05:03:41 +03:00
rob thijssen
1679153c43 docs: add CLAUDE.md and ignore .zed/ 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-06 04:43:00 +03:00
rob thijssen
0aa53d30db docs: rewrite readme to reflect current architecture 
Cover all data sources (github events/search/repo, gitea with org
discovery, hg revset queries, bugzilla), frontend routes (dash with
contribution graphs, activity timeline with timespan filtering,
project detail with readme/languages, cv), api endpoints including
forge proxy and og image, environment variables, and deployment.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 18:52:25 +03:00
rob thijssen
cd833b18f1 fix(ui): demote repos with >= 10k commits to end of dashboard 
Automated/bulk-commit repos score -1 so they sort last regardless
of recency or volume.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 18:48:52 +03:00
rob thijssen
293d112c18 fix: fall back to _repo in commit reshape for github-repo events 
The commit presentation layer only checked repository.full_name,
missing commits ingested by github_repo which store the repo name
in _repo instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 18:44:31 +03:00
rob thijssen
ef1e84a41b feat(ui): link forge icon to repo on project page 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 18:42:04 +03:00
rob thijssen
f8c13b5e21 fix: icon colors for dark backgrounds 2026-05-05 18:40:29 +03:00
rob thijssen
abc90c8da0 feat(ui): forge icon on project page header 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 18:30:13 +03:00
rob thijssen
d46a0e3777 fix: add _repo fallback to events repo filter for github-repo commits 
The events query's COALESCE for github source was missing _repo,
so per-repo commit events from github_repo had no repo match and
project pages showed 0 activities.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 18:28:32 +03:00
rob thijssen
642209068a feat(ui): forge icons on repo cards (github, gitea, mozilla) 
Add SVG icons for each forge before the repo name on dashboard cards.
Icons sourced from user-provided SVGs in ui/public/.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 18:24:47 +03:00
rob thijssen
c1e964de06 feat(ui): show all repos on dashboard instead of top 24 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 18:09:34 +03:00
rob thijssen
45fd45f5da fix: stamp _repo into github-repo commit payloads for project attribution 
The /repos/{owner}/{repo}/commits endpoint doesn't include repo info
in its response. Without _repo in the payload, these commits were
invisible to the projects query. Add _repo to parse_commit and include
it in the COALESCE chain for github source repo extraction.

After deploy, reset github-repo poller state to re-ingest with _repo:
  DELETE FROM poller_state WHERE source LIKE 'github-repo%';

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 17:59:31 +03:00
rob thijssen
03c816d2d3 feat(ui): show repo count in contribution graph summaries 
Add "in N repositories" to both the year and all-time graph summary
lines. Year graph counts repos with overlapping activity; all-time
graph uses total project count. OG image includes repo count too.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 17:50:15 +03:00
rob thijssen
13db392273 fix(nginx): exclude /api/ paths from static asset location block 
The static asset regex matched .png in /api/v1/og/contributions.png
before the /api/ proxy block, returning 404. Add negative lookahead
to skip /api/ paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 17:40:31 +03:00
rob thijssen
e63583877c feat(api): server-rendered OG image of all-time contribution graph 
Add /v1/og/contributions.png endpoint that builds an SVG of the
all-time weekly contribution graph (one row per year) from daily
counts, then rasterizes to PNG via resvg. Served with 1h cache.

Add og:image and twitter:card meta tags to index.html pointing at
the endpoint.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 17:37:19 +03:00
rob thijssen
2284a886d0 fix(ui): all-time graph as year rows with 52 weekly columns each 
Restructure the all-time contribution graph from a single row of ~700
circles (sub-pixel when scaled) to one row per year with ~52 weekly
columns, matching the width of the daily graph above. Year labels on
the left.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 17:15:49 +03:00
rob thijssen
1ca85fe632 feat(ui): all-time weekly contribution graph + date range timespan support 
Add AllTimeGraph component showing one circle per week across the full
history (earliest event to today). Uses the /sources endpoint to find
the earliest date, then fetches daily counts and aggregates to weekly.
Clicking a week navigates to /activity/YYYY-MM-DD..YYYY-MM-DD.

Update parseTimespan to handle both date-only (YYYY-MM-DD) and full
ISO datetime strings in range expressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 17:13:49 +03:00
rob thijssen
822def3227 fix(ui): scale contribution graph to full container width 
Use viewBox + width=100% instead of fixed pixel dimensions so the
SVG scales to match the project card grid below.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 17:08:17 +03:00
rob thijssen
27ce16e630 feat(ui): contribution graph with daily activity heatmap 
Add /v1/activity/daily endpoint returning per-day event counts via
generate_series + LEFT JOIN. Frontend renders an SVG contribution
graph with circles colored by quantile-based thresholds. Clicking a
day navigates to /activity/YYYY-MM-DD showing that day's events.

New /activity/:timespan route parses single dates (YYYY-MM-DD) and
ranges (YYYY-MM-DD..YYYY-MM-DD) from the URL to initialize the
activity timeline filter.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 17:05:28 +03:00
rob thijssen
7de23303bd chore(ui): add favicon set to index.html 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 16:50:19 +03:00
rob thijssen
0d350ce584 fix: decode base64 readme content as utf-8 instead of latin-1 
atob() produces Latin-1 strings, mangling multi-byte UTF-8 characters
like box-drawing glyphs. Use TextDecoder for correct UTF-8 handling.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 16:28:40 +03:00
rob thijssen
1275a7785f chore: update Cargo.lock for reqwest in moments-api 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 16:25:19 +03:00
rob thijssen
6b9ce99a06 fix: proxy forge API requests to avoid CORS, case-insensitive readme 
Add /v1/forge/{source}/* proxy endpoint to the API server with an
allowlisted set of hosts. Frontend readme and language requests now
go through the proxy instead of hitting forge APIs directly (Gitea
has no CORS headers). Gitea readme fetch tries README.md, readme.md,
and Readme.md casings.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 16:24:32 +03:00
rob thijssen
f676ecdc19 fix: try multiple readme filename casings for Gitea repos 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 16:19:34 +03:00
rob thijssen
46ef63a68e fix: source-aware repo extraction, Gitea readme/languages endpoints 
Use CASE/source instead of COALESCE for repo name extraction — Gitea's
repo.name is the short name while full_name includes the owner prefix.
Fix Gitea README fetch to use /contents/README.md with base64 decoding
instead of the nonexistent /readme endpoint.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 16:18:40 +03:00
rob thijssen
ba216580ea feat(ui): project readme, language bars, and per-card language summary 
ProjectPage fetches README (raw markdown) and language breakdown from
GitHub/Gitea REST APIs, rendering the readme as markdown and languages
as a colored proportional bar with labels.

Dashboard cards lazily fetch top 3 languages per repo and display them
inline. Language color map covers common languages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 15:28:15 +03:00
rob thijssen
80f3f7c5cb feat(ui): project drill-down route with repo-filtered event timeline 
Add repo filter param to /v1/events (SQL COALESCE across payload
shapes per source). New /project/:source/* route renders a filtered
activity timeline for a single repo. Dashboard cards link to the
drill-down page.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 15:22:11 +03:00
rob thijssen
a70fab4feb feat(ui): add /dash route, shared nav, project dashboard with /v1/projects API 
Restructure routes: / and /dash show a project overview dashboard,
/activity hosts the existing timeline, /cv remains. Shared Layout
component provides consistent nav header and footer across all routes.

New /v1/projects endpoint aggregates per-repo activity stats (commits,
issues, PRs, date range) from existing event data via SQL. Dashboard
ranks projects by weighted recency + volume score and renders a card
grid.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 15:19:49 +03:00
rob thijssen
a71b4e6b84 feat(github): per-repo commit enumeration for full history backfill 
Adds a new github-repo EventSource that enumerates all repos via
/user/repos and walks each repo's /commits?author= endpoint, which
has no 1000-result cap unlike the Search API. Events use the same
github-commit:{sha} ID scheme as github_search for dedup. Per-repo
poller state enables full backfill on first run, page-1-only on
subsequent polls. Weekly poll interval by default.

Closes #1

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 14:59:26 +03:00
rob thijssen
2da9461b44 fix(hg): show clone errors, stable cwd; shrink timeline fonts 
Remove /dev/null redirects in hg-ingest.sh so errors are visible.
cd to work dir before loop to prevent getcwd failures after rm.
Use $HOME instead of ~ for proper expansion in default values.

Reduce timeline entry title, subtitle, and body font sizes for a
more compact activity feed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 14:45:26 +03:00
rob thijssen
3f3a1fb33e fix: connection string 2026-05-05 14:22:42 +03:00
rob thijssen
88fbbba60b feat(hg): revset-based author query, group discovery, one-shot ingest script 
Rewrites the hg worker to use json-log?rev=author() which matches the
changeset author (not the pusher), capturing commits landed by sheriffs.
Repos are discovered within configured groups plus individually listed
repos. The worker skips entirely after the first successful backfill.

Adds script/hg-ingest.sh for offline ingestion via local hg clones —
clones one repo at a time, caches extracted changesets to .tsv, inserts
via psql, and sets poller_state when done.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 13:58:21 +03:00
rob thijssen
1bbe55dc84 feat(gitea): poll org activity feeds to capture cross-namespace events 
The user activity feed only returns events from the user's own namespace.
This adds org discovery via /api/v1/user/orgs and polls each org's
activity feed, filtering for events by the configured user. Per-org
poller state keys enable independent backfill. Org feed errors are
non-fatal to avoid disrupting the user feed poll.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-05 12:23:25 +03:00
rob thijssen
4c8a663288 feat(ui): add /cv route, site-wide lowercase, no-cookies footer 
reproduces the legacy cv (previously at grenade.github.io/cv) as a
react-router /cv route, fetched at runtime from the same gist. moves
the lowercase aesthetic from per-element overrides to a single body-
level rule so a future toggle can flip it from one place. adds a small
site-wide footer noting why no cookie consent banner is shown.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-04 17:22:44 +03:00
rob thijssen
8867ff5df3 feat(deploy): manifest-driven config, teardown + db-perms, hardening 
deploy.sh:
- never rsync into /; stage to /tmp on the remote and install at final
  paths via sudo bash heredoc, closing the parent-dir attribute leak
  that broke three hosts in the earlier rsync incident
- shell-quote heredoc args via ${var@Q}
- drop -A -X on the remaining (web) rsyncs
- generic worker.secrets loop reads (env-var → pass path) from manifest;
  GITEA_TOKEN now flows through automatically
- in-memory bash substitution for templates (secrets never on argv)
- simplify semanage port labelling: --add 2>/dev/null || --modify (the
  old grep pre-check matched only the first listed port)
- restorecon back to short flags (Fedora policycoreutils has no long
  forms; --recursive errored at deploy time)
- quieter health probe loop: curl diagnostics only on final failure

manifest as source of truth:
- api.config.bind drives BIND_ADDR, firewalld port, semanage label,
  health-probe URL
- web.config.{server_name,root,api_upstream} drives nginx render,
  rsync targets, restorecon scope
- nginx config renamed to site.conf.tmpl; firewalld svc to
  moments-api.xml.tmpl; both rendered at deploy time
- topology flip: api → nikola, worker → frootmig (anjie freed)

new scripts:
- script/teardown.sh: idempotent component teardown, never rsyncs,
  shared-state cleanup gated on absence of remaining env files,
  --remove-docroot guard against shallow / system paths
- script/db-perms.sh: rewritten — fixes grep/append role mismatch that
  appended duplicates on re-run, adds postgres reload, hits primary +
  standby in a single invocation

readme: genericized; deployment topology no longer carries real host
or site names.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-04 16:39:10 +03:00
rob thijssen
f30f949895 fix: ensure root ownership when syncing staged folders 2026-05-04 13:32:12 +03:00
rob thijssen
7843c2c13f chore(deploy): co-locate api + worker on anjie 
nikola and frootmig are flagging power events and drive warnings on
the iLO interface and need drive replacement. Move both moments
components onto anjie.kosherinata.internal until those hosts are
back in service. Update the nginx upstream and the readme topology
table to match; the postgres pg_ident.conf on magrathea now needs
to map anjie's cert CN to both moments_ro and moments_rw (two lines
for the same cert_cn).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-04 08:24:21 +03:00
rob thijssen
c81512fa3e fix: conventional paths, oolon fqdn, public cert 2026-05-04 07:54:23 +03:00
rob thijssen
abce3803ca chore(deploy): strip infra commentary from asset/ config files 
These ship in a public repo; topology narration in nginx, systemd,
firewalld, and env templates is gratuitous. Keep the config terse —
directives speak for themselves.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 20:23:11 +03:00
rob thijssen
52b7d0be9b fix(deploy): split ingress to oolon, expose api on nikola interface 
The per-site nginx ingress for rob.tn lives on oolon (the host the
external router forwards 443 traffic to), not on nikola. Adjust the
topology so:

- web (static ui + nginx) → oolon.hanzalova.internal
- api binds 0.0.0.0:42424 on nikola.kosherinata.internal so oolon
  can reverse-proxy across the WG mesh
- new firewalld service moments-api opens 42424 in the default zone
  on nikola
- oolon labels port 42424 http_port_t so httpd_t may name_connect
  outbound to it (httpd_can_network_connect was already set)
- nginx ssl_certificate switched to oolon's host cert; upstream
  rewritten to nikola.kosherinata.internal:42424

Plaintext between oolon and nikola for now — the WG mesh provides
the encryption layer and the data is already public. Documented
the deferral so a future move to per-hop mTLS is obvious.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 20:20:07 +03:00
rob thijssen
110b523fd0 chore(deploy): add manifest, systemd units, nginx config, deploy.sh 
Wires up the prod deployment per architecture-doc conventions:

- api → nikola.kosherinata.internal, loopback bind 127.0.0.1:42424
  (less-common port, registered with SELinux as http_port_t).
- worker → frootmig.kosherinata.internal, no listening port.
- web (static ui/dist + nginx server_name rob.tn) → nikola, with
  /api/* reverse-proxied to the loopback API.
- db → existing magrathea cluster via mTLS, hostname-baked DATABASE_URL
  rendered into /etc/moments/{api,worker}.env at deploy time.

Cert rotation: step-ca renews host certs every 24h; .path units watch
/etc/pki/tls/misc/<host>.pem and trigger systemctl restart of the
relevant service. Both binaries hold cert state in rustls and read
once at startup, so restart is the right reload semantics.

deploy.sh contract matches the architecture doc: positional env arg,
component list (or `all` / `default`), --dry-run support. Renders
config templates from `pass`, rsyncs over ssh+sudo, runs sysusers /
restorecon / semanage / systemctl / nginx -t idempotently.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 20:17:17 +03:00
rob thijssen
7919a2d9ab feat(worker): add hg-edge and bugzilla pollers 
Wires two historical sources for completeness with the 2019 timeline:

- hg-edge.mozilla.org: scans json-pushes for a configured set of
  build/* repos and matches changeset author client-side, since the
  pushlog `user=` filter targets the pusher (sheriffs/reviewers in
  this case) rather than the author. Daily poll cadence — mozilla
  retired hg, no new events expected.
- bugzilla.mozilla.org: queries /rest/bug?creator=<email>. Without
  an api key the unauthenticated endpoint only returns public bugs,
  which is what the public timeline wants anyway.

Reshape renders "<author> committed <short_node> in <repo>" for hg
and "filed bug #<id> in <product>" for bugzilla, both linking back
to the canonical upstream URL via a stamped `_host` payload field.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 19:55:41 +03:00
rob thijssen
f750e8de47 feat(worker): add gitea activity feed poller 
Hits /api/v1/users/{user}/activities/feeds?only-performed-by=true
on the configured gitea host (default git.lair.cafe). Page-1 polling
on a 10-min cadence; first run paginates back through up to 20
pages (1000 items) to seed history.

Gitea has no ETag support on this endpoint, so each tick is a fresh
fetch — relying on idempotent upsert by `gitea:<id>` for dedup.

Reshape covers the gitea op_type set:
  commit_repo  → "pushed N commits to repo:branch" + commits body,
                  parsing the JSON-encoded `content` field
  push_tag     → "tagged X in repo"
  create_repo  → "created repo"
  rename/transfer/delete_branch/delete_tag/star/fork — straightforward
  create/close/reopen_issue        → "{verb} issue #N in repo: title"
  create/close/reopen_pull_request → "{verb} pull request #N"
  merge_pull_request               → GitMerge icon
  comment_issue, comment_pull      → markdown body from comment.body
  approve/reject_pull_request, publish_release
  fallback for anything else (mirror_sync_*, future op_types)

Issue / PR / release events use gitea's pipe-separated
`<index>|<title>` content field; pushes have JSON-encoded content.

Host stamping: parse_gitea_event injects `_host` into each row's
payload so the reshape layer can construct web URLs without a
config dependency. Multi-host gitea would still work as long as
each source instance has its own host configured.

Worker config:
  GITEA_HOST                  default git.lair.cafe
  GITEA_USER                  default grenade
  GITEA_TOKEN                 optional (raises rate limit; required
                                for private repo activity to surface)
  GITEA_POLL_INTERVAL_SECS    default 600

Tests: +2 in moments-data (commit_repo parses, private flag
captured), +4 in moments-core (commit_repo with body, create_issue
pipe-content, merge icon swap, fallback) — 27 total green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 19:41:55 +03:00
rob thijssen
4355353395 fix(presentation): handle force-push, branch-create, empty pushes 
PushEvent payloads carry `created`, `forced`, `distinct_size`, and
`ref` flags that I wasn't consulting — the result on the timeline
was "pushed 0 commits" for what were actually branch creations
(distinct_size 0 because the commits already existed elsewhere)
and force-pushes that didn't change the resulting tree.

  * created=true        → "created branch X in repo" + GitBranchCreate icon
  * forced + size>0     → "force-pushed N commits to repo:branch"
  * forced + size==0    → "force-pushed repo:branch"
  * normal + size>0     → "pushed N commits to repo:branch" (unchanged)
  * normal + size==0    → "pushed to repo:branch" (no awkward "0 commits")

Also: drop the instagram, facebook, and steel-horse-adventures
links from the UI header — those represent personae the user no
longer wants to surface from rob.tn.

Tests: +3 in presentation/github.rs covering the new push
branches — 21 total green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 19:37:40 +03:00
rob thijssen
bf04f8a1ff fix(api): log internal handler errors 
Tower-http's TraceLayer logged the failure status code but not the
underlying error, leaving 500s opaque without curling the response
body. Log the error from the internal() helper so server logs carry
the actual cause (permission denied, query error, etc.).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 19:31:10 +03:00
rob thijssen
bf7f829d02 fix(api): don't run migrations as moments_ro 
The api connects as the read-only role and was failing on startup
with `permission denied for schema public` because moments_ro lacks
CREATE rights — moments_rw owns the database and runs migrations.

Migrations are now owned exclusively by moments-worker. In deploy
(step 7) systemd ordering ensures the worker runs at least once
before the api unit starts, so the schema is in place by the time
the api accepts traffic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 19:28:32 +03:00
rob thijssen
b04afd83f9 feat(ui): scaffold vite + react 19 frontend 
Replaces the CRA + React 16 + class-component frontend with the
shape from architecture/generic.md §4: vite + react + swc + ts,
served as static from nginx in prod, vite dev server in dev with
/api proxied to localhost:8080.

Layout:
  ui/
    package.json, vite.config.ts, tsconfig.{json,app,node}.json
    index.html
    src/
      main.tsx           — react root + react-query provider
      App.tsx            — header, filters, vertical timeline
      App.css            — dark backdrop, hot-pink links
      api/client.ts      — TS types mirroring moments-entities;
                            fetchEvents, fetchSources via /api/v1
      components/
        Filters.tsx      — source toggles, count slider, date range
        TimelineEntry.tsx — renders one TimelineItem with body
                             support for markdown, commits, links
      lib/icon.tsx       — TimelineIcon → react-bootstrap-icons map
                            + colour per icon

Stack: react 19, @tanstack/react-query 5, react-bootstrap 2 (on
bootstrap 5), react-vertical-timeline-component 3, rc-slider 11
(<Slider range /> replaces the removed v8 Range), react-markdown 9.

Dev proxy: /api/* → http://localhost:8080/* (rewrite strips /api).
Backend stays location-agnostic at /v1; ingress prefix is added
by nginx (and the dev proxy) so the same fetch shape works in
both environments.

Verified: tsc -b clean, vite build clean (417 KB js / 245 KB css
gzip 128 / 33), vite dev server serves the index. NOT verified
visually in a browser — that's a `pnpm run dev` away on roosta
once the api is up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 19:18:32 +03:00
rob thijssen
7772393598 feat(worker): add commits to github search backfill 
Walk back the earlier decision to skip /search/commits. The fork
inflation that worried me isn't misattribution — those commits
really were authored by the user; they just persist in forks after
the original repo went away. Skipping them dropped legitimate
historical work from the timeline.

The duplicate-SHA-across-forks issue is a pure dedup concern:
  * keyed `github-commit:<sha>` (SHA only — globally unique by Git's
    content addressing; same commit in two forks lands in one row);
  * within a single page, dedup by id before INSERT (postgres ON
    CONFLICT errors when the conflict target appears twice in one
    statement);
  * across pages and runs, last-write-wins via upsert. The repo
    association may flip between forks but the commit content is
    identical.

Visibility is read inline from `repository.private` on the search
item, no extra lookup needed. Also opportunistically populates the
shared visibility cache so the issue loop in the same poll skips
/repos/{full_name} GETs for any repo it already saw via commits.

Reshape: presentation/github.rs gains a Commit path — short SHA
linked, repo linked, first line of the commit message as subtitle.
GitCommit icon.

Tests: +3 in github_search (parse uses sha as id, marks private,
rejects non-github URL), +1 in presentation (commit reshape uses
short sha + first message line) — 18 total green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 18:54:32 +03:00
rob thijssen
e4052c4c9a feat(worker): add github search api source for historical backfill 
The Events API is hard-capped at 90 days (15 events for grenade
right now). The Search API has its own 1000-result-per-query cap
but reaches the start of the user's GitHub history — for grenade,
430 issues/PRs going back to 2012-08-08.

  GET /search/issues?q=author:<user>&sort=created&order=desc

Polled hourly by default but defaults to 24h interval since this is
backfill, not a live feed. After the first run most upserts are
no-ops. Stored as Source::Github with action "Issue" or "PullRequest"
(distinguished by the .pull_request field on the search item),
keyed `github-issue:<owner>/<repo>#<n>`.

/search/commits is deliberately not used: GitHub matches the same
commit across every fork that contains it, so 275k of grenade's
"commits" are mostly duplicated fork hits in repos he never authored
to. If commit history becomes valuable we should enumerate his repos
and walk per-repo /commits?author= instead.

Visibility: search/issues items don't carry .private, so we lookup
/repos/{full_name} once per unique repo encountered (cached for the
duration of the poll). Failure to resolve is treated as private —
better to under-expose than over-expose on the public timeline.

Reshape: presentation/github.rs gains an Issue/PullRequest path that
extracts from the search item shape (html_url, number, title, state,
.pull_request.merged_at) rather than the events-API wrapper. Merged
PRs use the GitMerge icon, mirroring the events-API path.

Worker now spawns two tokio tasks (events + search), aborts both
on SIGINT. New env: SEARCH_POLL_INTERVAL_SECS (default 86400).

Tests: +2 in moments-data (URL parsing), +2 in moments-core
(search Issue + merged-PR reshape) — 14 total green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 18:49:06 +03:00
rob thijssen
3c0253519f feat: ingest private events; surface public-only 
The DB now stores everything GitHub will give us, the API only ever
returns public events (for now).

Endpoint switch in the github poller: when GITHUB_TOKEN is set we
hit /users/{u}/events (public + private), otherwise fall back to
/users/{u}/events/public. Either way each event's top-level `public`
boolean is captured into a new column.

Schema:
  migration 0003_event_public.sql adds events.public BOOLEAN NOT NULL
  DEFAULT true, plus an index on (public, occurred_at DESC).

Wire:
  Event gains a `public: bool` field.
  EventQuery gains `include_private: bool` (default false).
  list_events and source_summaries gate on it.
  moments-api pins include_private = false at every call site —
  threading it as a query param is a future-auth concern, not now.

The default-true on the column keeps existing rows correct: the 11
events already in the DB came from /events/public and are genuinely
public.

After this change, clear poller_state so the next worker run does a
fresh backfill via /events:

  DELETE FROM poller_state WHERE source = 'github';

Tests: +2 in github poller (private flag captured, default-public
on missing field) — 10 total green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 18:33:40 +03:00
rob thijssen
003f427e98 feat(api): reshape raw events into TimelineItem 
GET /v1/events now returns the presentation form rather than raw
upstream payloads. The frontend stays dumb: it renders title /
subtitle / body segments and picks an icon from a small kebab-case
enum. Title and subtitle are arrays of {text} | {text, url} segments
so the UI can interleave plain copy with anchors without parsing.

New entities (in moments-entities):

  TimelineItem        — id, source, action, occurred_at, icon, title, subtitle, body
  TitleSegment        — Text | Link
  TimelineBody        — Markdown | Commits | Links
  CommitSummary       — sha, short_sha, message, url, author
  TimelineIcon        — kebab-case enum; UI falls back to Generic on unknowns

Reshape lives in moments-core::presentation, dispatched by source.
github.rs covers the event types observed on grenade's feed:
PushEvent, PullRequest{,Review,ReviewComment}Event, Issues{,Comment}Event,
Create/Delete/Fork/Watch/Release/CommitComment/PublicEvent.
Anything else falls back to a generic "<action> on <repo>" line.
Other sources (gitea, hg, bugzilla) currently use a stub fallback;
they get their own reshape modules in steps 5 and 6.

4 unit tests cover the load-bearing cases (push commit list, merged
PR icon swap, issue-comment markdown body, unknown-event fallback).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 18:08:18 +03:00
rob thijssen
418834c960 docs(asset/sql): document mtls and ssh-sudo run modes 
The previous bootstrap docs implied a `-U postgres` connection that
won't work over the network — postgres peer auth is local-socket
only. Document the two paths that actually work on this infra:

  (a) mTLS as the network superuser `grenade` using the host cert
      via PGSSL* env vars (cert paths from /etc/pki/tls per §11).
  (b) ssh to the db host and sudo to the local postgres peer.

No script changes — only comments in bootstrap.sql and
bootstrap-moments.sql.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 18:07:57 +03:00
88 changed files with 11964 additions and 50 deletions
Expand all files Collapse all files

2
.gitignore vendored
View File

@@ -2,11 +2,13 @@
**/*.rs.bk
.env
.env.local
.zed/
# frontend
/ui/node_modules
/ui/dist
/ui/.vite
*.tsbuildinfo
# rendered configs (templates committed, rendered output never)
/asset/config/*.toml

78
CLAUDE.md Normal file
View File

@@ -0,0 +1,78 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
**moments** is a personal activity timeline and portfolio site. It ingests developer activity from multiple forges (GitHub, Gitea, Mercurial, Bugzilla), stores raw JSON payloads in PostgreSQL, and serves a React frontend showing contribution graphs, a ranked project dashboard, and a filterable activity timeline.
## Architecture
Hexagonal (ports & adapters) Rust backend with a React/TypeScript frontend.
### Crate Dependency Graph
```
moments-entities — pure types/DTOs, no DB or HTTP deps
^
moments-core — port traits (EventReader, EventWriter, EventSource, PollerStateStore)
+ presentation reshape + poller loop
^
moments-data — sole adapter: PgStore implements all core traits
+ EventSource impls (github, gitea, hg, bugzilla)
+ SQL migrations
^
moments-api — axum HTTP API binary (read-only, connects as moments_ro)
moments-worker — ingestion daemon binary (runs migrations, connects as moments_rw)
```
### Key Design Decisions
- **Raw payload storage**: upstream JSON is stored verbatim in `events.payload` (JSONB). The `reshape()` function in `moments-core/src/presentation.rs` transforms payloads into `TimelineItem` at request time — no re-ingestion needed to change presentation.
- **Public/private gate**: `events.public` boolean controls API visibility. Only `public = true` rows are served.
- **Wire types are hand-maintained**: `ui/src/api/client.ts` mirrors Rust entity types manually.
- **Migrations**: run automatically on worker startup via `sqlx::migrate!`. The API binary never runs migrations.
### Frontend
React 19 + Vite 6 (SWC) + TypeScript + Bootstrap 5. State/data via `@tanstack/react-query`. Package manager is **pnpm**.
Routes: `/` (dashboard), `/activity` (timeline), `/project/:source/*` (project detail), `/cv` (resume).
## Build & Dev Commands
### Rust
```sh
cargo build --workspace # build all crates
cargo build --workspace --release # release build
cargo clippy --workspace # lint
cargo fmt --check # format check
cargo test --workspace # run tests
# Run binaries (need DATABASE_URL)
DATABASE_URL=postgres://localhost/moments cargo run -p moments-api
DATABASE_URL=postgres://localhost/moments cargo run -p moments-worker
```
### Frontend
```sh
cd ui
pnpm install # install deps
pnpm dev # dev server on :5173 (proxies /api/* to localhost:8080)
pnpm lint # tsc --noEmit type-check
pnpm build # production build (tsc -b && vite build)
```
## Database
PostgreSQL with three migrations in `crates/moments-data/migrations/`. Two roles: `moments_rw` (worker, full access) and `moments_ro` (API, SELECT-only).
## API Endpoints
All under `/v1/`: `healthz`, `events`, `sources`, `projects`, `activity/daily`, `forge/{source}/*`, `og/contributions.png`.
## Deployment
Production uses `./script/deploy.sh`. Services run under systemd with hardened units. Secrets resolved from `pass` store via template substitution. Nginx reverse-proxies `/api/` to the API host.

391
Cargo.lock generated
View File

@@ -88,6 +88,18 @@ version = "1.0.102"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c"
[[package]]
name = "arrayref"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb"
[[package]]
name = "arrayvec"
version = "0.7.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50"
[[package]]
name = "async-compression"
version = "0.4.42"
@@ -196,6 +208,12 @@ version = "1.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06"
[[package]]
name = "bitflags"
version = "1.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
[[package]]
name = "bitflags"
version = "2.11.1"
@@ -220,12 +238,24 @@ version = "3.20.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d20789868f4b01b2f2caec9f5c4e0213b41e3e5702a50157d699ae31ced2fcb"
[[package]]
name = "bytemuck"
version = "1.25.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec"
[[package]]
name = "byteorder"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b"
[[package]]
name = "byteorder-lite"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f1fe948ff07f4bd06c30984e69f5b4899c516a3ef74f34df92a2df2ab535495"
[[package]]
name = "bytes"
version = "1.11.1"
@@ -306,6 +336,12 @@ version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9"
[[package]]
name = "color_quant"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3d7b894f5411737b7867f4827955924d7c254fc9f4d91a6aad6b097804b1018b"
[[package]]
name = "colorchoice"
version = "1.0.5"
@@ -350,6 +386,15 @@ version = "0.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
[[package]]
name = "core_maths"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "77745e017f5edba1a9c1d854f6f3a52dac8a12dd5af5d2f54aecf61e43d80d30"
dependencies = [
"libm",
]
[[package]]
name = "cpufeatures"
version = "0.2.17"
@@ -408,6 +453,12 @@ dependencies = [
"typenum",
]
[[package]]
name = "data-url"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be1e0bca6c3637f992fc1cc7cbc52a78c1ef6db076dbf1059c4323d6a2048376"
[[package]]
name = "der"
version = "0.7.10"
@@ -484,6 +535,15 @@ dependencies = [
"windows-sys 0.48.0",
]
[[package]]
name = "euclid"
version = "0.22.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f1a05365e3b1c6d1650318537c7460c6923f1abdd272ad6842baa2b509957a06"
dependencies = [
"num-traits",
]
[[package]]
name = "event-listener"
version = "5.4.1"
@@ -495,6 +555,15 @@ dependencies = [
"pin-project-lite",
]
[[package]]
name = "fdeflate"
version = "0.3.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e6853b52649d4ac5c0bd02320cddc5ba956bdb407c4b75a2c6b75bf51500f8c"
dependencies = [
"simd-adler32",
]
[[package]]
name = "find-msvc-tools"
version = "0.1.9"
@@ -511,6 +580,12 @@ dependencies = [
"miniz_oxide",
]
[[package]]
name = "float-cmp"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "98de4bbd547a563b716d8dfa9aad1cb19bfab00f4fa09a6a4ed21dbcf44ce9c4"
[[package]]
name = "flume"
version = "0.11.1"
@@ -528,6 +603,29 @@ version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2"
[[package]]
name = "fontconfig-parser"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbc773e24e02d4ddd8395fd30dc147524273a83e54e0f312d986ea30de5f5646"
dependencies = [
"roxmltree",
]
[[package]]
name = "fontdb"
version = "0.23.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "457e789b3d1202543297a350643cf459f836cade38934e7a4cf6a39e7cde2905"
dependencies = [
"fontconfig-parser",
"log",
"memmap2",
"slotmap",
"tinyvec",
"ttf-parser",
]
[[package]]
name = "form_urlencoded"
version = "1.2.2"
@@ -645,6 +743,16 @@ dependencies = [
"wasm-bindgen",
]
[[package]]
name = "gif"
version = "0.13.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4ae047235e33e2829703574b54fdec96bfbad892062d97fed2f76022287de61b"
dependencies = [
"color_quant",
"weezl",
]
[[package]]
name = "hashbrown"
version = "0.15.5"
@@ -941,6 +1049,22 @@ dependencies = [
"icu_properties",
]
[[package]]
name = "image-webp"
version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "525e9ff3e1a4be2fbea1fdf0e98686a6d98b4d8f937e1bf7402245af1909e8c3"
dependencies = [
"byteorder-lite",
"quick-error",
]
[[package]]
name = "imagesize"
version = "0.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "edcd27d72f2f071c64249075f42e205ff93c9a4c5f6c6da53e79ed9f9832c285"
[[package]]
name = "indexmap"
version = "2.14.0"
@@ -991,6 +1115,17 @@ dependencies = [
"wasm-bindgen",
]
[[package]]
name = "kurbo"
version = "0.11.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c62026ae44756f8a599ba21140f350303d4f08dcdcc71b5ad9c9bb8128c13c62"
dependencies = [
"arrayvec",
"euclid",
"smallvec",
]
[[package]]
name = "lazy_static"
version = "1.5.0"
@@ -1018,7 +1153,7 @@ version = "0.1.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e02f3bb43d335493c96bf3fd3a321600bf6bd07ed34bc64118e9293bdffea46c"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"libc",
"plain",
"redox_syscall 0.7.4",
@@ -1092,6 +1227,15 @@ version = "2.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79"
[[package]]
name = "memmap2"
version = "0.9.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "714098028fe011992e1c3962653c96b2d578c4b4bce9036e15ff220319b1e0e3"
dependencies = [
"libc",
]
[[package]]
name = "mime"
version = "0.3.17"
@@ -1127,9 +1271,12 @@ dependencies = [
"axum",
"chrono",
"clap",
"fontdb",
"moments-core",
"moments-data",
"moments-entities",
"reqwest",
"resvg",
"serde",
"serde_json",
"tokio",
@@ -1160,6 +1307,7 @@ dependencies = [
"chrono",
"moments-core",
"moments-entities",
"percent-encoding",
"reqwest",
"serde",
"serde_json",
@@ -1307,6 +1455,12 @@ version = "2.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
[[package]]
name = "pico-args"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5be167a7af36ee22fe3115051bc51f6e6c7054c9348e28deb4f49bd6f705a315"
[[package]]
name = "pin-project-lite"
version = "0.2.17"
@@ -1346,6 +1500,19 @@ version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4596b6d070b27117e987119b4dac604f3c58cfb0b191112e24771b2faeac1a6"
[[package]]
name = "png"
version = "0.17.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "82151a2fc869e011c153adc57cf2789ccb8d9906ce52c0b39a6b5697749d7526"
dependencies = [
"bitflags 1.3.2",
"crc32fast",
"fdeflate",
"flate2",
"miniz_oxide",
]
[[package]]
name = "potential_utf"
version = "0.1.5"
@@ -1373,6 +1540,12 @@ dependencies = [
"unicode-ident",
]
[[package]]
name = "quick-error"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a993555f31e5a609f617c12db6250dedcac1b0a85076912c436e6fc9b2c8e6a3"
[[package]]
name = "quinn"
version = "0.11.9"
@@ -1508,7 +1681,7 @@ version = "0.5.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d"
dependencies = [
"bitflags",
"bitflags 2.11.1",
]
[[package]]
@@ -1517,7 +1690,7 @@ version = "0.7.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f450ad9c3b1da563fb6948a8e0fb0fb9269711c9c73d9ea1de5058c79c8d643a"
dependencies = [
"bitflags",
"bitflags 2.11.1",
]
[[package]]
@@ -1575,6 +1748,32 @@ dependencies = [
"webpki-roots 1.0.7",
]
[[package]]
name = "resvg"
version = "0.45.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a8928798c0a55e03c9ca6c4c6846f76377427d2c1e1f7e6de3c06ae57942df43"
dependencies = [
"gif",
"image-webp",
"log",
"pico-args",
"rgb",
"svgtypes",
"tiny-skia",
"usvg",
"zune-jpeg",
]
[[package]]
name = "rgb"
version = "0.8.53"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "47b34b781b31e5d73e9fbc8689c70551fd1ade9a19e3e28cfec8580a79290cc4"
dependencies = [
"bytemuck",
]
[[package]]
name = "ring"
version = "0.17.14"
@@ -1589,6 +1788,12 @@ dependencies = [
"windows-sys 0.52.0",
]
[[package]]
name = "roxmltree"
version = "0.20.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6c20b6793b5c2fa6553b250154b78d6d0db37e72700ae35fad9387a46f487c97"
[[package]]
name = "rsa"
version = "0.9.10"
@@ -1656,6 +1861,24 @@ version = "1.0.22"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d"
[[package]]
name = "rustybuzz"
version = "0.20.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fd3c7c96f8a08ee34eff8857b11b49b07d71d1c3f4e88f8a88d4c9e9f90b1702"
dependencies = [
"bitflags 2.11.1",
"bytemuck",
"core_maths",
"log",
"smallvec",
"ttf-parser",
"unicode-bidi-mirroring",
"unicode-ccc",
"unicode-properties",
"unicode-script",
]
[[package]]
name = "ryu"
version = "1.0.23"
@@ -1797,12 +2020,36 @@ version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214"
[[package]]
name = "simplecss"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7a9c6883ca9c3c7c90e888de77b7a5c849c779d25d74a1269b0218b14e8b136c"
dependencies = [
"log",
]
[[package]]
name = "siphasher"
version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649"
[[package]]
name = "slab"
version = "0.4.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
[[package]]
name = "slotmap"
version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bdd58c3c93c3d278ca835519292445cb4b0d4dc59ccfdf7ceadaab3f8aeb4038"
dependencies = [
"version_check",
]
[[package]]
name = "smallvec"
version = "1.15.1"
@@ -1937,7 +2184,7 @@ checksum = "aa003f0038df784eb8fecbbac13affe3da23b45194bd57dba231c8f48199c526"
dependencies = [
"atoi",
"base64",
"bitflags",
"bitflags 2.11.1",
"byteorder",
"bytes",
"chrono",
@@ -1980,7 +2227,7 @@ checksum = "db58fcd5a53cf07c184b154801ff91347e4c30d17a3562a635ff028ad5deda46"
dependencies = [
"atoi",
"base64",
"bitflags",
"bitflags 2.11.1",
"byteorder",
"chrono",
"crc",
@@ -2041,6 +2288,15 @@ version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596"
[[package]]
name = "strict-num"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6637bab7722d379c8b41ba849228d680cc12d0a45ba1fa2b48f2a30577a06731"
dependencies = [
"float-cmp",
]
[[package]]
name = "stringprep"
version = "0.1.5"
@@ -2064,6 +2320,16 @@ version = "2.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
[[package]]
name = "svgtypes"
version = "0.15.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68c7541fff44b35860c1a7a47a7cadf3e4a304c457b58f9870d9706ece028afc"
dependencies = [
"kurbo",
"siphasher",
]
[[package]]
name = "syn"
version = "2.0.117"
@@ -2124,6 +2390,32 @@ dependencies = [
"cfg-if",
]
[[package]]
name = "tiny-skia"
version = "0.11.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "83d13394d44dae3207b52a326c0c85a8bf87f1541f23b0d143811088497b09ab"
dependencies = [
"arrayref",
"arrayvec",
"bytemuck",
"cfg-if",
"log",
"png",
"tiny-skia-path",
]
[[package]]
name = "tiny-skia-path"
version = "0.11.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9c9e7fc0c2e86a30b117d0462aa261b72b7a99b7ebd7deb3a14ceda95c5bdc93"
dependencies = [
"arrayref",
"bytemuck",
"strict-num",
]
[[package]]
name = "tinystr"
version = "0.8.3"
@@ -2233,7 +2525,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d4e6559d53cc268e5031cd8429d05415bc4cb4aefc4aa5d6cc35fbf5b924a1f8"
dependencies = [
"async-compression",
"bitflags",
"bitflags 2.11.1",
"bytes",
"futures-core",
"futures-util",
@@ -2343,6 +2635,15 @@ version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
[[package]]
name = "ttf-parser"
version = "0.25.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d2df906b07856748fa3f6e0ad0cbaa047052d4a7dd609e231c4f72cee8c36f31"
dependencies = [
"core_maths",
]
[[package]]
name = "typenum"
version = "1.20.0"
@@ -2355,6 +2656,18 @@ version = "0.3.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5"
[[package]]
name = "unicode-bidi-mirroring"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5dfa6e8c60bb66d49db113e0125ee8711b7647b5579dc7f5f19c42357ed039fe"
[[package]]
name = "unicode-ccc"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ce61d488bcdc9bc8b5d1772c404828b17fc481c0a582b5581e95fb233aef503e"
[[package]]
name = "unicode-ident"
version = "1.0.24"
@@ -2376,6 +2689,18 @@ version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7df058c713841ad818f1dc5d3fd88063241cc61f49f5fbea4b951e8cf5a8d71d"
[[package]]
name = "unicode-script"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "383ad40bb927465ec0ce7720e033cb4ca06912855fc35db31b5755d0de75b1ee"
[[package]]
name = "unicode-vo"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b1d386ff53b415b7fe27b50bb44679e2cc4660272694b7b6f3326d8480823a94"
[[package]]
name = "untrusted"
version = "0.9.0"
@@ -2394,6 +2719,33 @@ dependencies = [
"serde",
]
[[package]]
name = "usvg"
version = "0.45.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "80be9b06fbae3b8b303400ab20778c80bbaf338f563afe567cf3c9eea17b47ef"
dependencies = [
"base64",
"data-url",
"flate2",
"fontdb",
"imagesize",
"kurbo",
"log",
"pico-args",
"roxmltree",
"rustybuzz",
"simplecss",
"siphasher",
"strict-num",
"svgtypes",
"tiny-skia-path",
"unicode-bidi",
"unicode-script",
"unicode-vo",
"xmlwriter",
]
[[package]]
name = "utf8_iter"
version = "1.0.4"
@@ -2547,6 +2899,12 @@ dependencies = [
"rustls-pki-types",
]
[[package]]
name = "weezl"
version = "0.1.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a28ac98ddc8b9274cb41bb4d9d4d5c425b6020c50c46f25559911905610b4a88"
[[package]]
name = "whoami"
version = "1.6.1"
@@ -2859,6 +3217,12 @@ version = "0.6.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4"
[[package]]
name = "xmlwriter"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec7a2a501ed189703dba8b08142f057e887dfc4b2cc4db2d343ac6376ba3e0b9"
[[package]]
name = "yoke"
version = "0.8.2"
@@ -2967,3 +3331,18 @@ name = "zmij"
version = "1.0.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa"
[[package]]
name = "zune-core"
version = "0.4.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f423a2c17029964870cfaabb1f13dfab7d092a62a29a89264f4d36990ca414a"
[[package]]
name = "zune-jpeg"
version = "0.4.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29ce2c8a9384ad323cf564b67da86e21d3cfdff87908bc1223ed5c99bc792713"
dependencies = [
"zune-core",
]

2
Cargo.toml
View File

@@ -30,6 +30,8 @@ anyhow = "1"
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json", "gzip"] }
figment = { version = "0.10", features = ["toml", "env"] }
clap = { version = "4", features = ["derive", "env"] }
resvg = "0.45"
fontdb = "0.23"
# internal
moments-entities = { path = "crates/moments-entities", version = "=0.1.0" }

6
asset/config/api.env.tmpl Normal file
View File

@@ -0,0 +1,6 @@
JOURNAL_STREAM=1
RUST_LOG=info,sqlx=warn,tower_http=info
BIND_ADDR={{BIND}}
DATABASE_URL=postgres://moments_ro@magrathea.kosherinata.internal:5432/moments?sslmode=verify-full&sslrootcert=/etc/pki/ca-trust/source/anchors/root-internal.pem&sslcert=/etc/pki/tls/misc/{{HOSTNAME}}.pem&sslkey=/etc/pki/tls/private/{{HOSTNAME}}.pem

25
asset/config/worker.env.tmpl Normal file
View File

@@ -0,0 +1,25 @@
JOURNAL_STREAM=1
RUST_LOG=info,sqlx=warn
DATABASE_URL=postgres://moments_rw@magrathea.kosherinata.internal:5432/moments?sslmode=verify-full&sslrootcert=/etc/pki/ca-trust/source/anchors/root-internal.pem&sslcert=/etc/pki/tls/misc/{{HOSTNAME}}.pem&sslkey=/etc/pki/tls/private/{{HOSTNAME}}.pem
GITHUB_USER=grenade
GITHUB_TOKEN={{GITHUB_TOKEN}}
POLL_INTERVAL_SECS=600
SEARCH_POLL_INTERVAL_SECS=86400
REPO_POLL_INTERVAL_SECS=604800
GITEA_HOST=git.lair.cafe
GITEA_USER=grenade
GITEA_TOKEN={{GITEA_TOKEN}}
GITEA_POLL_INTERVAL_SECS=600
HG_HOST=hg-edge.mozilla.org
HG_GROUPS=build,integration
HG_REPOS=mozilla-central
HG_AUTHOR_TERMS=rthijssen,grenade
HG_POLL_INTERVAL_SECS=86400
BUGZILLA_HOST=bugzilla.mozilla.org
BUGZILLA_EMAIL=rthijssen@mozilla.com
BUGZILLA_POLL_INTERVAL_SECS=86400

6
asset/firewalld/moments-api.xml.tmpl Normal file
View File

@@ -0,0 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<service>
<short>moments-api</short>
<description>moments read-only HTTP API</description>
<port protocol="tcp" port="{{API_PORT}}"/>
</service>

36
asset/manifest.yml Normal file
View File

@@ -0,0 +1,36 @@
app: moments
environments:
prod:
components:
api:
hosts: [nikola.kosherinata.internal]
config:
bind: 0.0.0.0:42424
db_role: moments_ro
db_host: magrathea.kosherinata.internal
db_port: 5432
db_name: moments
worker:
hosts: [frootmig.kosherinata.internal]
config:
db_role: moments_rw
db_host: magrathea.kosherinata.internal
db_port: 5432
db_name: moments
github_user: grenade
gitea_host: git.lair.cafe
gitea_user: grenade
hg_host: hg-edge.mozilla.org
hg_repos: build/puppet,build/tools,build/buildbot-configs
hg_author_terms: thijssen,grenade
bugzilla_host: bugzilla.mozilla.org
bugzilla_email: rthijssen@mozilla.com
secrets:
GITHUB_TOKEN: github.com/grenade/admin-token
GITEA_TOKEN: git.lair.cafe/grenade/admin-token
web:
hosts: [oolon.kosherinata.internal]
config:
server_name: rob.tn
root: /var/www/rob.tn
api_upstream: http://nikola.kosherinata.internal:42424

43
asset/nginx/site.conf.tmpl Normal file
View File

@@ -0,0 +1,43 @@
upstream moments_api {
server {{API_UPSTREAM_ADDR}} max_fails=3 fail_timeout=30s;
keepalive 8;
}
server {
server_name {{SERVER_NAME}};
listen 443 ssl;
http2 on;
ssl_certificate /etc/letsencrypt/live/{{SERVER_NAME}}/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/{{SERVER_NAME}}/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
root {{DOCROOT}};
index index.html;
location / {
try_files $uri $uri/ /index.html;
add_header Cache-Control "no-cache" always;
}
location ~* ^(?!/api/)\S+\.(js|css|woff2?|ttf|eot|svg|png|jpg|jpeg|gif|ico|webp|avif)$ {
expires 30d;
add_header Cache-Control "public, max-age=2592000, immutable";
try_files $uri =404;
}
location /api/ {
rewrite ^/api/(.*)$ /$1 break;
proxy_pass {{API_UPSTREAM_SCHEME}}://moments_api;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 30s;
proxy_connect_timeout 5s;
}
access_log /var/log/nginx/{{SERVER_NAME}}.access.log;
error_log /var/log/nginx/{{SERVER_NAME}}.error.log;
}

14
asset/sql/bootstrap-moments.sql
View File

@@ -2,9 +2,21 @@
-- Run after asset/sql/bootstrap.sql, against the moments database.
-- Idempotent — safe to re-run on every deploy.
--
-- psql -h magrathea.kosherinata.internal -U postgres -d moments \
-- (a) mTLS as `grenade`:
--
-- PGSSLMODE=verify-full \
-- PGSSLCERT=/etc/pki/tls/misc/$(hostname -f).pem \
-- PGSSLKEY=/etc/pki/tls/private/$(hostname -f).pem \
-- PGSSLROOTCERT=/etc/pki/ca-trust/source/anchors/root-internal.pem \
-- psql -h magrathea.kosherinata.internal -U grenade -d moments \
-- -f asset/sql/bootstrap-moments.sql
--
-- (b) ssh + sudo to the local postgres peer:
--
-- ssh magrathea.kosherinata.internal \
-- sudo --user postgres psql -d moments -f - \
-- < asset/sql/bootstrap-moments.sql
--
-- The schema itself is created by sqlx migrations executed by moments-api
-- on startup (which runs as moments_rw, the database owner). This file
-- only manages the read-only role's access to whatever moments_rw creates.

18
asset/sql/bootstrap.sql
View File

@@ -2,9 +2,25 @@
-- Run as a postgres superuser against the cluster's `postgres` database.
-- Idempotent — safe to re-run on every deploy.
--
-- psql -h magrathea.kosherinata.internal -U postgres -d postgres \
-- Two run modes — pick whichever fits your operator path:
--
-- (a) mTLS as the network superuser `grenade` (already mapped via pg_ident
-- on magrathea + frankie). The host cert is picked up from the standard
-- /etc/pki/tls paths via the PG* env vars:
--
-- PGSSLMODE=verify-full \
-- PGSSLCERT=/etc/pki/tls/misc/$(hostname -f).pem \
-- PGSSLKEY=/etc/pki/tls/private/$(hostname -f).pem \
-- PGSSLROOTCERT=/etc/pki/ca-trust/source/anchors/root-internal.pem \
-- psql -h magrathea.kosherinata.internal -U grenade -d postgres \
-- -f asset/sql/bootstrap.sql
--
-- (b) ssh to the db host and run as the local `postgres` peer:
--
-- ssh magrathea.kosherinata.internal \
-- sudo --user postgres psql -d postgres -f - \
-- < asset/sql/bootstrap.sql
--
-- After this completes, run asset/sql/bootstrap-moments.sql against the
-- newly created `moments` database to apply the in-database grants.

6
asset/systemd/moments-api-cert-reload.service Normal file
View File

@@ -0,0 +1,6 @@
[Unit]
Description=Restart moments-api on host cert change
[Service]
Type=oneshot
ExecStart=/bin/systemctl restart moments-api.service

9
asset/systemd/moments-api-cert.path Normal file
View File

@@ -0,0 +1,9 @@
[Unit]
Description=Watch host cert for moments-api
[Path]
PathChanged=/etc/pki/tls/misc/{{HOSTNAME}}.pem
Unit=moments-api-cert-reload.service
[Install]
WantedBy=multi-user.target

32
asset/systemd/moments-api.service Normal file
View File

@@ -0,0 +1,32 @@
[Unit]
Description=moments read-only HTTP API
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=moments
Group=moments
EnvironmentFile=/etc/moments/api.env
ExecStart=/usr/local/bin/moments-api
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true
SystemCallArchitectures=native
ReadWritePaths=/var/lib/moments
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
[Install]
WantedBy=multi-user.target

6
asset/systemd/moments-worker-cert-reload.service Normal file
View File

@@ -0,0 +1,6 @@
[Unit]
Description=Restart moments-worker on host cert change
[Service]
Type=oneshot
ExecStart=/bin/systemctl restart moments-worker.service

9
asset/systemd/moments-worker-cert.path Normal file
View File

@@ -0,0 +1,9 @@
[Unit]
Description=Watch host cert for moments-worker
[Path]
PathChanged=/etc/pki/tls/misc/{{HOSTNAME}}.pem
Unit=moments-worker-cert-reload.service
[Install]
WantedBy=multi-user.target

32
asset/systemd/moments-worker.service Normal file
View File

@@ -0,0 +1,32 @@
[Unit]
Description=moments ingestion worker
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=moments
Group=moments
EnvironmentFile=/etc/moments/worker.env
ExecStart=/usr/local/bin/moments-worker
Restart=on-failure
RestartSec=10s
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true
SystemCallArchitectures=native
ReadWritePaths=/var/lib/moments
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
[Install]
WantedBy=multi-user.target

2
asset/systemd/moments.sysusers.conf Normal file
View File

@@ -0,0 +1,2 @@
#Type Name ID GECOS Home directory Shell
u moments - "moments service account" /var/lib/moments /usr/sbin/nologin

3
crates/moments-api/Cargo.toml
View File

@@ -20,3 +20,6 @@ serde.workspace = true
serde_json.workspace = true
chrono.workspace = true
clap.workspace = true
reqwest.workspace = true
resvg.workspace = true
fontdb.workspace = true

379
crates/moments-api/src/main.rs
View File

@@ -1,17 +1,17 @@
use std::{net::SocketAddr, sync::Arc};
use std::{net::SocketAddr, sync::Arc, time::Duration};
use axum::{
Json, Router,
extract::{Query, State},
extract::{Path, Query, State},
http::StatusCode,
response::IntoResponse,
routing::get,
};
use chrono::{DateTime, Utc};
use chrono::{DateTime, Datelike, NaiveDate, Utc};
use clap::Parser;
use moments_core::EventReader;
use moments_core::{EventReader, reshape};
use moments_data::PgStore;
use moments_entities::{Event, EventQuery, Source, SourceSummary};
use moments_entities::{DailyCount, EventQuery, HourlyAvg, LanguageDailyCount, ProjectSummary, RepoLanguage, Source, SourceSummary, TimelineItem};
use serde::Deserialize;
use tower_http::{cors::CorsLayer, trace::TraceLayer};
use tracing::info;
@@ -29,6 +29,7 @@ struct Args {
#[derive(Clone)]
struct AppState {
store: Arc<PgStore>,
http: reqwest::Client,
}
#[tokio::main]
@@ -36,16 +37,31 @@ async fn main() -> anyhow::Result<()> {
init_tracing();
let args = Args::parse();
// The api connects as moments_ro and never writes — migrations are owned
// by moments-worker, which is the database owner (moments_rw). Running
// migrations from here would fail with `permission denied for schema
// public`. The worker must have run at least once before the api accepts
// traffic; in deploy this is ordered via systemd dependencies (§3).
let store = PgStore::connect(&args.database_url).await?;
store.migrate().await?;
let http = reqwest::Client::builder()
.timeout(Duration::from_secs(15))
.build()?;
let state = AppState {
store: Arc::new(store),
http,
};
let app = Router::new()
.route("/v1/healthz", get(healthz))
.route("/v1/events", get(list_events))
.route("/v1/sources", get(list_sources))
.route("/v1/projects", get(list_projects))
.route("/v1/activity/daily", get(daily_counts))
.route("/v1/activity/hourly", get(hourly_avgs))
.route("/v1/languages/daily", get(language_daily_counts))
.route("/v1/languages/repos", get(repo_languages))
.route("/v1/forge/{source}/{*rest}", get(forge_proxy))
.route("/v1/og/contributions.png", get(og_contributions))
.with_state(state)
.layer(TraceLayer::new_for_http())
.layer(CorsLayer::permissive());
@@ -77,13 +93,15 @@ struct EventsQueryParams {
to: Option<DateTime<Utc>>,
/// Comma-separated list, e.g. `source=github,gitea`.
source: Option<String>,
/// Filter to a specific repo, e.g. `repo=grenade/moments`.
repo: Option<String>,
limit: Option<u32>,
}
async fn list_events(
State(state): State<AppState>,
Query(params): Query<EventsQueryParams>,
) -> Result<Json<Vec<Event>>, ApiError> {
) -> Result<Json<Vec<TimelineItem>>, ApiError> {
let sources = params
.source
.as_deref()
@@ -96,20 +114,359 @@ async fn list_events(
from: params.from,
to: params.to,
sources,
repo: params.repo,
// Public timeline only — private events stay in the DB but are never
// surfaced. A future authenticated path can flip this.
include_private: false,
limit,
};
let events = state.store.list_events(&query).await.map_err(internal)?;
Ok(Json(events))
let items: Vec<TimelineItem> = events.iter().map(reshape).collect();
Ok(Json(items))
}
async fn list_sources(
State(state): State<AppState>,
) -> Result<Json<Vec<SourceSummary>>, ApiError> {
let summaries = state.store.source_summaries().await.map_err(internal)?;
let summaries = state
.store
.source_summaries(/* include_private */ true)
.await
.map_err(internal)?;
Ok(Json(summaries))
}
async fn list_projects(
State(state): State<AppState>,
) -> Result<Json<Vec<ProjectSummary>>, ApiError> {
let projects = state.store.list_projects().await.map_err(internal)?;
Ok(Json(projects))
}
#[derive(Debug, Deserialize)]
struct DailyCountsParams {
from: Option<NaiveDate>,
to: Option<NaiveDate>,
}
async fn daily_counts(
State(state): State<AppState>,
Query(params): Query<DailyCountsParams>,
) -> Result<Json<Vec<DailyCount>>, ApiError> {
let to = params.to.unwrap_or_else(|| Utc::now().date_naive());
let from = params.from.unwrap_or_else(|| to - chrono::Duration::days(365));
let counts = state.store.daily_counts(from, to, /* include_private */ true).await.map_err(internal)?;
Ok(Json(counts))
}
async fn language_daily_counts(
State(state): State<AppState>,
Query(params): Query<DailyCountsParams>,
) -> Result<Json<Vec<LanguageDailyCount>>, ApiError> {
let to = params.to.unwrap_or_else(|| Utc::now().date_naive());
let from = params.from.unwrap_or_else(|| to - chrono::Duration::days(365));
let counts = state.store.language_daily_counts(from, to, /* include_private */ true).await.map_err(internal)?;
Ok(Json(counts))
}
#[derive(Debug, Deserialize)]
struct HourlyAvgsParams {
from: Option<NaiveDate>,
to: Option<NaiveDate>,
/// IANA timezone name (e.g. "Europe/Helsinki"). Defaults to UTC.
/// Hour buckets are computed in this zone so the chart matches the
/// clock the user sees.
tz: Option<String>,
}
async fn hourly_avgs(
State(state): State<AppState>,
Query(params): Query<HourlyAvgsParams>,
) -> Result<Json<Vec<HourlyAvg>>, ApiError> {
let to = params.to.unwrap_or_else(|| Utc::now().date_naive());
let from = params.from.unwrap_or_else(|| to - chrono::Duration::days(365));
let tz = params.tz.as_deref().unwrap_or("UTC");
// Validate the tz string before handing it to postgres — a bad name
// here would surface as an opaque 500 from the DB. chrono-tz would do
// it for free but we don't depend on it; instead reject obvious shell
// injection vectors (the value is bound, not interpolated, so this is
// belt-and-braces).
if tz.len() > 64 || tz.chars().any(|c| !(c.is_ascii_alphanumeric() || matches!(c, '/' | '_' | '+' | '-'))) {
return Err(ApiError {
status: StatusCode::BAD_REQUEST,
message: "invalid tz".into(),
});
}
let avgs = state.store.hourly_avgs(from, to, tz, /* include_private */ true).await.map_err(internal)?;
Ok(Json(avgs))
}
async fn repo_languages(
State(state): State<AppState>,
) -> Result<Json<Vec<RepoLanguage>>, ApiError> {
let langs = state.store.repo_languages().await.map_err(internal)?;
Ok(Json(langs))
}
async fn og_contributions(
State(state): State<AppState>,
) -> Result<impl IntoResponse, ApiError> {
// Get date range from source summaries
let summaries = state
.store
.source_summaries(/* include_private */ true)
.await
.map_err(internal)?;
let earliest = summaries
.iter()
.filter_map(|s| s.earliest)
.min()
.unwrap_or_else(Utc::now)
.date_naive();
let today = Utc::now().date_naive();
let counts = state
.store
.daily_counts(earliest, today, /* include_private */ true)
.await
.map_err(internal)?;
let projects = state.store.list_projects().await.map_err(internal)?;
let repo_count = projects.len();
let png = render_contributions_png(&counts, earliest, today, repo_count).map_err(|e| ApiError {
status: StatusCode::INTERNAL_SERVER_ERROR,
message: e,
})?;
Ok((
StatusCode::OK,
[
(axum::http::header::CONTENT_TYPE, "image/png"),
(axum::http::header::CACHE_CONTROL, "public, max-age=3600"),
],
png,
))
}
fn render_contributions_png(
counts: &[DailyCount],
from: NaiveDate,
to: NaiveDate,
repo_count: usize,
) -> Result<Vec<u8>, String> {
use std::collections::HashMap;
let count_map: HashMap<NaiveDate, i64> = counts.iter().map(|d| (d.date, d.count)).collect();
// OG image canvas: 1200x630
let og_w = 1200_f64;
let og_h = 630_f64;
let padding = 40_f64;
let bg = "#2c3e50";
let year_label_w = 50_f64;
let max_cols = 53;
// Scale cell size to fill available width
let avail_w = og_w - 2.0 * padding - year_label_w;
let step = (avail_w / max_cols as f64).floor();
let gap = (step * 0.17).round();
let cell = step - gap;
let radius = cell / 2.0;
let colors = ["rgba(255,255,255,0.05)", "#0e4429", "#006d32", "#26a641", "#39d353"];
// Build weekly data per year
struct YearRow {
year: i32,
weeks: Vec<(NaiveDate, NaiveDate, i64)>, // start, end, count
}
let start_year = from.year();
let end_year = to.year();
let mut rows: Vec<YearRow> = Vec::new();
for yr in start_year..=end_year {
let year_start = NaiveDate::from_ymd_opt(yr, 1, 1).unwrap();
let year_end = if yr == end_year {
to
} else {
NaiveDate::from_ymd_opt(yr, 12, 31).unwrap()
};
let offset = year_start.weekday().num_days_from_sunday();
let mut cursor = year_start - chrono::Duration::days(offset as i64);
let mut weeks = Vec::new();
while cursor <= year_end {
let week_start = cursor;
let mut week_count = 0i64;
for _ in 0..7 {
week_count += count_map.get(&cursor).copied().unwrap_or(0);
cursor += chrono::Duration::days(1);
}
let week_end = cursor - chrono::Duration::days(1);
weeks.push((week_start, week_end, week_count));
}
rows.push(YearRow { year: yr, weeks });
}
// Quantile thresholds
let mut non_zero: Vec<i64> = rows
.iter()
.flat_map(|r| r.weeks.iter().map(|w| w.2))
.filter(|&c| c > 0)
.collect();
non_zero.sort();
let thresholds = if non_zero.is_empty() {
[1i64, 2, 3]
} else {
let p = |pct: f64| non_zero[(pct * non_zero.len() as f64).min(non_zero.len() as f64 - 1.0) as usize];
[p(0.25), p(0.5), p(0.75)]
};
let color_for = |count: i64| -> &str {
if count == 0 { colors[0] }
else if count <= thresholds[0] { colors[1] }
else if count <= thresholds[1] { colors[2] }
else if count <= thresholds[2] { colors[3] }
else { colors[4] }
};
let n_rows = rows.len();
let graph_h = (n_rows as f64) * step;
let total: i64 = counts.iter().map(|d| d.count).sum();
let repo_text = if repo_count > 0 {
format!(" in {repo_count} repositories")
} else {
String::new()
};
// Layout: headline at top, graph vertically centered in remaining space
let offset_x = padding;
let headline_y = padding + 36.0;
let subtitle_y = headline_y + 28.0;
let graph_top = subtitle_y + 16.0;
let avail_graph_h = og_h - graph_top - padding;
let graph_y = graph_top + (avail_graph_h - graph_h).max(0.0) / 2.0;
let mut svg = format!(
r#"<svg xmlns="http://www.w3.org/2000/svg" width="{og_w}" height="{og_h}" viewBox="0 0 {og_w} {og_h}"><rect width="100%" height="100%" fill="{bg}"/>"#,
);
// Headline
svg.push_str(&format!(
r##"<text x="{x}" y="{y}" fill="#ecf0f1" font-family="sans-serif" font-size="36" font-weight="bold">rob thijssen</text>"##,
x = offset_x + year_label_w,
y = headline_y,
));
// Subtitle
svg.push_str(&format!(
r##"<text x="{x}" y="{y}" fill="#ecf0f1" font-family="sans-serif" font-size="16" opacity="0.6">{total} contributions since {from}{repo_text}</text>"##,
x = offset_x + year_label_w,
y = subtitle_y,
));
let label_font_size = (step * 0.7).round().max(8.0).min(14.0);
for (row_idx, row) in rows.iter().enumerate() {
let y_base = graph_y + (row_idx as f64) * step;
svg.push_str(&format!(
r##"<text x="{x}" y="{y}" text-anchor="end" dominant-baseline="central" fill="#ecf0f1" font-family="sans-serif" font-size="{fs}" opacity="0.6">{yr}</text>"##,
x = offset_x + year_label_w - 6.0,
y = y_base + radius,
fs = label_font_size,
yr = row.year,
));
for (col, (_, _, count)) in row.weeks.iter().enumerate() {
let cx = offset_x + year_label_w + (col as f64) * step + radius;
let cy = y_base + radius;
let fill = color_for(*count);
svg.push_str(&format!(
r#"<circle cx="{cx}" cy="{cy}" r="{r}" fill="{fill}"/>"#,
r = radius - 1.0,
));
}
}
svg.push_str("</svg>");
// Rasterize at 1200x630
let mut fontdb = fontdb::Database::new();
fontdb.load_system_fonts();
let mut opts = resvg::usvg::Options::default();
opts.fontdb = std::sync::Arc::new(fontdb);
opts.font_family = "Noto Sans".to_owned();
let tree = resvg::usvg::Tree::from_str(&svg, &opts)
.map_err(|e| format!("svg parse: {e}"))?;
let mut pixmap =
resvg::tiny_skia::Pixmap::new(og_w as u32, og_h as u32).ok_or_else(|| "pixmap alloc failed".to_string())?;
resvg::render(&tree, resvg::tiny_skia::Transform::default(), &mut pixmap.as_mut());
pixmap
.encode_png()
.map_err(|e| format!("png encode: {e}"))
}
/// Allowlisted forge hosts that the proxy may contact.
const ALLOWED_HOSTS: &[&str] = &["api.github.com", "git.lair.cafe"];
#[derive(Debug, Deserialize)]
struct ForgeProxyParams {
host: Option<String>,
}
/// Proxy requests to forge APIs to avoid CORS issues.
/// `GET /v1/forge/{source}/{path}?host=git.lair.cafe`
async fn forge_proxy(
State(state): State<AppState>,
Path((source, rest)): Path<(String, String)>,
Query(params): Query<ForgeProxyParams>,
) -> Result<impl IntoResponse, ApiError> {
let (base, api_prefix) = match source.as_str() {
"github" => ("https://api.github.com".to_string(), ""),
"gitea" => {
let host = params.host.as_deref().unwrap_or("git.lair.cafe");
if !ALLOWED_HOSTS.contains(&host) {
return Err(ApiError::bad_request(format!("host not allowed: {host}")));
}
(format!("https://{host}"), "/api/v1")
}
_ => return Err(ApiError::bad_request(format!("unsupported source: {source}"))),
};
let url = format!("{base}{api_prefix}/{rest}");
let resp = state
.http
.get(&url)
.header("Accept", "application/json")
.header("User-Agent", "moments-api")
.send()
.await
.map_err(|e| {
tracing::warn!(url = %url, error = %e, "forge proxy request failed");
ApiError {
status: StatusCode::BAD_GATEWAY,
message: e.to_string(),
}
})?;
let status = StatusCode::from_u16(resp.status().as_u16()).unwrap_or(StatusCode::BAD_GATEWAY);
let body = resp.bytes().await.map_err(|e| ApiError {
status: StatusCode::BAD_GATEWAY,
message: e.to_string(),
})?;
Ok((
status,
[(axum::http::header::CONTENT_TYPE, "application/json")],
body,
))
}
fn parse_sources(raw: &str) -> Result<Vec<Source>, ApiError> {
raw.split(',')
.map(str::trim)
@@ -133,9 +490,11 @@ impl ApiError {
}
fn internal<E: std::fmt::Display>(e: E) -> ApiError {
let message = e.to_string();
tracing::error!(error = %message, "internal handler error");
ApiError {
status: StatusCode::INTERNAL_SERVER_ERROR,
message: e.to_string(),
message,
}
}

13
crates/moments-core/src/lib.rs
View File

@@ -1,9 +1,12 @@
pub mod presentation;
pub mod sources;
pub use presentation::reshape;
pub use sources::{EventSource, PollerState, PollerStateStore, SourceError, run_poller};
use async_trait::async_trait;
use moments_entities::{Event, EventQuery, SourceSummary};
use chrono::NaiveDate;
use moments_entities::{DailyCount, Event, EventQuery, HourlyAvg, LanguageDailyCount, ProjectSummary, RepoLanguage, SourceSummary};
#[derive(Debug, thiserror::Error)]
pub enum StoreError {
@@ -15,11 +18,17 @@ pub enum StoreError {
#[async_trait]
pub trait EventReader: Send + Sync {
async fn list_events(&self, query: &EventQuery) -> Result<Vec<Event>, StoreError>;
async fn source_summaries(&self) -> Result<Vec<SourceSummary>, StoreError>;
async fn source_summaries(&self, include_private: bool) -> Result<Vec<SourceSummary>, StoreError>;
async fn list_projects(&self) -> Result<Vec<ProjectSummary>, StoreError>;
async fn daily_counts(&self, from: NaiveDate, to: NaiveDate, include_private: bool) -> Result<Vec<DailyCount>, StoreError>;
async fn language_daily_counts(&self, from: NaiveDate, to: NaiveDate, include_private: bool) -> Result<Vec<LanguageDailyCount>, StoreError>;
async fn hourly_avgs(&self, from: NaiveDate, to: NaiveDate, tz: &str, include_private: bool) -> Result<Vec<HourlyAvg>, StoreError>;
async fn repo_languages(&self) -> Result<Vec<RepoLanguage>, StoreError>;
}
/// Write-side port consumed by `moments-worker`. Idempotent upserts on `id`.
#[async_trait]
pub trait EventWriter: Send + Sync {
async fn upsert_events(&self, events: &[Event]) -> Result<usize, StoreError>;
async fn upsert_repo_languages(&self, languages: &[RepoLanguage]) -> Result<usize, StoreError>;
}

20
crates/moments-core/src/presentation.rs Normal file
View File

@@ -0,0 +1,20 @@
//! Reshape raw stored events into the presentation shape consumed by the UI.
//!
//! Storage holds the upstream payload verbatim; transformation lives here so
//! the rendering can evolve without re-fetching upstream data.
use moments_entities::{Event, Source, TimelineItem};
mod bugzilla;
mod gitea;
mod github;
mod hg;
pub fn reshape(event: &Event) -> TimelineItem {
match event.source {
Source::Github => github::reshape(event),
Source::Gitea => gitea::reshape(event),
Source::Hg => hg::reshape(event),
Source::Bugzilla => bugzilla::reshape(event),
}
}

79
crates/moments-core/src/presentation/bugzilla.rs Normal file
View File

@@ -0,0 +1,79 @@
use moments_entities::{Event, Source, TimelineIcon, TimelineItem, TitleSegment};
use serde_json::Value;
const FALLBACK_HOST: &str = "bugzilla.mozilla.org";
pub(crate) fn reshape(event: &Event) -> TimelineItem {
let p = &event.payload;
let host = p
.get("_host")
.and_then(Value::as_str)
.unwrap_or(FALLBACK_HOST);
let id = p.get("id").and_then(Value::as_i64).unwrap_or(0);
let summary = p.get("summary").and_then(Value::as_str).unwrap_or("");
let product = p.get("product").and_then(Value::as_str);
let mut title = vec![
TitleSegment::text("filed bug "),
TitleSegment::link(
format!("#{id}"),
format!("https://{host}/show_bug.cgi?id={id}"),
),
];
if let Some(prod) = product {
title.push(TitleSegment::text(format!(" in {prod}")));
}
let subtitle = (!summary.is_empty()).then(|| vec![TitleSegment::text(summary.to_string())]);
TimelineItem {
id: event.id.clone(),
source: Source::Bugzilla,
action: event.action.clone(),
occurred_at: event.occurred_at,
icon: TimelineIcon::Bug,
title,
subtitle,
body: None,
}
}
#[cfg(test)]
mod tests {
use super::*;
use chrono::{TimeZone, Utc};
use serde_json::json;
#[test]
fn reshape_bug_create() {
let raw = json!({
"_host": "bugzilla.mozilla.org",
"id": 1158879,
"summary": "Commit Access (Level 1) for Rob Thijssen",
"product": "mozilla.org"
});
let event = Event {
id: "bugzilla:1158879".into(),
source: Source::Bugzilla,
action: "BugCreate".into(),
occurred_at: Utc.with_ymd_and_hms(2015, 4, 27, 16, 29, 59).unwrap(),
public: true,
payload: raw,
};
let item = reshape(&event);
assert_eq!(item.icon, TimelineIcon::Bug);
let r: String = item
.title
.iter()
.map(|s| match s {
TitleSegment::Text { text } => text.clone(),
TitleSegment::Link { text, .. } => text.clone(),
})
.collect();
assert!(r.contains("filed bug #1158879 in mozilla.org"), "got: {r}");
assert_eq!(
item.subtitle.unwrap(),
vec![TitleSegment::text("Commit Access (Level 1) for Rob Thijssen")]
);
}
}

496
crates/moments-core/src/presentation/gitea.rs Normal file
View File

@@ -0,0 +1,496 @@
use moments_entities::{
CommitSummary, Event, Source, TimelineBody, TimelineIcon, TimelineItem, TitleSegment,
};
use serde_json::Value;
const FALLBACK_HOST: &str = "git.lair.cafe";
pub(crate) fn reshape(event: &Event) -> TimelineItem {
let p = &event.payload;
let host = p
.get("_host")
.and_then(Value::as_str)
.unwrap_or(FALLBACK_HOST);
let repo = p
.get("repo")
.and_then(|r| r.get("full_name"))
.and_then(Value::as_str);
let actor = p
.get("act_user")
.and_then(|u| u.get("login"))
.and_then(Value::as_str);
let ref_name = p.get("ref_name").and_then(Value::as_str);
let content = p.get("content").and_then(Value::as_str);
let comment = p.get("comment");
let (icon, title, subtitle, body) = match event.action.as_str() {
"commit_repo" => commit_repo(host, repo, ref_name, content),
"push_tag" => push_tag(host, repo, ref_name),
"create_repo" => create_repo(host, repo),
"rename_repo" => rename_repo(host, repo, content),
"transfer_repo" => transfer_repo(host, repo, content),
"fork_repo" => fork_repo(host, repo),
"delete_branch" => delete_branch(host, repo, ref_name),
"delete_tag" => delete_tag(host, repo, ref_name),
"star_repo" => star_repo(host, repo),
"create_issue" => issue_action("opened", TimelineIcon::Issue, host, repo, content),
"close_issue" => issue_action("closed", TimelineIcon::Issue, host, repo, content),
"reopen_issue" => issue_action("reopened", TimelineIcon::Issue, host, repo, content),
"comment_issue" => comment_on_issue(host, repo, content, comment),
"create_pull_request" => {
pr_action("opened", TimelineIcon::PullRequest, host, repo, content)
}
"close_pull_request" => {
pr_action("closed", TimelineIcon::PullRequest, host, repo, content)
}
"reopen_pull_request" => {
pr_action("reopened", TimelineIcon::PullRequest, host, repo, content)
}
"merge_pull_request" | "auto_merge_pull_request" => {
pr_action("merged", TimelineIcon::GitMerge, host, repo, content)
}
"comment_pull" => comment_on_pr(host, repo, content, comment),
"approve_pull_request" => {
pr_action("approved", TimelineIcon::PullRequest, host, repo, content)
}
"reject_pull_request" => {
pr_action(
"requested changes on",
TimelineIcon::PullRequest,
host,
repo,
content,
)
}
"publish_release" => publish_release(host, repo, content),
_ => fallback(host, repo, &event.action),
};
let title = if let Some(actor_login) = actor {
let mut segs = Vec::with_capacity(title.len() + 2);
segs.push(TitleSegment::link(
actor_login.to_string(),
format!("https://{host}/{actor_login}"),
));
segs.push(TitleSegment::text(" "));
segs.extend(title);
segs
} else {
title
};
TimelineItem {
id: event.id.clone(),
source: Source::Gitea,
action: event.action.clone(),
occurred_at: event.occurred_at,
icon,
title,
subtitle,
body,
}
}
type Reshaped = (
TimelineIcon,
Vec<TitleSegment>,
Option<Vec<TitleSegment>>,
Option<TimelineBody>,
);
fn repo_link(host: &str, repo: &str) -> TitleSegment {
TitleSegment::link(repo.to_string(), format!("https://{host}/{repo}"))
}
fn commit_url(host: &str, repo: &str, sha: &str) -> String {
format!("https://{host}/{repo}/commit/{sha}")
}
fn issue_url(host: &str, repo: &str, index: i64) -> String {
format!("https://{host}/{repo}/issues/{index}")
}
fn pr_url(host: &str, repo: &str, index: i64) -> String {
format!("https://{host}/{repo}/pulls/{index}")
}
fn ref_branch(r: &str) -> &str {
r.strip_prefix("refs/heads/").unwrap_or(r)
}
fn ref_tag(r: &str) -> &str {
r.strip_prefix("refs/tags/").unwrap_or(r)
}
/// Parse `<index>|<title>` content used by issue / PR / release events.
fn parse_pipe_content(content: Option<&str>) -> Option<(i64, &str)> {
let s = content?;
let (idx_str, title) = s.split_once('|')?;
let idx: i64 = idx_str.parse().ok()?;
Some((idx, title))
}
fn commit_repo(
host: &str,
repo: Option<&str>,
ref_name: Option<&str>,
content: Option<&str>,
) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let branch = ref_name.map(ref_branch).unwrap_or("");
// content is JSON-encoded { Commits, HeadCommit, CompareURL, Len }.
let parsed: Option<Value> = content.and_then(|s| serde_json::from_str(s).ok());
let commits: Vec<CommitSummary> = parsed
.as_ref()
.and_then(|v| v.get("Commits"))
.and_then(Value::as_array)
.map(|arr| {
arr.iter()
.filter_map(|c| {
let sha = c.get("Sha1").and_then(Value::as_str)?;
let message = c
.get("Message")
.and_then(Value::as_str)
.unwrap_or("")
.lines()
.next()
.unwrap_or("")
.to_string();
let author = c
.get("AuthorName")
.and_then(Value::as_str)
.map(str::to_string);
Some(CommitSummary {
short_sha: sha.chars().take(7).collect(),
sha: sha.to_string(),
message,
url: commit_url(host, repo, sha),
author,
})
})
.collect()
})
.unwrap_or_default();
let count = parsed
.as_ref()
.and_then(|v| v.get("Len"))
.and_then(Value::as_i64)
.unwrap_or(commits.len() as i64);
let title = if count > 0 {
let plural = if count == 1 { "" } else { "s" };
vec![
TitleSegment::text(format!("pushed {count} commit{plural} to ")),
repo_link(host, repo),
TitleSegment::text(format!(":{branch}")),
]
} else {
vec![
TitleSegment::text("pushed to "),
repo_link(host, repo),
TitleSegment::text(format!(":{branch}")),
]
};
let body = (!commits.is_empty()).then_some(TimelineBody::Commits { commits });
(TimelineIcon::GitPush, title, None, body)
}
fn push_tag(host: &str, repo: Option<&str>, ref_name: Option<&str>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let tag = ref_name.map(ref_tag).unwrap_or("");
let title = vec![
TitleSegment::text(format!("tagged {tag} in ")),
repo_link(host, repo),
];
(TimelineIcon::Release, title, None, None)
}
fn create_repo(host: &str, repo: Option<&str>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let title = vec![TitleSegment::text("created "), repo_link(host, repo)];
(TimelineIcon::GitBranchCreate, title, None, None)
}
fn rename_repo(host: &str, repo: Option<&str>, content: Option<&str>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let mut title = vec![TitleSegment::text("renamed ")];
if let Some(old) = content {
title.push(TitleSegment::text(format!("{old} → ")));
}
title.push(repo_link(host, repo));
(TimelineIcon::Generic, title, None, None)
}
fn transfer_repo(host: &str, repo: Option<&str>, content: Option<&str>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let mut title = vec![TitleSegment::text("transferred ")];
if let Some(prev) = content {
title.push(TitleSegment::text(format!("{prev} to ")));
} else {
title.push(TitleSegment::text("to "));
}
title.push(repo_link(host, repo));
(TimelineIcon::Generic, title, None, None)
}
fn fork_repo(host: &str, repo: Option<&str>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let title = vec![TitleSegment::text("forked "), repo_link(host, repo)];
(TimelineIcon::GitFork, title, None, None)
}
fn delete_branch(host: &str, repo: Option<&str>, ref_name: Option<&str>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let branch = ref_name.map(ref_branch).unwrap_or("");
let title = vec![
TitleSegment::text(format!("deleted branch {branch} in ")),
repo_link(host, repo),
];
(TimelineIcon::GitBranchDelete, title, None, None)
}
fn delete_tag(host: &str, repo: Option<&str>, ref_name: Option<&str>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let tag = ref_name.map(ref_tag).unwrap_or("");
let title = vec![
TitleSegment::text(format!("deleted tag {tag} in ")),
repo_link(host, repo),
];
(TimelineIcon::GitBranchDelete, title, None, None)
}
fn star_repo(host: &str, repo: Option<&str>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let title = vec![TitleSegment::text("starred "), repo_link(host, repo)];
(TimelineIcon::Star, title, None, None)
}
fn issue_action(
verb: &str,
icon: TimelineIcon,
host: &str,
repo: Option<&str>,
content: Option<&str>,
) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let (idx, issue_title) = parse_pipe_content(content).unwrap_or((0, ""));
let title = vec![
TitleSegment::text(format!("{verb} issue ")),
TitleSegment::link(format!("#{idx}"), issue_url(host, repo, idx)),
TitleSegment::text(" in "),
repo_link(host, repo),
];
let subtitle =
(!issue_title.is_empty()).then(|| vec![TitleSegment::text(issue_title.to_string())]);
(icon, title, subtitle, None)
}
fn pr_action(
verb: &str,
icon: TimelineIcon,
host: &str,
repo: Option<&str>,
content: Option<&str>,
) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let (idx, pr_title) = parse_pipe_content(content).unwrap_or((0, ""));
let title = vec![
TitleSegment::text(format!("{verb} pull request ")),
TitleSegment::link(format!("#{idx}"), pr_url(host, repo, idx)),
TitleSegment::text(" in "),
repo_link(host, repo),
];
let subtitle =
(!pr_title.is_empty()).then(|| vec![TitleSegment::text(pr_title.to_string())]);
(icon, title, subtitle, None)
}
fn comment_on_issue(
host: &str,
repo: Option<&str>,
content: Option<&str>,
comment: Option<&Value>,
) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let (idx, issue_title) = parse_pipe_content(content).unwrap_or((0, ""));
let body_text = comment
.and_then(|c| c.get("body"))
.and_then(Value::as_str)
.unwrap_or("");
let title = vec![
TitleSegment::text("commented on "),
TitleSegment::link(format!("#{idx}"), issue_url(host, repo, idx)),
TitleSegment::text(" in "),
repo_link(host, repo),
];
let subtitle =
(!issue_title.is_empty()).then(|| vec![TitleSegment::text(issue_title.to_string())]);
let body = (!body_text.is_empty()).then(|| TimelineBody::Markdown {
text: body_text.to_string(),
});
(TimelineIcon::Comment, title, subtitle, body)
}
fn comment_on_pr(
host: &str,
repo: Option<&str>,
content: Option<&str>,
comment: Option<&Value>,
) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let (idx, pr_title) = parse_pipe_content(content).unwrap_or((0, ""));
let body_text = comment
.and_then(|c| c.get("body"))
.and_then(Value::as_str)
.unwrap_or("");
let title = vec![
TitleSegment::text("commented on "),
TitleSegment::link(format!("#{idx}"), pr_url(host, repo, idx)),
TitleSegment::text(" in "),
repo_link(host, repo),
];
let subtitle =
(!pr_title.is_empty()).then(|| vec![TitleSegment::text(pr_title.to_string())]);
let body = (!body_text.is_empty()).then(|| TimelineBody::Markdown {
text: body_text.to_string(),
});
(TimelineIcon::Comment, title, subtitle, body)
}
fn publish_release(host: &str, repo: Option<&str>, content: Option<&str>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let name = content.unwrap_or("");
let title = if name.is_empty() {
vec![TitleSegment::text("published a release in "), repo_link(host, repo)]
} else {
vec![
TitleSegment::text(format!("released {name} in ")),
repo_link(host, repo),
]
};
(TimelineIcon::Release, title, None, None)
}
fn fallback(host: &str, repo: Option<&str>, action: &str) -> Reshaped {
let title = match repo {
Some(r) => vec![
TitleSegment::text(format!("{action} on ")),
repo_link(host, r),
],
None => vec![TitleSegment::text(action.to_string())],
};
(TimelineIcon::Generic, title, None, None)
}
#[cfg(test)]
mod tests {
use super::*;
use chrono::{TimeZone, Utc};
use serde_json::json;
fn ev(action: &str, payload: Value) -> Event {
Event {
id: "gitea:1".into(),
source: Source::Gitea,
action: action.into(),
occurred_at: Utc.with_ymd_and_hms(2026, 5, 3, 16, 37, 45).unwrap(),
public: true,
payload,
}
}
fn render(item: &TimelineItem) -> String {
item.title
.iter()
.map(|s| match s {
TitleSegment::Text { text } => text.clone(),
TitleSegment::Link { text, .. } => text.clone(),
})
.collect()
}
#[test]
fn commit_repo_with_commits_body() {
let raw = json!({
"_host": "git.lair.cafe",
"act_user": { "login": "grenade" },
"repo": { "full_name": "grenade/moments" },
"ref_name": "refs/heads/main",
"content": "{\"Commits\":[{\"Sha1\":\"abcdef1234\",\"Message\":\"first\",\"AuthorName\":\"rob\"}],\"Len\":1}"
});
let item = reshape(&ev("commit_repo", raw));
assert_eq!(item.icon, TimelineIcon::GitPush);
let r = render(&item);
assert!(
r.contains("pushed 1 commit to grenade/moments:main"),
"got: {r}"
);
match item.body.unwrap() {
TimelineBody::Commits { commits } => {
assert_eq!(commits.len(), 1);
assert_eq!(commits[0].short_sha, "abcdef1");
assert_eq!(
commits[0].url,
"https://git.lair.cafe/grenade/moments/commit/abcdef1234"
);
}
_ => panic!("expected Commits body"),
}
}
#[test]
fn create_issue_uses_pipe_content() {
let raw = json!({
"_host": "git.lair.cafe",
"act_user": { "login": "grenade" },
"repo": { "full_name": "grenade/moments" },
"content": "1|implement per-repo enumeration for full commit history"
});
let item = reshape(&ev("create_issue", raw));
assert_eq!(item.icon, TimelineIcon::Issue);
let r = render(&item);
assert!(
r.contains("opened issue #1 in grenade/moments"),
"got: {r}"
);
assert_eq!(
item.subtitle.unwrap(),
vec![TitleSegment::text(
"implement per-repo enumeration for full commit history"
)]
);
}
#[test]
fn merge_pull_request_uses_merge_icon() {
let raw = json!({
"_host": "git.lair.cafe",
"act_user": { "login": "grenade" },
"repo": { "full_name": "grenade/moments" },
"content": "7|wire it up"
});
let item = reshape(&ev("merge_pull_request", raw));
assert_eq!(item.icon, TimelineIcon::GitMerge);
let r = render(&item);
assert!(
r.contains("merged pull request #7 in grenade/moments"),
"got: {r}"
);
}
#[test]
fn fallback_for_unknown_op_type() {
let raw = json!({
"_host": "git.lair.cafe",
"act_user": { "login": "grenade" },
"repo": { "full_name": "grenade/x" }
});
let item = reshape(&ev("mirror_sync_push", raw));
assert_eq!(item.icon, TimelineIcon::Generic);
let r = render(&item);
assert!(r.contains("mirror_sync_push on grenade/x"), "got: {r}");
}
}

806
crates/moments-core/src/presentation/github.rs Normal file
View File

@@ -0,0 +1,806 @@
use moments_entities::{
CommitSummary, Event, Source, TimelineBody, TimelineIcon, TimelineItem, TitleSegment,
};
use serde_json::Value;
pub(crate) fn reshape(event: &Event) -> TimelineItem {
// Search-API items have a different payload shape (the search item itself
// rather than a wrapped event), so dispatch them through a separate path.
match event.action.as_str() {
"Issue" | "PullRequest" => return search_reshape(event),
"Commit" => return commit_reshape(event),
_ => {}
}
let p = &event.payload;
let repo_name = p.get("repo").and_then(|r| r.get("name")).and_then(Value::as_str);
let actor_login = p
.get("actor")
.and_then(|a| a.get("display_login").or_else(|| a.get("login")))
.and_then(Value::as_str);
let inner = p.get("payload");
let (icon, title, subtitle, body) = match event.action.as_str() {
"PushEvent" => push(repo_name, inner),
"PullRequestEvent" => pull_request(repo_name, inner),
"PullRequestReviewEvent" => pull_request_review(repo_name, inner),
"PullRequestReviewCommentEvent" => pull_request_review_comment(repo_name, inner),
"IssuesEvent" => issues(repo_name, inner),
"IssueCommentEvent" => issue_comment(repo_name, inner),
"CreateEvent" => create(repo_name, inner),
"DeleteEvent" => delete(repo_name, inner),
"ForkEvent" => fork(repo_name, inner),
"WatchEvent" => watch(repo_name),
"ReleaseEvent" => release(repo_name, inner),
"CommitCommentEvent" => commit_comment(repo_name, inner),
"PublicEvent" => public(repo_name),
_ => fallback(repo_name, &event.action),
};
let title = if let Some(actor) = actor_login {
let mut segs = Vec::with_capacity(title.len() + 1);
segs.push(TitleSegment::link(
actor.to_string(),
format!("https://github.com/{actor}"),
));
segs.push(TitleSegment::text(" "));
segs.extend(title);
segs
} else {
title
};
TimelineItem {
id: event.id.clone(),
source: Source::Github,
action: event.action.clone(),
occurred_at: event.occurred_at,
icon,
title,
subtitle,
body,
}
}
fn repo_link(repo: &str) -> TitleSegment {
TitleSegment::link(repo.to_string(), format!("https://github.com/{repo}"))
}
fn pr_url(repo: &str, number: i64) -> String {
format!("https://github.com/{repo}/pull/{number}")
}
fn issue_url(repo: &str, number: i64) -> String {
format!("https://github.com/{repo}/issues/{number}")
}
fn commit_url(repo: &str, sha: &str) -> String {
format!("https://github.com/{repo}/commit/{sha}")
}
fn ref_branch(r: &str) -> &str {
r.strip_prefix("refs/heads/").unwrap_or(r)
}
type Reshaped = (
TimelineIcon,
Vec<TitleSegment>,
Option<Vec<TitleSegment>>,
Option<TimelineBody>,
);
fn push(repo: Option<&str>, p: Option<&Value>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let distinct_size = p
.and_then(|v| v.get("distinct_size"))
.and_then(Value::as_i64)
.unwrap_or(0);
let forced = p
.and_then(|v| v.get("forced"))
.and_then(Value::as_bool)
.unwrap_or(false);
let created = p
.and_then(|v| v.get("created"))
.and_then(Value::as_bool)
.unwrap_or(false);
let branch = p
.and_then(|v| v.get("ref"))
.and_then(Value::as_str)
.map(ref_branch)
.unwrap_or("");
// Branch-creation pushes have distinct_size = 0 (the commits already
// existed on another branch) and a different intent than a code push.
// Force-pushes and ordinary pushes both render with the GitPush icon
// but are phrased differently.
let (icon, title) = if created {
(
TimelineIcon::GitBranchCreate,
vec![
TitleSegment::text(format!("created branch {branch} in ")),
repo_link(repo),
],
)
} else if distinct_size == 0 {
let verb = if forced { "force-pushed" } else { "pushed to" };
(
TimelineIcon::GitPush,
vec![
TitleSegment::text(format!("{verb} ")),
repo_link(repo),
TitleSegment::text(format!(":{branch}")),
],
)
} else {
let verb = if forced { "force-pushed" } else { "pushed" };
let plural = if distinct_size == 1 { "" } else { "s" };
(
TimelineIcon::GitPush,
vec![
TitleSegment::text(format!("{verb} {distinct_size} commit{plural} to ")),
repo_link(repo),
TitleSegment::text(format!(":{branch}")),
],
)
};
let commits: Vec<CommitSummary> = p
.and_then(|v| v.get("commits"))
.and_then(Value::as_array)
.map(|arr| {
arr.iter()
.filter_map(|c| {
let sha = c.get("sha").and_then(Value::as_str)?;
let message = c
.get("message")
.and_then(Value::as_str)
.unwrap_or("")
.lines()
.next()
.unwrap_or("")
.to_string();
let author = c
.get("author")
.and_then(|a| a.get("name"))
.and_then(Value::as_str)
.map(str::to_string);
Some(CommitSummary {
short_sha: sha.chars().take(7).collect(),
sha: sha.to_string(),
message,
url: commit_url(repo, sha),
author,
})
})
.collect()
})
.unwrap_or_default();
let body = if commits.is_empty() {
None
} else {
Some(TimelineBody::Commits { commits })
};
(icon, title, None, body)
}
fn pull_request(repo: Option<&str>, p: Option<&Value>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let action = p
.and_then(|v| v.get("action"))
.and_then(Value::as_str)
.unwrap_or("touched");
let pr = p.and_then(|v| v.get("pull_request"));
let number = p.and_then(|v| v.get("number")).and_then(Value::as_i64).unwrap_or(0);
let pr_title = pr.and_then(|v| v.get("title")).and_then(Value::as_str).unwrap_or("");
let merged = pr
.and_then(|v| v.get("merged"))
.and_then(Value::as_bool)
.unwrap_or(false);
let verb = if action == "closed" && merged {
"merged"
} else {
action
};
let icon = if verb == "merged" {
TimelineIcon::GitMerge
} else {
TimelineIcon::PullRequest
};
let title = vec![
TitleSegment::text(format!("{verb} pull request ")),
TitleSegment::link(format!("#{number}"), pr_url(repo, number)),
TitleSegment::text(" in "),
repo_link(repo),
];
let subtitle = (!pr_title.is_empty()).then(|| vec![TitleSegment::text(pr_title.to_string())]);
(icon, title, subtitle, None)
}
fn pull_request_review(repo: Option<&str>, p: Option<&Value>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let pr = p.and_then(|v| v.get("pull_request"));
let number = pr.and_then(|v| v.get("number")).and_then(Value::as_i64).unwrap_or(0);
let pr_title = pr.and_then(|v| v.get("title")).and_then(Value::as_str).unwrap_or("");
let state = p
.and_then(|v| v.get("review"))
.and_then(|r| r.get("state"))
.and_then(Value::as_str)
.unwrap_or("commented");
let title = vec![
TitleSegment::text(format!("{state} review on ")),
TitleSegment::link(format!("#{number}"), pr_url(repo, number)),
TitleSegment::text(" in "),
repo_link(repo),
];
let subtitle = (!pr_title.is_empty()).then(|| vec![TitleSegment::text(pr_title.to_string())]);
(TimelineIcon::PullRequest, title, subtitle, None)
}
fn pull_request_review_comment(repo: Option<&str>, p: Option<&Value>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let pr = p.and_then(|v| v.get("pull_request"));
let number = pr.and_then(|v| v.get("number")).and_then(Value::as_i64).unwrap_or(0);
let pr_title = pr.and_then(|v| v.get("title")).and_then(Value::as_str).unwrap_or("");
let body_text = p
.and_then(|v| v.get("comment"))
.and_then(|c| c.get("body"))
.and_then(Value::as_str)
.unwrap_or("");
let title = vec![
TitleSegment::text("commented on review of "),
TitleSegment::link(format!("#{number}"), pr_url(repo, number)),
TitleSegment::text(" in "),
repo_link(repo),
];
let subtitle = (!pr_title.is_empty()).then(|| vec![TitleSegment::text(pr_title.to_string())]);
let body = (!body_text.is_empty()).then(|| TimelineBody::Markdown {
text: body_text.to_string(),
});
(TimelineIcon::Comment, title, subtitle, body)
}
fn issues(repo: Option<&str>, p: Option<&Value>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let action = p
.and_then(|v| v.get("action"))
.and_then(Value::as_str)
.unwrap_or("touched");
let issue = p.and_then(|v| v.get("issue"));
let number = issue.and_then(|v| v.get("number")).and_then(Value::as_i64).unwrap_or(0);
let issue_title = issue.and_then(|v| v.get("title")).and_then(Value::as_str).unwrap_or("");
let title = vec![
TitleSegment::text(format!("{action} issue ")),
TitleSegment::link(format!("#{number}"), issue_url(repo, number)),
TitleSegment::text(" in "),
repo_link(repo),
];
let subtitle = (!issue_title.is_empty()).then(|| vec![TitleSegment::text(issue_title.to_string())]);
(TimelineIcon::Issue, title, subtitle, None)
}
fn issue_comment(repo: Option<&str>, p: Option<&Value>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let issue = p.and_then(|v| v.get("issue"));
let number = issue.and_then(|v| v.get("number")).and_then(Value::as_i64).unwrap_or(0);
let issue_title = issue.and_then(|v| v.get("title")).and_then(Value::as_str).unwrap_or("");
let body_text = p
.and_then(|v| v.get("comment"))
.and_then(|c| c.get("body"))
.and_then(Value::as_str)
.unwrap_or("");
let title = vec![
TitleSegment::text("commented on "),
TitleSegment::link(format!("#{number}"), issue_url(repo, number)),
TitleSegment::text(" in "),
repo_link(repo),
];
let subtitle = (!issue_title.is_empty()).then(|| vec![TitleSegment::text(issue_title.to_string())]);
let body = (!body_text.is_empty()).then(|| TimelineBody::Markdown {
text: body_text.to_string(),
});
(TimelineIcon::Comment, title, subtitle, body)
}
fn create(repo: Option<&str>, p: Option<&Value>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let ref_type = p.and_then(|v| v.get("ref_type")).and_then(Value::as_str).unwrap_or("ref");
let ref_name = p.and_then(|v| v.get("ref")).and_then(Value::as_str);
let mut title = vec![TitleSegment::text(format!("created {ref_type} "))];
if let Some(name) = ref_name {
title.push(TitleSegment::text(format!("{name} in ")));
} else {
title.push(TitleSegment::text("in "));
}
title.push(repo_link(repo));
(TimelineIcon::GitBranchCreate, title, None, None)
}
fn delete(repo: Option<&str>, p: Option<&Value>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let ref_type = p.and_then(|v| v.get("ref_type")).and_then(Value::as_str).unwrap_or("ref");
let ref_name = p.and_then(|v| v.get("ref")).and_then(Value::as_str).unwrap_or("");
let title = vec![
TitleSegment::text(format!("deleted {ref_type} {ref_name} in ")),
repo_link(repo),
];
(TimelineIcon::GitBranchDelete, title, None, None)
}
fn fork(repo: Option<&str>, p: Option<&Value>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let forkee = p.and_then(|v| v.get("forkee"));
let forkee_full = forkee.and_then(|f| f.get("full_name")).and_then(Value::as_str);
let mut title = vec![TitleSegment::text("forked "), repo_link(repo)];
if let Some(full) = forkee_full {
title.push(TitleSegment::text(" to "));
title.push(TitleSegment::link(
full.to_string(),
format!("https://github.com/{full}"),
));
}
(TimelineIcon::GitFork, title, None, None)
}
fn watch(repo: Option<&str>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let title = vec![TitleSegment::text("starred "), repo_link(repo)];
(TimelineIcon::Star, title, None, None)
}
fn release(repo: Option<&str>, p: Option<&Value>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let release = p.and_then(|v| v.get("release"));
let name = release
.and_then(|r| r.get("name").or_else(|| r.get("tag_name")))
.and_then(Value::as_str)
.unwrap_or("(release)");
let url = release.and_then(|r| r.get("html_url")).and_then(Value::as_str);
let label = if let Some(u) = url {
TitleSegment::link(name.to_string(), u.to_string())
} else {
TitleSegment::text(name.to_string())
};
let title = vec![
TitleSegment::text("released "),
label,
TitleSegment::text(" in "),
repo_link(repo),
];
(TimelineIcon::Release, title, None, None)
}
fn commit_comment(repo: Option<&str>, p: Option<&Value>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let body_text = p
.and_then(|v| v.get("comment"))
.and_then(|c| c.get("body"))
.and_then(Value::as_str)
.unwrap_or("");
let title = vec![TitleSegment::text("commented on a commit in "), repo_link(repo)];
let body = (!body_text.is_empty()).then(|| TimelineBody::Markdown {
text: body_text.to_string(),
});
(TimelineIcon::Comment, title, None, body)
}
fn public(repo: Option<&str>) -> Reshaped {
let repo = repo.unwrap_or("(unknown repo)");
let title = vec![TitleSegment::text("made "), repo_link(repo), TitleSegment::text(" public")];
(TimelineIcon::Generic, title, None, None)
}
fn search_reshape(event: &Event) -> TimelineItem {
let p = &event.payload;
let html_url = p.get("html_url").and_then(Value::as_str).unwrap_or("");
let number = p.get("number").and_then(Value::as_i64).unwrap_or(0);
let issue_title = p.get("title").and_then(Value::as_str).unwrap_or("");
let state = p.get("state").and_then(Value::as_str).unwrap_or("");
let pr_obj = p.get("pull_request");
let is_pr = pr_obj.is_some();
let merged = pr_obj
.and_then(|pr| pr.get("merged_at"))
.map(|v| !v.is_null())
.unwrap_or(false);
let user_login = p
.get("user")
.and_then(|u| u.get("login"))
.and_then(Value::as_str);
let repo = repo_from_url(html_url).unwrap_or_else(|| "(unknown repo)".into());
let verb = match (is_pr, state, merged) {
(true, "closed", true) => "merged",
(true, "closed", false) => "closed",
(true, _, _) => "opened",
(false, "closed", _) => "closed",
(false, _, _) => "opened",
};
let kind = if is_pr { "pull request" } else { "issue" };
let icon = match (is_pr, verb) {
(true, "merged") => TimelineIcon::GitMerge,
(true, _) => TimelineIcon::PullRequest,
(false, _) => TimelineIcon::Issue,
};
let mut title = Vec::new();
if let Some(actor) = user_login {
title.push(TitleSegment::link(
actor.to_string(),
format!("https://github.com/{actor}"),
));
title.push(TitleSegment::text(" "));
}
title.push(TitleSegment::text(format!("{verb} {kind} ")));
title.push(TitleSegment::link(format!("#{number}"), html_url.to_string()));
title.push(TitleSegment::text(" in "));
title.push(repo_link(&repo));
let subtitle = (!issue_title.is_empty()).then(|| vec![TitleSegment::text(issue_title.to_string())]);
TimelineItem {
id: event.id.clone(),
source: Source::Github,
action: event.action.clone(),
occurred_at: event.occurred_at,
icon,
title,
subtitle,
body: None,
}
}
fn commit_reshape(event: &Event) -> TimelineItem {
let p = &event.payload;
let sha = p.get("sha").and_then(Value::as_str).unwrap_or("");
let short_sha: String = sha.chars().take(7).collect();
let html_url = p.get("html_url").and_then(Value::as_str).unwrap_or("");
let message_first_line = p
.get("commit")
.and_then(|c| c.get("message"))
.and_then(Value::as_str)
.unwrap_or("")
.lines()
.next()
.unwrap_or("")
.to_string();
let repo = p
.get("repository")
.and_then(|r| r.get("full_name"))
.and_then(Value::as_str)
.or_else(|| p.get("_repo").and_then(Value::as_str))
.unwrap_or("(unknown repo)");
let author_login = p
.get("author")
.and_then(|a| a.get("login"))
.and_then(Value::as_str);
let mut title = Vec::new();
if let Some(actor) = author_login {
title.push(TitleSegment::link(
actor.to_string(),
format!("https://github.com/{actor}"),
));
title.push(TitleSegment::text(" "));
}
title.push(TitleSegment::text("committed "));
title.push(TitleSegment::link(short_sha, html_url.to_string()));
title.push(TitleSegment::text(" in "));
title.push(repo_link(repo));
let subtitle = (!message_first_line.is_empty())
.then(|| vec![TitleSegment::text(message_first_line)]);
TimelineItem {
id: event.id.clone(),
source: Source::Github,
action: event.action.clone(),
occurred_at: event.occurred_at,
icon: TimelineIcon::GitCommit,
title,
subtitle,
body: None,
}
}
fn repo_from_url(url: &str) -> Option<String> {
let stripped = url.strip_prefix("https://github.com/")?;
let mut parts = stripped.splitn(3, '/');
let owner = parts.next()?;
let repo = parts.next()?;
(!owner.is_empty() && !repo.is_empty()).then(|| format!("{owner}/{repo}"))
}
fn fallback(repo: Option<&str>, action: &str) -> Reshaped {
let title = match repo {
Some(r) => vec![
TitleSegment::text(format!("{action} on ")),
repo_link(r),
],
None => vec![TitleSegment::text(action.to_string())],
};
(TimelineIcon::Generic, title, None, None)
}
#[cfg(test)]
mod tests {
use super::*;
use chrono::{TimeZone, Utc};
use serde_json::json;
fn ev(action: &str, payload: Value) -> Event {
Event {
id: "github:1".into(),
source: Source::Github,
action: action.into(),
occurred_at: Utc.with_ymd_and_hms(2026, 4, 14, 10, 0, 0).unwrap(),
public: true,
payload,
}
}
#[test]
fn push_event_reshape() {
let raw = json!({
"actor": { "login": "grenade", "display_login": "grenade" },
"repo": { "name": "grenade/vortex" },
"payload": {
"ref": "refs/heads/main",
"size": 2,
"distinct_size": 2,
"commits": [
{ "sha": "abcdef1234567890", "message": "fix the thing", "author": { "name": "rob" } },
{ "sha": "1111111111111111", "message": "and another\nbody", "author": { "name": "rob" } }
]
}
});
let item = reshape(&ev("PushEvent", raw));
assert_eq!(item.icon, TimelineIcon::GitPush);
// first segment is the actor link, then "pushed N commits to <repo>:<branch>"
assert!(matches!(item.title[0], TitleSegment::Link { .. }));
let rendered: String = item
.title
.iter()
.map(|s| match s {
TitleSegment::Text { text } => text.clone(),
TitleSegment::Link { text, .. } => text.clone(),
})
.collect();
assert!(rendered.contains("pushed 2 commits to grenade/vortex:main"), "got: {rendered}");
match item.body.unwrap() {
TimelineBody::Commits { commits } => {
assert_eq!(commits.len(), 2);
assert_eq!(commits[0].short_sha, "abcdef1");
// multi-line message gets first line only
assert_eq!(commits[1].message, "and another");
}
_ => panic!("expected Commits body"),
}
}
fn render(item: &TimelineItem) -> String {
item.title
.iter()
.map(|s| match s {
TitleSegment::Text { text } => text.clone(),
TitleSegment::Link { text, .. } => text.clone(),
})
.collect()
}
#[test]
fn push_branch_creation_uses_create_icon() {
let raw = json!({
"actor": { "login": "grenade" },
"repo": { "name": "grenade/vortex" },
"payload": {
"ref": "refs/heads/fix-double-panic",
"size": 0,
"distinct_size": 0,
"created": true,
"forced": false,
"commits": []
}
});
let item = reshape(&ev("PushEvent", raw));
assert_eq!(item.icon, TimelineIcon::GitBranchCreate);
let r = render(&item);
assert!(
r.contains("created branch fix-double-panic in grenade/vortex"),
"got: {r}"
);
}
#[test]
fn force_push_with_commits_says_force_pushed() {
let raw = json!({
"actor": { "login": "grenade" },
"repo": { "name": "grenade/x" },
"payload": {
"ref": "refs/heads/main",
"size": 1,
"distinct_size": 1,
"created": false,
"forced": true,
"commits": [{ "sha": "deadbeefcafe1234", "message": "rebase" }]
}
});
let item = reshape(&ev("PushEvent", raw));
assert_eq!(item.icon, TimelineIcon::GitPush);
let r = render(&item);
assert!(r.contains("force-pushed 1 commit to grenade/x:main"), "got: {r}");
}
#[test]
fn empty_push_omits_commit_count() {
let raw = json!({
"actor": { "login": "grenade" },
"repo": { "name": "grenade/x" },
"payload": {
"ref": "refs/heads/main",
"size": 0,
"distinct_size": 0,
"created": false,
"forced": false,
"commits": []
}
});
let item = reshape(&ev("PushEvent", raw));
assert_eq!(item.icon, TimelineIcon::GitPush);
let r = render(&item);
assert!(r.contains("pushed to grenade/x:main"), "got: {r}");
assert!(!r.contains("0 commit"), "should not say '0 commits': {r}");
}
#[test]
fn merged_pr_uses_merge_icon() {
let raw = json!({
"actor": { "login": "grenade" },
"repo": { "name": "grenade/moments" },
"payload": {
"action": "closed",
"number": 7,
"pull_request": { "title": "wire it up", "merged": true }
}
});
let item = reshape(&ev("PullRequestEvent", raw));
assert_eq!(item.icon, TimelineIcon::GitMerge);
let rendered: String = item
.title
.iter()
.map(|s| match s {
TitleSegment::Text { text } => text.clone(),
TitleSegment::Link { text, .. } => text.clone(),
})
.collect();
assert!(rendered.contains("merged pull request #7 in grenade/moments"));
assert_eq!(
item.subtitle.unwrap(),
vec![TitleSegment::text("wire it up")]
);
}
#[test]
fn issue_comment_carries_markdown_body() {
let raw = json!({
"actor": { "login": "grenade" },
"repo": { "name": "Nehliin/vortex" },
"payload": {
"issue": { "number": 42, "title": "perf regression" },
"comment": { "body": "looks like the io_uring batching changed" }
}
});
let item = reshape(&ev("IssueCommentEvent", raw));
assert_eq!(item.icon, TimelineIcon::Comment);
match item.body.unwrap() {
TimelineBody::Markdown { text } => {
assert!(text.contains("io_uring"));
}
_ => panic!("expected Markdown body"),
}
}
#[test]
fn search_issue_reshape_open() {
let raw = json!({
"number": 125,
"title": "Feature: peer blocklist",
"state": "open",
"html_url": "https://github.com/Nehliin/vortex/issues/125",
"user": { "login": "grenade" }
});
let item = reshape(&ev("Issue", raw));
assert_eq!(item.icon, TimelineIcon::Issue);
let rendered: String = item
.title
.iter()
.map(|s| match s {
TitleSegment::Text { text } => text.clone(),
TitleSegment::Link { text, .. } => text.clone(),
})
.collect();
assert!(
rendered.contains("opened issue #125 in Nehliin/vortex"),
"got: {rendered}"
);
}
#[test]
fn search_pr_reshape_merged_uses_merge_icon() {
let raw = json!({
"number": 42,
"title": "wire it up",
"state": "closed",
"html_url": "https://github.com/grenade/moments/pull/42",
"user": { "login": "grenade" },
"pull_request": { "merged_at": "2026-04-15T10:00:00Z" }
});
let item = reshape(&ev("PullRequest", raw));
assert_eq!(item.icon, TimelineIcon::GitMerge);
let rendered: String = item
.title
.iter()
.map(|s| match s {
TitleSegment::Text { text } => text.clone(),
TitleSegment::Link { text, .. } => text.clone(),
})
.collect();
assert!(
rendered.contains("merged pull request #42 in grenade/moments"),
"got: {rendered}"
);
}
#[test]
fn commit_reshape_uses_short_sha_and_first_message_line() {
let raw = json!({
"sha": "a6fcefbe909a97ad5a049b9fa48bc74309af10d9",
"html_url": "https://github.com/faith1337z/Trade/commit/a6fcefbe909a97ad5a049b9fa48bc74309af10d9",
"commit": {
"message": "split multiline message into multiple irc messages\n\nbody body body"
},
"repository": { "full_name": "faith1337z/Trade" },
"author": { "login": "grenade" }
});
let item = reshape(&ev("Commit", raw));
assert_eq!(item.icon, TimelineIcon::GitCommit);
let rendered: String = item
.title
.iter()
.map(|s| match s {
TitleSegment::Text { text } => text.clone(),
TitleSegment::Link { text, .. } => text.clone(),
})
.collect();
assert!(rendered.contains("committed a6fcefb in faith1337z/Trade"), "got: {rendered}");
// body of the commit message is dropped; only first line in subtitle
assert_eq!(
item.subtitle.unwrap(),
vec![TitleSegment::text("split multiline message into multiple irc messages")]
);
}
#[test]
fn unknown_event_falls_back() {
let raw = json!({
"actor": { "login": "grenade" },
"repo": { "name": "grenade/x" },
"payload": {}
});
let item = reshape(&ev("SponsorshipEvent", raw));
assert_eq!(item.icon, TimelineIcon::Generic);
assert_eq!(item.action, "SponsorshipEvent");
}
}

126
crates/moments-core/src/presentation/hg.rs Normal file
View File

@@ -0,0 +1,126 @@
use moments_entities::{Event, Source, TimelineIcon, TimelineItem, TitleSegment};
use serde_json::Value;
const FALLBACK_HOST: &str = "hg-edge.mozilla.org";
pub(crate) fn reshape(event: &Event) -> TimelineItem {
let p = &event.payload;
let host = p
.get("_host")
.and_then(Value::as_str)
.unwrap_or(FALLBACK_HOST);
let repo = p
.get("_repo")
.and_then(Value::as_str)
.unwrap_or("(unknown repo)");
let node = p.get("node").and_then(Value::as_str).unwrap_or("");
let short_node: String = node.chars().take(12).collect();
let desc = p
.get("desc")
.and_then(Value::as_str)
.unwrap_or("")
.lines()
.next()
.unwrap_or("")
.to_string();
let author = p
.get("author")
.and_then(Value::as_str)
.map(author_name);
let mut title = Vec::new();
if let Some(name) = author {
title.push(TitleSegment::text(format!("{name} ")));
}
title.push(TitleSegment::text("committed "));
title.push(TitleSegment::link(
short_node,
format!("https://{host}/{repo}/rev/{node}"),
));
title.push(TitleSegment::text(" in "));
title.push(TitleSegment::link(
repo.to_string(),
format!("https://{host}/{repo}"),
));
let subtitle = (!desc.is_empty()).then(|| vec![TitleSegment::text(desc)]);
TimelineItem {
id: event.id.clone(),
source: Source::Hg,
action: event.action.clone(),
occurred_at: event.occurred_at,
icon: TimelineIcon::GitCommit,
title,
subtitle,
body: None,
}
}
/// Drop the `<email>` portion of an hg author string ("Name <email>") and
/// trim — leaves just the display name. If there's no email, return the
/// trimmed input.
fn author_name(s: &str) -> String {
if let Some(idx) = s.find('<') {
s[..idx].trim().to_string()
} else {
s.trim().to_string()
}
}
#[cfg(test)]
mod tests {
use super::*;
use chrono::{TimeZone, Utc};
use serde_json::json;
fn ev(payload: Value) -> Event {
Event {
id: "hg:build/puppet:abc".into(),
source: Source::Hg,
action: "Commit".into(),
occurred_at: Utc.with_ymd_and_hms(2018, 5, 1, 12, 0, 0).unwrap(),
public: true,
payload,
}
}
fn render(item: &TimelineItem) -> String {
item.title
.iter()
.map(|s| match s {
TitleSegment::Text { text } => text.clone(),
TitleSegment::Link { text, .. } => text.clone(),
})
.collect()
}
#[test]
fn reshape_hg_commit() {
let raw = json!({
"_host": "hg-edge.mozilla.org",
"_repo": "build/puppet",
"node": "abcdef1234567890abcdef",
"desc": "Bug 1234 - fix something\n\nlonger body",
"author": "Rob Thijssen <rthijssen@mozilla.com>"
});
let item = reshape(&ev(raw));
assert_eq!(item.icon, TimelineIcon::GitCommit);
let r = render(&item);
assert!(
r.contains("Rob Thijssen committed abcdef123456 in build/puppet"),
"got: {r}"
);
assert_eq!(
item.subtitle.unwrap(),
vec![TitleSegment::text("Bug 1234 - fix something")]
);
}
#[test]
fn drops_email_from_author() {
assert_eq!(author_name("Rob Thijssen <rob@example>"), "Rob Thijssen");
assert_eq!(author_name("nobody"), "nobody");
assert_eq!(author_name(" spaced "), "spaced");
}
}

1
crates/moments-data/Cargo.toml
View File

@@ -17,3 +17,4 @@ tracing.workspace = true
async-trait.workspace = true
reqwest.workspace = true
serde.workspace = true
percent-encoding = "2"

2
crates/moments-data/migrations/0003_event_public.sql Normal file
View File

@@ -0,0 +1,2 @@
ALTER TABLE events ADD COLUMN public BOOLEAN NOT NULL DEFAULT true;
CREATE INDEX events_public_occurred_at_desc ON events (public, occurred_at DESC);

9
crates/moments-data/migrations/0004_repo_languages.sql Normal file
View File

@@ -0,0 +1,9 @@
CREATE TABLE repo_languages (
source TEXT NOT NULL,
repo TEXT NOT NULL,
language TEXT NOT NULL,
bytes BIGINT NOT NULL,
color TEXT,
fetched_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (source, repo, language)
);

55
crates/moments-data/migrations/0005_dedup_gitea_events.sql Normal file
View File

@@ -0,0 +1,55 @@
-- Collapse duplicate Gitea events introduced by polling both the user
-- activity feed and per-org activity feeds.
--
-- Gitea writes one Action row per interested user-context: a push to
-- helexa/cortex by user grenade produces two rows, one with
-- user_id=grenade and one with user_id=helexa. Everything else (op_type,
-- act_user_id, repo_id, ref_name, comment_id, created) is identical.
-- Our prior id scheme (gitea:{action_row_id}) gave them different ids,
-- so the upsert-on-id dedup never fired and the timeline rendered each
-- push twice.
--
-- This migration re-keys every existing gitea row to the same canonical
-- formula `parse_gitea_event` now emits, deleting duplicates encountered
-- along the way. Idempotent: running it again is a no-op because the
-- canonical id of a canonical id is itself.
-- Snapshot the canonical id for every gitea row.
CREATE TEMP TABLE _gitea_canonical AS
SELECT
id AS old_id,
'gitea:'
|| coalesce(payload->>'op_type', '') || ':'
|| coalesce(payload->>'act_user_id', payload->'act_user'->>'id', '0') || ':'
|| coalesce(payload->>'repo_id', payload->'repo'->>'id', '0') || ':'
|| coalesce(payload->>'ref_name', '') || ':'
|| coalesce(payload->>'comment_id', '0') || ':'
|| coalesce(payload->>'created', '')
AS new_id
FROM events
WHERE source = 'gitea';
-- For each canonical id, keep the row whose current id is lexicographically
-- smallest (stable, arbitrary tie-break) and delete the rest. The "old id
-- already matches the new id" case lands here too — DELETE skips it because
-- rn = 1 for any singleton group.
DELETE FROM events
WHERE id IN (
SELECT old_id FROM (
SELECT old_id, new_id,
row_number() OVER (PARTITION BY new_id ORDER BY old_id) AS rn
FROM _gitea_canonical
) ranked
WHERE rn > 1
);
-- Rename remaining rows to the canonical id. Postgres defers PK uniqueness
-- to statement end, so swapping ids across rows in one UPDATE is fine
-- provided the final set is unique (dedup above guarantees that).
UPDATE events e
SET id = c.new_id
FROM _gitea_canonical c
WHERE e.id = c.old_id
AND e.id <> c.new_id;
DROP TABLE _gitea_canonical;

176
crates/moments-data/src/bugzilla.rs Normal file
View File

@@ -0,0 +1,176 @@
//! Bugzilla REST API ingestion.
//!
//! Hits `/rest/bug?creator=<email>` to pull bugs the user filed. Without
//! an api key, only public bugs are returned, which is what we want for
//! the public timeline anyway.
//!
//! No incremental story for v1 — each tick refetches the full list and
//! relies on idempotent upsert by `bugzilla:<id>`. Comments and bug
//! changes (status flips, assignment, etc.) aren't ingested for v1
//! because they require per-bug API calls; revisit if that history
//! becomes desirable.
use std::sync::Arc;
use async_trait::async_trait;
use chrono::{DateTime, Utc};
use moments_core::{EventSource, EventWriter, PollerStateStore, SourceError};
use moments_entities::{Event, Source};
use reqwest::{Client, header};
use serde_json::Value;
use tracing::debug;
const SOURCE_NAME: &str = "bugzilla";
const USER_AGENT: &str = concat!(
"moments/",
env!("CARGO_PKG_VERSION"),
" (+https://rob.tn)"
);
#[derive(Clone, Debug)]
pub struct BugzillaConfig {
pub host: String,
pub creator_email: String,
pub api_key: Option<String>,
/// Bugs requested per page; bugzilla allows up to 1000.
pub limit: u32,
}
impl Default for BugzillaConfig {
fn default() -> Self {
Self {
host: "bugzilla.mozilla.org".into(),
creator_email: "rthijssen@mozilla.com".into(),
api_key: None,
limit: 500,
}
}
}
pub struct BugzillaSource {
client: Client,
writer: Arc<dyn EventWriter>,
state: Arc<dyn PollerStateStore>,
config: BugzillaConfig,
}
impl BugzillaSource {
pub fn new(
client: Client,
writer: Arc<dyn EventWriter>,
state: Arc<dyn PollerStateStore>,
config: BugzillaConfig,
) -> Self {
Self {
client,
writer,
state,
config,
}
}
}
#[async_trait]
impl EventSource for BugzillaSource {
fn name(&self) -> &'static str {
SOURCE_NAME
}
async fn poll(&self) -> Result<usize, SourceError> {
let url = format!(
"https://{}/rest/bug?creator={}&limit={}\
&include_fields=id,summary,creation_time,last_change_time,status,resolution,product,component",
self.config.host, self.config.creator_email, self.config.limit
);
let mut req = self
.client
.get(&url)
.header(header::USER_AGENT, USER_AGENT)
.header(header::ACCEPT, "application/json");
if let Some(key) = &self.config.api_key {
req = req.header("X-BUGZILLA-API-KEY", key);
}
let resp = req
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
if !resp.status().is_success() {
return Err(SourceError::Http(format!("{} GET {}", resp.status(), url)));
}
let body: Value = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
let bugs = body
.get("bugs")
.and_then(Value::as_array)
.cloned()
.unwrap_or_default();
let events: Vec<Event> = bugs
.iter()
.filter_map(|b| parse_bug(b, &self.config.host))
.collect();
let n = self.writer.upsert_events(&events).await?;
self.state.touch(SOURCE_NAME).await?;
debug!(ingested = n, total = bugs.len(), "bugzilla poll complete");
Ok(n)
}
}
fn parse_bug(bug: &Value, host: &str) -> Option<Event> {
let id = bug.get("id").and_then(Value::as_i64)?;
let creation_time = bug.get("creation_time").and_then(Value::as_str)?;
let occurred_at = DateTime::parse_from_rfc3339(creation_time)
.ok()?
.with_timezone(&Utc);
let mut payload = bug.clone();
if let Some(obj) = payload.as_object_mut() {
obj.insert("_host".into(), Value::String(host.into()));
}
Some(Event {
id: format!("bugzilla:{id}"),
source: Source::Bugzilla,
action: "BugCreate".into(),
occurred_at,
// The unauth REST API only returns publicly visible bugs.
public: true,
payload,
})
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn parses_bug_with_creation_time() {
let raw = json!({
"id": 1158879,
"summary": "Commit Access (Level 1) for Rob Thijssen",
"creation_time": "2015-04-27T16:29:59Z",
"last_change_time": "2015-04-28T14:06:48Z",
"status": "RESOLVED",
"product": "mozilla.org"
});
let ev = parse_bug(&raw, "bugzilla.mozilla.org").expect("parses");
assert_eq!(ev.id, "bugzilla:1158879");
assert_eq!(ev.action, "BugCreate");
assert_eq!(ev.source, Source::Bugzilla);
assert!(ev.public);
assert_eq!(
ev.payload.get("_host").and_then(|v| v.as_str()),
Some("bugzilla.mozilla.org")
);
}
#[test]
fn rejects_bug_missing_id_or_creation_time() {
let raw = json!({ "summary": "x" });
assert!(parse_bug(&raw, "bugzilla.mozilla.org").is_none());
}
}

440
crates/moments-data/src/gitea.rs Normal file
View File

@@ -0,0 +1,440 @@
//! Gitea activity feed ingestion.
//!
//! Hits `/api/v1/users/{user}/activities/feeds?only-performed-by=true`
//! which returns events the user themselves caused (not received events
//! from others they follow). No ETag support upstream, so each tick fetches
//! page 1 and relies on idempotent upsert. First run paginates further to
//! seed history.
//!
//! Each item carries a self-contained payload — including the event-emitting
//! host — so the reshape layer can construct URLs without needing config.
use std::collections::HashSet;
use std::sync::Arc;
use async_trait::async_trait;
use chrono::{DateTime, Utc};
use moments_core::{EventSource, EventWriter, PollerStateStore, SourceError};
use moments_entities::{Event, RepoLanguage, Source};
use reqwest::{Client, header};
use serde_json::Value;
use tracing::debug;
const SOURCE_NAME: &str = "gitea";
const USER_AGENT: &str = concat!(
"moments/",
env!("CARGO_PKG_VERSION"),
" (+https://rob.tn)"
);
const MAX_BACKFILL_PAGES: u32 = 20;
#[derive(Clone, Debug)]
pub struct GiteaConfig {
/// e.g. `git.lair.cafe`. Used to construct URLs the API doesn't return
/// directly (issue / PR / commit web links) and stamped into each event
/// payload for the reshape layer.
pub host: String,
pub user: String,
pub token: Option<String>,
pub per_page: u32,
}
impl Default for GiteaConfig {
fn default() -> Self {
Self {
host: "git.lair.cafe".into(),
user: "grenade".into(),
token: None,
per_page: 50,
}
}
}
pub struct GiteaSource {
client: Client,
writer: Arc<dyn EventWriter>,
state: Arc<dyn PollerStateStore>,
config: GiteaConfig,
}
impl GiteaSource {
pub fn new(
client: Client,
writer: Arc<dyn EventWriter>,
state: Arc<dyn PollerStateStore>,
config: GiteaConfig,
) -> Self {
Self {
client,
writer,
state,
config,
}
}
fn user_feed_base_url(&self) -> String {
format!(
"https://{}/api/v1/users/{}/activities/feeds?only-performed-by=true&limit={}",
self.config.host, self.config.user, self.config.per_page
)
}
fn org_feed_base_url(&self, org: &str) -> String {
format!(
"https://{}/api/v1/orgs/{}/activities/feeds?limit={}",
self.config.host, org, self.config.per_page
)
}
fn apply_headers(&self, mut req: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
req = req
.header(header::ACCEPT, "application/json")
.header(header::USER_AGENT, USER_AGENT);
if let Some(token) = &self.config.token {
req = req.header(header::AUTHORIZATION, format!("token {token}"));
}
req
}
/// Discover organizations the authenticated user belongs to.
/// Returns an empty vec if no token is configured or the request fails.
async fn discover_orgs(&self) -> Result<Vec<String>, SourceError> {
if self.config.token.is_none() {
return Ok(vec![]);
}
let url = format!("https://{}/api/v1/user/orgs", self.config.host);
let req = self.apply_headers(self.client.get(&url));
let resp = req
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
if !resp.status().is_success() {
tracing::warn!(status = %resp.status(), "failed to discover gitea orgs");
return Ok(vec![]);
}
let orgs: Vec<Value> = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
Ok(orgs
.iter()
.filter_map(|o| o.get("username").and_then(Value::as_str).map(String::from))
.collect())
}
/// Poll a single activity feed, paginating on first run. When `filter_user`
/// is true, only events performed by `self.config.user` are ingested (used
/// for org feeds which contain all members' activity).
///
/// `base_url` should contain everything except the `&page=N` suffix.
/// Returns (ingested_count, set_of_repo_full_names).
async fn poll_feed(
&self,
state_key: &str,
base_url: &str,
filter_user: bool,
) -> Result<(usize, HashSet<String>), SourceError> {
let prior = self.state.load(state_key).await?;
let first_run = prior.is_none();
let max_pages = if first_run { MAX_BACKFILL_PAGES } else { 1 };
let mut total = 0usize;
let mut repos = HashSet::new();
for page in 1..=max_pages {
let url = format!("{base_url}&page={page}");
let req = self.apply_headers(self.client.get(&url));
let resp = req
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
if !resp.status().is_success() {
return Err(SourceError::Http(format!("{} GET {}", resp.status(), url)));
}
let items: Vec<Value> = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
if items.is_empty() {
break;
}
// Collect repo names from feed items
for item in &items {
if let Some(name) = item
.get("repo")
.and_then(|r| r.get("full_name"))
.and_then(Value::as_str)
{
repos.insert(name.to_string());
}
}
let events: Vec<Event> = items
.iter()
.filter(|it| {
if !filter_user {
return true;
}
it.get("act_user")
.and_then(|u| u.get("login"))
.and_then(Value::as_str)
.map(|login| login.eq_ignore_ascii_case(&self.config.user))
.unwrap_or(false)
})
.filter_map(|it| parse_gitea_event(it, &self.config.host))
.collect();
total += self.writer.upsert_events(&events).await?;
if items.len() < self.config.per_page as usize {
break;
}
}
self.state.touch(state_key).await?;
Ok((total, repos))
}
/// Fetch language breakdowns for the given repos via the Gitea REST API.
async fn fetch_languages(&self, repos: &HashSet<String>) -> Result<usize, SourceError> {
let mut total = 0usize;
for repo in repos {
let url = format!(
"https://{}/api/v1/repos/{}/languages",
self.config.host, repo
);
let req = self.apply_headers(self.client.get(&url));
let resp = req
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
if !resp.status().is_success() {
tracing::warn!(repo = %repo, status = %resp.status(), "gitea language fetch failed; skipping");
continue;
}
let lang_map: std::collections::HashMap<String, i64> = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
let languages: Vec<RepoLanguage> = lang_map
.into_iter()
.map(|(language, bytes)| RepoLanguage {
source: Source::Gitea,
repo: repo.clone(),
language,
bytes,
color: None, // Gitea doesn't return colors
})
.collect();
total += self.writer.upsert_repo_languages(&languages).await?;
}
debug!(total, repos = repos.len(), "gitea repo languages updated");
Ok(total)
}
}
#[async_trait]
impl EventSource for GiteaSource {
fn name(&self) -> &'static str {
SOURCE_NAME
}
async fn poll(&self) -> Result<usize, SourceError> {
let mut all_repos = HashSet::new();
// Poll user's own activity feed (existing behavior).
let user_url = self.user_feed_base_url();
let (mut total, repos) = self.poll_feed(SOURCE_NAME, &user_url, false).await?;
all_repos.extend(repos);
// Discover orgs and poll each org's activity feed, filtering for
// events performed by this user.
let orgs = self.discover_orgs().await?;
for org in &orgs {
let state_key = format!("gitea:org:{org}");
let org_url = self.org_feed_base_url(org);
match self.poll_feed(&state_key, &org_url, true).await {
Ok((n, repos)) => {
total += n;
all_repos.extend(repos);
}
Err(e) => {
tracing::warn!(org = %org, error = %e, "failed to poll org feed");
}
}
}
if let Err(e) = self.fetch_languages(&all_repos).await {
tracing::warn!(error = %e, "gitea language fetch failed; continuing");
}
debug!(ingested = total, orgs = orgs.len(), "gitea poll complete");
Ok(total)
}
}
/// Convert a Gitea activity feed item into our Event row. The host gets
/// stamped into the payload as `_host` so the reshape layer can build
/// web URLs without needing global config.
///
/// The id is content-derived rather than using Gitea's `id` field directly:
/// Gitea creates one Action row per interested user-context, so a push to
/// an org repo by user U produces two rows (one under U's context, one
/// under the org's), distinguished only by `id` and `user_id`. Keying on
/// `(op_type, act_user_id, repo_id, ref_name, comment_id, created)` makes
/// those two rows collapse to the same event on upsert.
fn parse_gitea_event(item: &Value, host: &str) -> Option<Event> {
let op_type = item.get("op_type").and_then(Value::as_str)?.to_string();
let created_str = item.get("created").and_then(Value::as_str)?;
let occurred_at = DateTime::parse_from_rfc3339(created_str)
.ok()?
.with_timezone(&Utc);
let private = item.get("is_private").and_then(Value::as_bool).unwrap_or(false);
let id = gitea_canonical_id(item, &op_type, created_str);
let mut payload = item.clone();
if let Some(obj) = payload.as_object_mut() {
obj.insert("_host".into(), Value::String(host.into()));
}
Some(Event {
id,
source: Source::Gitea,
action: op_type,
occurred_at,
public: !private,
payload,
})
}
/// Build the canonical, content-derived id for a Gitea action. Must stay
/// in lockstep with the SQL formula in migration 0005 so back-fill and
/// new writes share the same id space.
fn gitea_canonical_id(item: &Value, op_type: &str, created: &str) -> String {
let act_user_id = item
.get("act_user_id")
.and_then(Value::as_i64)
.or_else(|| item.get("act_user").and_then(|u| u.get("id")).and_then(Value::as_i64))
.unwrap_or(0);
let repo_id = item
.get("repo_id")
.and_then(Value::as_i64)
.or_else(|| item.get("repo").and_then(|r| r.get("id")).and_then(Value::as_i64))
.unwrap_or(0);
let ref_name = item.get("ref_name").and_then(Value::as_str).unwrap_or("");
let comment_id = item.get("comment_id").and_then(Value::as_i64).unwrap_or(0);
format!("gitea:{op_type}:{act_user_id}:{repo_id}:{ref_name}:{comment_id}:{created}")
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn parse_commit_repo() {
let raw = json!({
"id": 973,
"op_type": "commit_repo",
"act_user_id": 42,
"repo_id": 7,
"ref_name": "refs/heads/main",
"is_private": false,
"content": "{\"Commits\":[{\"Sha1\":\"abc123\"}],\"Len\":1}",
"created": "2026-05-03T16:37:45Z",
"repo": { "id": 7, "full_name": "grenade/moments" }
});
let ev = parse_gitea_event(&raw, "git.lair.cafe").expect("parses");
assert_eq!(ev.id, "gitea:commit_repo:42:7:refs/heads/main:0:2026-05-03T16:37:45Z");
assert_eq!(ev.source, Source::Gitea);
assert_eq!(ev.action, "commit_repo");
assert!(ev.public);
// host stamped into payload
assert_eq!(
ev.payload.get("_host").and_then(|v| v.as_str()),
Some("git.lair.cafe")
);
}
#[test]
fn dup_action_rows_for_user_and_org_contexts_collapse_to_same_id() {
// Gitea creates two Action rows when grenade pushes to helexa/cortex:
// one with user_id=grenade (surfaced by the user feed), one with
// user_id=helexa (surfaced by the org feed). Everything except `id`
// and `user_id` is identical. The canonical id ignores both.
let user_ctx = json!({
"id": 1322,
"user_id": 42,
"op_type": "commit_repo",
"act_user_id": 42,
"act_user": { "login": "grenade", "id": 42 },
"repo_id": 99,
"repo": { "id": 99, "full_name": "helexa/cortex" },
"ref_name": "refs/heads/main",
"comment_id": 0,
"is_private": false,
"created": "2026-05-20T04:32:50Z"
});
let org_ctx = json!({
"id": 1323,
"user_id": 7,
"op_type": "commit_repo",
"act_user_id": 42,
"act_user": { "login": "grenade", "id": 42 },
"repo_id": 99,
"repo": { "id": 99, "full_name": "helexa/cortex" },
"ref_name": "refs/heads/main",
"comment_id": 0,
"is_private": false,
"created": "2026-05-20T04:32:50Z"
});
let a = parse_gitea_event(&user_ctx, "git.lair.cafe").expect("parses");
let b = parse_gitea_event(&org_ctx, "git.lair.cafe").expect("parses");
assert_eq!(a.id, b.id, "duplicate action rows must collide on id");
}
#[test]
fn org_event_user_filter_predicate() {
let by_user = json!({
"id": 500, "op_type": "commit_repo", "is_private": false,
"created": "2026-05-03T10:00:00Z",
"act_user": { "login": "grenade" },
"repo": { "full_name": "myorg/somerepo" }
});
let by_other = json!({
"id": 501, "op_type": "commit_repo", "is_private": false,
"created": "2026-05-03T10:01:00Z",
"act_user": { "login": "otherperson" },
"repo": { "full_name": "myorg/somerepo" }
});
// Both parse as valid events
assert!(parse_gitea_event(&by_user, "git.lair.cafe").is_some());
assert!(parse_gitea_event(&by_other, "git.lair.cafe").is_some());
// The user-filter predicate used by poll_feed
let is_user = |item: &Value, user: &str| -> bool {
item.get("act_user")
.and_then(|u| u.get("login"))
.and_then(Value::as_str)
.map(|login| login.eq_ignore_ascii_case(user))
.unwrap_or(false)
};
assert!(is_user(&by_user, "grenade"));
assert!(!is_user(&by_other, "grenade"));
// Case-insensitive match
assert!(is_user(&by_user, "Grenade"));
}
#[test]
fn private_event_marked_private() {
let raw = json!({
"id": 100,
"op_type": "commit_repo",
"is_private": true,
"created": "2026-05-03T00:00:00Z",
"repo": { "full_name": "grenade/private" }
});
let ev = parse_gitea_event(&raw, "git.lair.cafe").expect("parses");
assert!(!ev.public);
}
}

51
crates/moments-data/src/github.rs
View File

@@ -58,11 +58,17 @@ impl GithubSource {
}
fn first_page_url(&self) -> String {
// Public events endpoint: works without auth (60/hr unauth, 5000/hr authed).
// The non-public `/users/{u}/events` endpoint now requires auth and returns
// private-repo activity, which we don't want on a public timeline anyway.
// With a token: hit `/events`, which returns public + private events the
// authenticated user can see. We store everything; the API gates what
// gets surfaced to the public timeline via the `public` column.
// Without a token: fall back to `/events/public` (anonymous-readable).
let endpoint = if self.config.token.is_some() {
"events"
} else {
"events/public"
};
format!(
"https://api.github.com/users/{}/events/public?per_page={}",
"https://api.github.com/users/{}/{endpoint}?per_page={}",
self.config.user, self.config.per_page
)
}
@@ -172,11 +178,17 @@ fn parse_github_event(raw: serde_json::Value) -> Option<Event> {
let occurred_at = DateTime::parse_from_rfc3339(created_at_str)
.ok()?
.with_timezone(&Utc);
// GitHub marks each event with a top-level `public` boolean. Events from
// `/events/public` are always true; `/events` may include false. Default
// to true if missing — that matches the safer-of-the-two-mistakes (under-
// expose) and the `/events/public` endpoint behaviour.
let public = raw.get("public").and_then(serde_json::Value::as_bool).unwrap_or(true);
Some(Event {
id: format!("github:{id}"),
source: Source::Github,
action: event_type,
occurred_at,
public,
payload: raw,
})
}
@@ -208,6 +220,7 @@ mod tests {
"id": "12345",
"type": "PushEvent",
"created_at": "2026-04-15T10:30:00Z",
"public": true,
"actor": { "login": "grenade" },
"repo": { "name": "grenade/moments" },
"payload": { "ref": "refs/heads/main" }
@@ -216,9 +229,39 @@ mod tests {
assert_eq!(ev.id, "github:12345");
assert_eq!(ev.source, Source::Github);
assert_eq!(ev.action, "PushEvent");
assert!(ev.public);
assert_eq!(ev.payload, raw);
}
#[test]
fn private_event_marked_private() {
let raw = serde_json::json!({
"id": "67890",
"type": "PushEvent",
"created_at": "2026-04-15T10:30:00Z",
"public": false,
"actor": { "login": "grenade" },
"repo": { "name": "grenade/private-thing" },
"payload": {}
});
let ev = parse_github_event(raw).expect("parses");
assert!(!ev.public);
}
#[test]
fn missing_public_field_defaults_to_public() {
let raw = serde_json::json!({
"id": "11111",
"type": "PushEvent",
"created_at": "2026-04-15T10:30:00Z",
"actor": { "login": "grenade" },
"repo": { "name": "grenade/x" },
"payload": {}
});
let ev = parse_github_event(raw).expect("parses");
assert!(ev.public);
}
#[test]
fn rejects_event_missing_id() {
let raw = serde_json::json!({ "type": "PushEvent", "created_at": "2026-01-01T00:00:00Z" });

808
crates/moments-data/src/github_repo.rs Normal file
View File

@@ -0,0 +1,808 @@
//! Per-repo commit enumeration for full GitHub history.
//!
//! Discovers repos via two sources:
//! 1. REST `/user/repos` — repos where the user is owner, collaborator,
//! or org member.
//! 2. GraphQL `repositoriesContributedTo` — repos the user has committed
//! to, opened issues/PRs on, or reviewed, even without collaborator
//! status. No result cap (cursor-paginated).
//!
//! Then walks each branch's commit history via
//! `/repos/{owner}/{repo}/commits?author={user}&sha={branch}` with a
//! per-branch `since` cursor to avoid re-fetching known commits. Walking
//! every branch (not just the default) is what catches work-in-progress
//! on feature branches and pushes to fork branches that never get merged
//! upstream — neither the user events feed nor /search/commits surface
//! those reliably.
//!
//! Events use `github-commit:{sha}` as their ID, matching the scheme in
//! `github_search`, so duplicates are resolved via idempotent upsert
//! (the same commit reached via two branches just upserts twice).
use std::collections::HashSet;
use std::sync::Arc;
use async_trait::async_trait;
use chrono::{DateTime, Utc};
use moments_core::{EventSource, EventWriter, PollerStateStore, SourceError};
use moments_entities::{Event, RepoLanguage, Source};
use percent_encoding::{AsciiSet, CONTROLS, utf8_percent_encode};
use reqwest::{Client, header};
use serde_json::Value;
use tracing::{debug, warn};
/// Encode characters that have meaning in a URL query — branch names can
/// contain `/`, `#`, `?`, etc. Whitelisting is too fragile; encode anything
/// outside the unreserved set plus a few safe characters.
const BRANCH_ENCODE_SET: &AsciiSet = &CONTROLS
.add(b' ')
.add(b'"')
.add(b'#')
.add(b'<')
.add(b'>')
.add(b'?')
.add(b'`')
.add(b'{')
.add(b'}')
.add(b'/')
.add(b'&')
.add(b'=')
.add(b'+')
.add(b'%');
const SOURCE_NAME: &str = "github-repo";
const USER_AGENT: &str = concat!(
"moments/",
env!("CARGO_PKG_VERSION"),
" (+https://rob.tn)"
);
const MAX_BACKFILL_PAGES: u32 = 100;
#[derive(Clone, Debug)]
pub struct GithubRepoConfig {
pub user: String,
pub token: Option<String>,
pub per_page: u32,
}
impl Default for GithubRepoConfig {
fn default() -> Self {
Self {
user: "grenade".into(),
token: None,
per_page: 100,
}
}
}
pub struct GithubRepoSource {
client: Client,
writer: Arc<dyn EventWriter>,
state: Arc<dyn PollerStateStore>,
config: GithubRepoConfig,
}
impl GithubRepoSource {
pub fn new(
client: Client,
writer: Arc<dyn EventWriter>,
state: Arc<dyn PollerStateStore>,
config: GithubRepoConfig,
) -> Self {
Self {
client,
writer,
state,
config,
}
}
fn apply_headers(&self, mut req: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
req = req
.header(header::ACCEPT, "application/vnd.github+json")
.header("X-GitHub-Api-Version", "2022-11-28")
.header(header::USER_AGENT, USER_AGENT);
if let Some(token) = &self.config.token {
req = req.header(header::AUTHORIZATION, format!("Bearer {token}"));
}
req
}
/// Discover all repos the authenticated user can access.
async fn discover_repos(&self) -> Result<Vec<Repo>, SourceError> {
if self.config.token.is_none() {
return Ok(vec![]);
}
let mut repos = Vec::new();
for page in 1..=50 {
let url = format!(
"https://api.github.com/user/repos?affiliation=owner,collaborator,organization_member&visibility=all&per_page={}&page={}",
self.config.per_page, page
);
let req = self.apply_headers(self.client.get(&url));
let resp = req
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
if !resp.status().is_success() {
return Err(SourceError::Http(format!("{} GET {}", resp.status(), url)));
}
let items: Vec<Value> = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
if items.is_empty() {
break;
}
for item in &items {
if let Some(r) = parse_repo(item) {
repos.push(r);
}
}
if items.len() < self.config.per_page as usize {
break;
}
}
// Supplement with repos from GraphQL repositoriesContributedTo.
// This catches repos where the user contributed via PRs but isn't
// an owner, collaborator, or org member — no result cap.
let mut known: HashSet<String> = repos.iter().map(|r| r.full_name.clone()).collect();
let contributed = self.discover_contributed_repos().await;
match contributed {
Ok(extra) => {
for r in extra {
if known.insert(r.full_name.clone()) {
repos.push(r);
}
}
}
Err(e) => {
warn!(error = %e, "GraphQL contributed-repos discovery failed; continuing with known repos");
}
}
Ok(repos)
}
/// Discover repos the user has contributed to via GraphQL.
/// Uses cursor-based pagination with no result cap.
async fn discover_contributed_repos(&self) -> Result<Vec<Repo>, SourceError> {
let token = match &self.config.token {
Some(t) => t,
None => return Ok(vec![]),
};
let mut repos = Vec::new();
let mut cursor: Option<String> = None;
loop {
let after = match &cursor {
Some(c) => format!(", after: \"{}\"", c),
None => String::new(),
};
let query = format!(
r#"{{ user(login: "{}") {{ repositoriesContributedTo(first: 100, contributionTypes: [COMMIT, PULL_REQUEST, ISSUE]{}) {{ pageInfo {{ hasNextPage endCursor }} nodes {{ nameWithOwner isPrivate }} }} }} }}"#,
self.config.user, after
);
let body = serde_json::json!({ "query": query });
let resp = self
.client
.post("https://api.github.com/graphql")
.header(header::AUTHORIZATION, format!("Bearer {token}"))
.header(header::USER_AGENT, USER_AGENT)
.header(header::CONTENT_TYPE, "application/json")
.json(&body)
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
if !resp.status().is_success() {
return Err(SourceError::Http(format!(
"{} POST graphql",
resp.status()
)));
}
let data: Value = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
// Check for GraphQL-level errors
if let Some(errors) = data.get("errors").and_then(Value::as_array) {
if let Some(msg) = errors.first().and_then(|e| e.get("message")).and_then(Value::as_str) {
return Err(SourceError::Http(format!("GraphQL error: {msg}")));
}
}
let contributed = &data["data"]["user"]["repositoriesContributedTo"];
let nodes = contributed["nodes"].as_array();
if let Some(nodes) = nodes {
for node in nodes {
let full_name = node
.get("nameWithOwner")
.and_then(Value::as_str);
let private = node
.get("isPrivate")
.and_then(Value::as_bool)
.unwrap_or(false);
if let Some(name) = full_name {
repos.push(Repo {
full_name: name.to_string(),
private,
});
}
}
}
let has_next = contributed["pageInfo"]["hasNextPage"]
.as_bool()
.unwrap_or(false);
if !has_next {
break;
}
cursor = contributed["pageInfo"]["endCursor"]
.as_str()
.map(String::from);
}
debug!(repos = repos.len(), "discovered contributed repos via GraphQL");
Ok(repos)
}
/// Branch discovery via GraphQL, filtered to branches whose HEAD
/// commit was authored by the user. Skips the long tail of
/// upstream-contributor branches in large forks (e.g. azure-docs).
///
/// Why HEAD author and not `history(author:).totalCount`: the latter
/// forces GraphQL to walk full commit history per branch looking for
/// matches, which times out (502) on forks with thousands of branches.
/// Checking the HEAD commit's author is O(1) per branch. The blind
/// spot — branches with the user's older commits but a different
/// HEAD author — is rare in practice for forks/feature branches.
///
/// On any GraphQL failure, callers should fall back to `list_branches`
/// (REST, walks everything; 500s from empty branches are silenced
/// inside `scan_repo_branch`).
async fn list_branches_with_commits(
&self,
repo: &Repo,
user_login: &str,
) -> Result<Vec<String>, SourceError> {
let token = match &self.config.token {
Some(t) => t,
None => return Err(SourceError::Http("no token; graphql unavailable".into())),
};
let parts: Vec<&str> = repo.full_name.splitn(2, '/').collect();
if parts.len() != 2 {
return Ok(Vec::new());
}
let (owner, name) = (parts[0], parts[1]);
let mut branches = Vec::new();
let mut cursor: Option<String> = None;
// Cap pages to bound cost on pathological repos. 50 pages × 100
// branches = 5000; well past anything plausible for a human user.
for _ in 0..50u32 {
let after = match &cursor {
Some(c) => format!(", after: \"{}\"", c),
None => String::new(),
};
// `author.user.login` resolves the commit's GitHub user (may
// differ from the raw commit author name); falling back to
// `author.email` is intentionally omitted to keep the query
// shape minimal — false negatives there are caught by the
// REST fallback on the next poll cycle.
let query = format!(
r#"{{ repository(owner: "{owner}", name: "{name}") {{ refs(refPrefix: "refs/heads/", first: 100{after}) {{ pageInfo {{ hasNextPage endCursor }} nodes {{ name target {{ ... on Commit {{ author {{ user {{ login }} }} }} }} }} }} }} }}"#,
);
let body = serde_json::json!({ "query": query });
let resp = self
.client
.post("https://api.github.com/graphql")
.header(header::AUTHORIZATION, format!("Bearer {token}"))
.header(header::USER_AGENT, USER_AGENT)
.header(header::CONTENT_TYPE, "application/json")
.json(&body)
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
if !resp.status().is_success() {
return Err(SourceError::Http(format!(
"{} POST graphql (branches {}/{})",
resp.status(),
owner,
name
)));
}
let data: Value = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
if let Some(errors) = data.get("errors").and_then(Value::as_array) {
if let Some(msg) = errors.first().and_then(|e| e.get("message")).and_then(Value::as_str) {
return Err(SourceError::Http(format!("GraphQL error listing branches: {msg}")));
}
}
let refs = &data["data"]["repository"]["refs"];
if refs.is_null() {
// Repo may be deleted or inaccessible — treat as empty.
return Ok(Vec::new());
}
if let Some(nodes) = refs["nodes"].as_array() {
for node in nodes {
let branch = node["name"].as_str();
let head_login = node["target"]["author"]["user"]["login"].as_str();
if let (Some(b), Some(login)) = (branch, head_login) {
if login.eq_ignore_ascii_case(user_login) {
branches.push(b.to_string());
}
}
}
}
let has_next = refs["pageInfo"]["hasNextPage"].as_bool().unwrap_or(false);
if !has_next {
break;
}
cursor = refs["pageInfo"]["endCursor"].as_str().map(String::from);
}
Ok(branches)
}
/// List every branch in a repo. Returns an empty vec for empty (409)
/// or missing (404) repos; surfaces rate-limit / transport errors so the
/// caller can decide whether to bail.
async fn list_branches(&self, repo: &Repo) -> Result<Vec<String>, SourceError> {
let mut branches = Vec::new();
for page in 1..=10u32 {
let url = format!(
"https://api.github.com/repos/{}/branches?per_page={}&page={}",
repo.full_name, self.config.per_page, page
);
let req = self.apply_headers(self.client.get(&url));
let resp = req
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
let status = resp.status();
if status.as_u16() == 404 || status.as_u16() == 409 {
return Ok(Vec::new());
}
if status.as_u16() == 403 || status.as_u16() == 429 {
return Err(SourceError::Http(format!("{} GET {}", status, url)));
}
if !status.is_success() {
return Err(SourceError::Http(format!("{} GET {}", status, url)));
}
let items: Vec<Value> = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
if items.is_empty() {
break;
}
for item in &items {
if let Some(name) = item.get("name").and_then(Value::as_str) {
branches.push(name.to_string());
}
}
if items.len() < self.config.per_page as usize {
break;
}
}
Ok(branches)
}
/// Fetch commits for a single repo across all branches the user has
/// touched. Per-branch state keys (`github-repo:{full_name}@{branch}`)
/// hold the newest seen commit timestamp so each branch can be
/// incremented independently — important because a brand new branch's
/// `since` cursor must start unset even when the default branch has
/// been polled many times already.
///
/// When `user_id` is supplied, branches are pre-filtered via GraphQL
/// to those with at least one commit by the user — vastly cheaper for
/// large upstream forks where most branches were never touched. On
/// GraphQL failure (or no token), falls back to the REST branch list
/// and relies on the per-branch 500-as-empty handling to discard the
/// noise.
async fn scan_repo(&self, repo: &Repo) -> Result<usize, SourceError> {
let branches = if self.config.token.is_some() {
match self.list_branches_with_commits(repo, &self.config.user).await {
Ok(b) => b,
Err(e) => {
warn!(repo = %repo.full_name, error = %e, "graphql branch filter failed; falling back to REST");
self.list_branches(repo).await?
}
}
} else {
self.list_branches(repo).await?
};
if branches.is_empty() {
return Ok(0);
}
let mut total = 0usize;
// Dedup commits seen via multiple branches in one tick. Without this
// the same SHA appears in the upsert batch twice (postgres rejects
// duplicate conflict targets in a single INSERT).
let mut seen_in_tick: HashSet<String> = HashSet::new();
for branch in &branches {
match self.scan_repo_branch(repo, branch, &mut seen_in_tick).await {
Ok(n) => total += n,
Err(SourceError::Http(ref msg)) if msg.starts_with("403") || msg.starts_with("429") => {
return Err(SourceError::Http(msg.clone()));
}
Err(e) => {
warn!(repo = %repo.full_name, branch = %branch, error = %e, "branch scan failed; continuing");
}
}
}
Ok(total)
}
async fn scan_repo_branch(
&self,
repo: &Repo,
branch: &str,
seen_in_tick: &mut HashSet<String>,
) -> Result<usize, SourceError> {
let state_key = format!("github-repo:{}@{}", repo.full_name, branch);
let prior = self.state.load(&state_key).await?;
let since = prior.as_ref().and_then(|s| s.last_modified);
let encoded_branch = utf8_percent_encode(branch, BRANCH_ENCODE_SET).to_string();
let mut total = 0usize;
let mut newest: Option<DateTime<Utc>> = since;
for page in 1..=MAX_BACKFILL_PAGES {
let mut url = format!(
"https://api.github.com/repos/{}/commits?author={}&sha={}&per_page={}&page={}",
repo.full_name, self.config.user, encoded_branch, self.config.per_page, page
);
if let Some(since_dt) = since {
url.push_str(&format!("&since={}", since_dt.to_rfc3339()));
}
let req = self.apply_headers(self.client.get(&url));
let resp = req
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
let status = resp.status();
// 409 = empty repo (no commits at all), not an error
if status.as_u16() == 409 {
break;
}
if status.as_u16() == 403 || status.as_u16() == 429 {
warn!(repo = %repo.full_name, branch = %branch, status = %status, "rate limited; stopping early");
return Err(SourceError::Http(format!("{} GET {}", status, url)));
}
if status.as_u16() == 404 {
warn!(repo = %repo.full_name, branch = %branch, "repo or branch not found; skipping");
break;
}
// GitHub's `/repos/.../commits?author=X&sha=branch` returns 500
// (not an empty array) when the user has zero commits on the
// specified branch. Treat it as "no commits on this branch"
// rather than a server error — surfacing it as a warning floods
// logs on forks whose branches were all authored by upstream.
if status.as_u16() == 500 {
debug!(repo = %repo.full_name, branch = %branch, "no commits by author on branch (500)");
break;
}
if !status.is_success() {
return Err(SourceError::Http(format!("{} GET {}", status, url)));
}
let items: Vec<Value> = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
if items.is_empty() {
break;
}
let mut events = Vec::with_capacity(items.len());
for item in &items {
if let Some(ev) = parse_commit(item, repo) {
if seen_in_tick.insert(ev.id.clone()) {
if let Some(n) = newest {
if ev.occurred_at > n {
newest = Some(ev.occurred_at);
}
} else {
newest = Some(ev.occurred_at);
}
events.push(ev);
} else {
// Already ingested via another branch this tick;
// still advance `newest` so the per-branch cursor
// doesn't get stuck behind shared history.
let occurred = parse_commit_date(item);
if let Some(t) = occurred {
newest = Some(match newest {
Some(n) if t > n => t,
Some(n) => n,
None => t,
});
}
}
}
}
total += self.writer.upsert_events(&events).await?;
if items.len() < self.config.per_page as usize {
break;
}
}
self.state.save(&state_key, None, newest).await?;
Ok(total)
}
/// Batch-fetch language breakdowns for repos via GraphQL, upserting
/// into repo_languages. Repos are batched using GraphQL aliases to
/// minimise round trips.
async fn fetch_languages(&self, repos: &[Repo]) -> Result<usize, SourceError> {
let token = match &self.config.token {
Some(t) => t,
None => return Ok(0),
};
let mut total = 0usize;
for chunk in repos.chunks(20) {
let mut fragments = Vec::with_capacity(chunk.len());
for (i, repo) in chunk.iter().enumerate() {
let parts: Vec<&str> = repo.full_name.splitn(2, '/').collect();
if parts.len() != 2 {
continue;
}
fragments.push(format!(
r#"r{i}: repository(owner: "{}", name: "{}") {{ languages(first: 20, orderBy: {{field: SIZE, direction: DESC}}) {{ edges {{ size node {{ name color }} }} }} }}"#,
parts[0], parts[1]
));
}
if fragments.is_empty() {
continue;
}
let query = format!("{{ {} }}", fragments.join(" "));
let body = serde_json::json!({ "query": query });
let resp = self
.client
.post("https://api.github.com/graphql")
.header(header::AUTHORIZATION, format!("Bearer {token}"))
.header(header::USER_AGENT, USER_AGENT)
.header(header::CONTENT_TYPE, "application/json")
.json(&body)
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
if !resp.status().is_success() {
warn!(status = %resp.status(), "GraphQL language fetch failed");
break;
}
let data: Value = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
if let Some(errors) = data.get("errors").and_then(Value::as_array) {
if let Some(msg) = errors.first().and_then(|e| e.get("message")).and_then(Value::as_str) {
warn!(error = %msg, "GraphQL language fetch had errors");
}
}
let data_obj = match data.get("data") {
Some(d) => d,
None => continue,
};
let mut languages = Vec::new();
for (i, repo) in chunk.iter().enumerate() {
let alias = format!("r{i}");
let edges = data_obj
.get(&alias)
.and_then(|r| r.get("languages"))
.and_then(|l| l.get("edges"))
.and_then(Value::as_array);
if let Some(edges) = edges {
for edge in edges {
let size = edge.get("size").and_then(Value::as_i64).unwrap_or(0);
let name = edge
.get("node")
.and_then(|n| n.get("name"))
.and_then(Value::as_str);
let color = edge
.get("node")
.and_then(|n| n.get("color"))
.and_then(Value::as_str);
if let Some(name) = name {
languages.push(RepoLanguage {
source: Source::Github,
repo: repo.full_name.clone(),
language: name.to_string(),
bytes: size,
color: color.map(String::from),
});
}
}
}
}
total += self.writer.upsert_repo_languages(&languages).await?;
}
debug!(total, "repo languages updated");
Ok(total)
}
}
#[async_trait]
impl EventSource for GithubRepoSource {
fn name(&self) -> &'static str {
SOURCE_NAME
}
async fn poll(&self) -> Result<usize, SourceError> {
let repos = self.discover_repos().await?;
debug!(repos = repos.len(), "discovered github repos");
let mut total = 0usize;
for repo in &repos {
match self.scan_repo(repo).await {
Ok(n) => {
if n > 0 {
debug!(repo = %repo.full_name, ingested = n, "repo commit scan complete");
}
total += n;
}
Err(SourceError::Http(ref msg)) if msg.starts_with("403") || msg.starts_with("429") => {
warn!("rate limited during repo scan; ending poll early");
break;
}
Err(e) => {
warn!(repo = %repo.full_name, error = %e, "repo scan failed; continuing");
}
}
}
if let Err(e) = self.fetch_languages(&repos).await {
warn!(error = %e, "language fetch failed; continuing");
}
self.state.touch(SOURCE_NAME).await?;
debug!(ingested = total, repos = repos.len(), "github-repo poll complete");
Ok(total)
}
}
#[derive(Debug, Clone)]
struct Repo {
full_name: String,
private: bool,
}
fn parse_repo(item: &Value) -> Option<Repo> {
let full_name = item.get("full_name").and_then(Value::as_str)?;
let private = item.get("private").and_then(Value::as_bool).unwrap_or(false);
Some(Repo {
full_name: full_name.to_string(),
private,
})
}
fn parse_commit_date(item: &Value) -> Option<DateTime<Utc>> {
let date_str = item
.get("commit")
.and_then(|c| c.get("author"))
.and_then(|a| a.get("date"))
.and_then(Value::as_str)
.or_else(|| {
item.get("commit")
.and_then(|c| c.get("committer"))
.and_then(|c| c.get("date"))
.and_then(Value::as_str)
})?;
Some(
DateTime::parse_from_rfc3339(date_str)
.ok()?
.with_timezone(&Utc),
)
}
fn parse_commit(item: &Value, repo: &Repo) -> Option<Event> {
let sha = item.get("sha").and_then(Value::as_str)?;
let occurred_at = parse_commit_date(item)?;
let mut payload = item.clone();
if let Some(obj) = payload.as_object_mut() {
obj.insert("_repo".into(), Value::String(repo.full_name.clone()));
}
Some(Event {
id: format!("github-commit:{sha}"),
source: Source::Github,
action: "Commit".into(),
occurred_at,
public: !repo.private,
payload,
})
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn parse_commit_uses_sha_as_id() {
let repo = Repo {
full_name: "grenade/moments".into(),
private: false,
};
let raw = json!({
"sha": "abc123",
"commit": {
"author": { "date": "2024-01-15T10:30:00Z" },
"message": "fix something"
}
});
let ev = parse_commit(&raw, &repo).expect("parses");
assert_eq!(ev.id, "github-commit:abc123");
assert_eq!(ev.action, "Commit");
assert!(ev.public);
}
#[test]
fn parse_commit_private_repo() {
let repo = Repo {
full_name: "grenade/secret".into(),
private: true,
};
let raw = json!({
"sha": "def456",
"commit": {
"author": { "date": "2024-01-15T10:30:00Z" },
"message": "secret change"
}
});
let ev = parse_commit(&raw, &repo).expect("parses");
assert!(!ev.public);
}
#[test]
fn parse_commit_falls_back_to_committer_date() {
let repo = Repo {
full_name: "grenade/moments".into(),
private: false,
};
let raw = json!({
"sha": "ghi789",
"commit": {
"committer": { "date": "2024-02-01T12:00:00Z" },
"message": "no author date"
}
});
let ev = parse_commit(&raw, &repo).expect("parses");
assert_eq!(ev.id, "github-commit:ghi789");
}
#[test]
fn parse_repo_extracts_fields() {
let raw = json!({
"full_name": "grenade/moments",
"private": false
});
let repo = parse_repo(&raw).expect("parses");
assert_eq!(repo.full_name, "grenade/moments");
assert!(!repo.private);
}
}

411
crates/moments-data/src/github_search.rs Normal file
View File

@@ -0,0 +1,411 @@
//! GitHub Search API ingestion for historical backfill.
//!
//! The Events API caps at 90 days; this source uses `/search/issues` and
//! `/search/commits` with `author:<user>` to recover issues, PRs, and
//! commits going back as far as GitHub retains them (1000-result ceiling
//! per query is the Search API's hard cap).
//!
//! Fork duplication on /search/commits — the same commit SHA appears in
//! every fork that still contains it — is handled by:
//! * deduplicating by `id = github-commit:<sha>` within each batch
//! before upsert (postgres ON CONFLICT errors if the same conflict
//! target appears twice in one INSERT);
//! * upserting with last-write-wins across batches and runs (the SHA
//! is the same; the repo association may flip between forks but the
//! commit itself is identical).
use std::collections::{HashMap, HashSet};
use std::sync::Arc;
use async_trait::async_trait;
use chrono::{DateTime, Utc};
use moments_core::{EventSource, EventWriter, PollerStateStore, SourceError};
use moments_entities::{Event, Source};
use reqwest::{Client, header};
use serde_json::Value;
use tracing::{debug, warn};
const SOURCE_NAME: &str = "github-search";
const USER_AGENT: &str = concat!(
"moments/",
env!("CARGO_PKG_VERSION"),
" (+https://rob.tn)"
);
#[derive(Clone, Debug)]
pub struct GithubSearchConfig {
pub user: String,
pub token: Option<String>,
pub per_page: u32,
/// Hard cap on pages walked per query. The Search API itself only returns
/// the first 1000 results across pages, so 10 × 100 covers everything.
pub max_pages: u32,
}
impl Default for GithubSearchConfig {
fn default() -> Self {
Self {
user: "grenade".into(),
token: None,
per_page: 100,
max_pages: 10,
}
}
}
pub struct GithubSearchSource {
client: Client,
writer: Arc<dyn EventWriter>,
state: Arc<dyn PollerStateStore>,
config: GithubSearchConfig,
}
impl GithubSearchSource {
pub fn new(
client: Client,
writer: Arc<dyn EventWriter>,
state: Arc<dyn PollerStateStore>,
config: GithubSearchConfig,
) -> Self {
Self {
client,
writer,
state,
config,
}
}
fn apply_headers(&self, mut req: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
req = req
.header(header::ACCEPT, "application/vnd.github+json")
.header("X-GitHub-Api-Version", "2022-11-28")
.header(header::USER_AGENT, USER_AGENT);
if let Some(token) = &self.config.token {
req = req.header(header::AUTHORIZATION, format!("Bearer {token}"));
}
req
}
/// Read repo visibility from `/repos/{full_name}`. Used for results from
/// /search/issues, which don't include the visibility flag inline.
async fn fetch_repo_private(&self, full_name: &str) -> Result<bool, SourceError> {
let url = format!("https://api.github.com/repos/{full_name}");
let req = self.apply_headers(self.client.get(&url));
let resp = req
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
if !resp.status().is_success() {
// Repo may be deleted / inaccessible. Treat as private (safer:
// we'd rather under-expose than over-expose).
return Err(SourceError::Http(format!("{} GET {}", resp.status(), url)));
}
let v: Value = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
Ok(v.get("private").and_then(Value::as_bool).unwrap_or(false))
}
async fn search_commits(
&self,
vis_cache: &mut HashMap<String, bool>,
) -> Result<usize, SourceError> {
let mut total = 0usize;
for page in 1..=self.config.max_pages {
// `fork:true` opts forks into the search — by default GitHub's
// search API excludes them entirely, which means commits on a
// user's fork (regardless of branch) never surface here.
let url = format!(
"https://api.github.com/search/commits?q=author:{}+fork:true&sort=author-date&order=desc&per_page={}&page={}",
self.config.user, self.config.per_page, page
);
let req = self.apply_headers(self.client.get(&url));
let resp = req
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
if !resp.status().is_success() {
return Err(SourceError::Http(format!("{} GET {}", resp.status(), url)));
}
let body: Value = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
let items = body
.get("items")
.and_then(Value::as_array)
.cloned()
.unwrap_or_default();
if items.is_empty() {
break;
}
// Dedup within the page by id (same commit in multiple forks
// returns multiple search items with the same SHA — postgres
// refuses ON CONFLICT when the conflict target appears twice
// in one INSERT).
let mut seen: HashSet<String> = HashSet::new();
let mut events = Vec::with_capacity(items.len());
for item in &items {
if let Some(ev) = parse_commit_event(item) {
if seen.insert(ev.id.clone()) {
// Opportunistically populate the visibility cache so
// search_issues can reuse it for the same repos.
if let Some(repo) = item
.get("repository")
.and_then(|r| r.get("full_name"))
.and_then(Value::as_str)
{
vis_cache.insert(repo.to_string(), !ev.public);
}
events.push(ev);
}
}
}
total += self.writer.upsert_events(&events).await?;
if items.len() < self.config.per_page as usize {
break;
}
}
Ok(total)
}
async fn search_issues(
&self,
vis_cache: &mut HashMap<String, bool>,
) -> Result<usize, SourceError> {
let mut total = 0usize;
for page in 1..=self.config.max_pages {
let url = format!(
"https://api.github.com/search/issues?q=author:{}&sort=created&order=desc&per_page={}&page={}",
self.config.user, self.config.per_page, page
);
let req = self.apply_headers(self.client.get(&url));
let resp = req
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
if !resp.status().is_success() {
return Err(SourceError::Http(format!("{} GET {}", resp.status(), url)));
}
let body: Value = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
let items = body
.get("items")
.and_then(Value::as_array)
.cloned()
.unwrap_or_default();
if items.is_empty() {
break;
}
let mut events = Vec::with_capacity(items.len());
for item in &items {
if let Some(ev) = self.search_issue_to_event(item, vis_cache).await {
events.push(ev);
}
}
total += self.writer.upsert_events(&events).await?;
// Last page if we got fewer than per_page items.
if items.len() < self.config.per_page as usize {
break;
}
}
Ok(total)
}
async fn search_issue_to_event(
&self,
item: &Value,
vis_cache: &mut HashMap<String, bool>,
) -> Option<Event> {
let number = item.get("number").and_then(Value::as_i64)?;
let html_url = item.get("html_url").and_then(Value::as_str)?;
let created_at_str = item.get("created_at").and_then(Value::as_str)?;
let occurred_at = DateTime::parse_from_rfc3339(created_at_str)
.ok()?
.with_timezone(&Utc);
let repo = repo_from_html_url(html_url)?;
let private = match vis_cache.get(&repo).copied() {
Some(p) => p,
None => match self.fetch_repo_private(&repo).await {
Ok(p) => {
vis_cache.insert(repo.clone(), p);
p
}
Err(e) => {
warn!(repo = %repo, error = %e, "repo visibility lookup failed; treating as private");
vis_cache.insert(repo.clone(), true);
true
}
},
};
let action = if item.get("pull_request").is_some() {
"PullRequest"
} else {
"Issue"
};
Some(Event {
id: format!("github-issue:{repo}#{number}"),
source: Source::Github,
action: action.into(),
occurred_at,
public: !private,
payload: item.clone(),
})
}
}
#[async_trait]
impl EventSource for GithubSearchSource {
fn name(&self) -> &'static str {
SOURCE_NAME
}
async fn poll(&self) -> Result<usize, SourceError> {
let mut vis_cache: HashMap<String, bool> = HashMap::new();
// Run commits first so vis_cache is partly seeded with inline-flag
// visibility before the issue loop hits its (more expensive) per-repo
// lookups.
let commits = self.search_commits(&mut vis_cache).await?;
let issues = self.search_issues(&mut vis_cache).await?;
self.state.touch(SOURCE_NAME).await?;
debug!(
commits,
issues,
unique_repos = vis_cache.len(),
"github-search poll complete"
);
Ok(commits + issues)
}
}
/// Convert a /search/commits item into our Event row. Returns None if the
/// item is missing required fields.
fn parse_commit_event(item: &Value) -> Option<Event> {
let sha = item.get("sha").and_then(Value::as_str)?;
let html_url = item.get("html_url").and_then(Value::as_str)?;
// Prefer author.date — it's when the work was written; committer.date
// can shift on rebase. Either is RFC3339.
let date_str = item
.get("commit")
.and_then(|c| c.get("author"))
.and_then(|a| a.get("date"))
.and_then(Value::as_str)
.or_else(|| {
item.get("commit")
.and_then(|c| c.get("committer"))
.and_then(|c| c.get("date"))
.and_then(Value::as_str)
})?;
let occurred_at = DateTime::parse_from_rfc3339(date_str)
.ok()?
.with_timezone(&Utc);
let private = item
.get("repository")
.and_then(|r| r.get("private"))
.and_then(Value::as_bool)
.unwrap_or(false);
// Sanity-check the html_url points at github.com so we don't ingest
// garbage if GitHub ever changes its URL shape.
if !html_url.starts_with("https://github.com/") {
return None;
}
Some(Event {
id: format!("github-commit:{sha}"),
source: Source::Github,
action: "Commit".into(),
occurred_at,
public: !private,
payload: item.clone(),
})
}
/// Extract `owner/repo` from a github.com URL like
/// `https://github.com/owner/repo/{issues,pull}/42`.
fn repo_from_html_url(url: &str) -> Option<String> {
let stripped = url.strip_prefix("https://github.com/")?;
let mut parts = stripped.splitn(3, '/');
let owner = parts.next()?;
let repo = parts.next()?;
if owner.is_empty() || repo.is_empty() {
return None;
}
Some(format!("{owner}/{repo}"))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn extracts_repo_from_html_url() {
assert_eq!(
repo_from_html_url("https://github.com/Nehliin/vortex/issues/125").as_deref(),
Some("Nehliin/vortex")
);
assert_eq!(
repo_from_html_url("https://github.com/grenade/moments/pull/3").as_deref(),
Some("grenade/moments")
);
}
#[test]
fn rejects_non_github_host() {
assert!(repo_from_html_url("https://gitlab.com/x/y/-/issues/1").is_none());
}
#[test]
fn parse_commit_uses_sha_as_id() {
let raw = serde_json::json!({
"sha": "a6fcefbe909a97ad5a049b9fa48bc74309af10d9",
"html_url": "https://github.com/faith1337z/Trade/commit/a6fcefbe909a97ad5a049b9fa48bc74309af10d9",
"commit": {
"author": { "name": "rob", "date": "2017-11-13T23:32:31+02:00" },
"committer": { "name": "rob", "date": "2017-11-13T22:32:31+01:00" },
"message": "split multiline message into multiple irc messages"
},
"repository": { "full_name": "faith1337z/Trade", "private": false }
});
let ev = parse_commit_event(&raw).expect("parses");
assert_eq!(ev.id, "github-commit:a6fcefbe909a97ad5a049b9fa48bc74309af10d9");
assert_eq!(ev.action, "Commit");
assert!(ev.public);
}
#[test]
fn parse_commit_marks_private_repo() {
let raw = serde_json::json!({
"sha": "deadbeef",
"html_url": "https://github.com/grenade/private-repo/commit/deadbeef",
"commit": {
"author": { "date": "2024-01-01T00:00:00Z" },
"message": "x"
},
"repository": { "full_name": "grenade/private-repo", "private": true }
});
let ev = parse_commit_event(&raw).expect("parses");
assert!(!ev.public);
}
#[test]
fn parse_commit_rejects_non_github_url() {
let raw = serde_json::json!({
"sha": "abc",
"html_url": "https://example.com/x/commit/abc",
"commit": { "author": { "date": "2024-01-01T00:00:00Z" } },
"repository": { "full_name": "x/y", "private": false }
});
assert!(parse_commit_event(&raw).is_none());
}
}

279
crates/moments-data/src/hg.rs Normal file
View File

@@ -0,0 +1,279 @@
//! hg-edge.mozilla.org changeset ingestion via `json-log` revset queries.
//!
//! Uses the `json-log?rev=author(term)` endpoint which returns changesets
//! by the *author* (not the pusher), so it captures commits landed by
//! sheriffs on behalf of the contributor.
//!
//! Repos are discovered within configured groups (e.g. `build`) via the
//! `/{group}/?style=json` index, plus any individually listed repos
//! (e.g. `mozilla-central`). Once the first successful scan completes
//! (poller state is touched), all subsequent polls are skipped — the
//! data is historical and will not change.
use std::sync::Arc;
use async_trait::async_trait;
use chrono::{DateTime, TimeZone, Utc};
use moments_core::{EventSource, EventWriter, PollerStateStore, SourceError};
use moments_entities::{Event, Source};
use reqwest::{Client, header};
use serde_json::Value;
use tracing::{debug, warn};
const SOURCE_NAME: &str = "hg";
const USER_AGENT: &str = concat!(
"moments/",
env!("CARGO_PKG_VERSION"),
" (+https://rob.tn)"
);
/// Maximum changesets returned per json-log request.
const REV_COUNT: u32 = 500;
#[derive(Clone, Debug)]
pub struct HgConfig {
pub host: String,
/// Substrings matched via `author(term)` revset queries.
pub author_terms: Vec<String>,
/// Repo groups to scan — each is enumerated via `/{group}/?style=json`.
pub groups: Vec<String>,
/// Individual repos to scan (e.g. `mozilla-central`).
pub repos: Vec<String>,
}
impl Default for HgConfig {
fn default() -> Self {
Self {
host: "hg-edge.mozilla.org".into(),
author_terms: vec!["rthijssen".into(), "grenade".into()],
groups: vec!["build".into(), "integration".into()],
repos: vec!["mozilla-central".into()],
}
}
}
pub struct HgSource {
client: Client,
writer: Arc<dyn EventWriter>,
state: Arc<dyn PollerStateStore>,
config: HgConfig,
}
impl HgSource {
pub fn new(
client: Client,
writer: Arc<dyn EventWriter>,
state: Arc<dyn PollerStateStore>,
config: HgConfig,
) -> Self {
Self {
client,
writer,
state,
config,
}
}
/// Discover repos in a group via `/{group}/?style=json`.
async fn discover_repos(&self, group: &str) -> Result<Vec<String>, SourceError> {
let url = format!("https://{}/{}/?style=json", self.config.host, group);
let resp = self
.client
.get(&url)
.header(header::USER_AGENT, USER_AGENT)
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
if !resp.status().is_success() {
warn!(group, status = %resp.status(), "failed to discover repos in group");
return Ok(vec![]);
}
let body: Value = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
Ok(body
.get("entries")
.and_then(Value::as_array)
.map(|entries| {
entries
.iter()
.filter_map(|e| {
e.get("name")
.and_then(Value::as_str)
.map(|name| format!("{group}/{name}"))
})
.collect()
})
.unwrap_or_default())
}
fn log_url(&self, repo: &str, author_term: &str) -> String {
format!(
"https://{}/{}/json-log?rev=author({})&style=json&revcount={}",
self.config.host, repo, author_term, REV_COUNT
)
}
async fn scan_repo(&self, repo: &str) -> Result<usize, SourceError> {
let mut all_events = Vec::new();
for term in &self.config.author_terms {
let url = self.log_url(repo, term);
let resp = self
.client
.get(&url)
.header(header::USER_AGENT, USER_AGENT)
.send()
.await
.map_err(|e| SourceError::Http(e.to_string()))?;
if !resp.status().is_success() {
return Err(SourceError::Http(format!("{} GET {}", resp.status(), url)));
}
let body: Value = resp
.json()
.await
.map_err(|e| SourceError::Parse(e.to_string()))?;
if let Some(entries) = body.get("entries").and_then(Value::as_array) {
for entry in entries {
let node = entry.get("node").and_then(Value::as_str).unwrap_or("");
if node.is_empty() {
continue;
}
let occurred_at = entry
.get("date")
.and_then(Value::as_array)
.and_then(|a| parse_hg_date(a))
.unwrap_or_else(Utc::now);
let mut payload = entry.clone();
if let Some(obj) = payload.as_object_mut() {
obj.insert("_repo".into(), Value::String(repo.into()));
obj.insert(
"_host".into(),
Value::String(self.config.host.clone()),
);
}
all_events.push(Event {
id: format!("hg:{repo}:{node}"),
source: Source::Hg,
action: "Commit".into(),
occurred_at,
public: true,
payload,
});
}
}
}
Ok(self.writer.upsert_events(&all_events).await?)
}
}
#[async_trait]
impl EventSource for HgSource {
fn name(&self) -> &'static str {
SOURCE_NAME
}
async fn poll(&self) -> Result<usize, SourceError> {
// hg repos are archived — one complete scan is sufficient.
if self.state.load(SOURCE_NAME).await?.is_some() {
debug!("hg already backfilled, skipping");
return Ok(0);
}
let mut repos: Vec<String> = self.config.repos.clone();
for group in &self.config.groups {
let discovered = self.discover_repos(group).await?;
debug!(group, repos = discovered.len(), "discovered hg repos");
repos.extend(discovered);
}
let mut total = 0usize;
for repo in &repos {
match self.scan_repo(repo).await {
Ok(n) => {
if n > 0 {
debug!(repo, ingested = n, "hg repo scan complete");
}
total += n;
}
Err(e) => {
warn!(repo, error = %e, "hg repo scan failed; continuing");
}
}
}
self.state.touch(SOURCE_NAME).await?;
debug!(ingested = total, "hg backfill complete");
Ok(total)
}
}
/// Parse a hgweb date array `[seconds, tz_offset_secs]` into UTC.
fn parse_hg_date(arr: &[Value]) -> Option<DateTime<Utc>> {
let secs = arr.first()?.as_f64()? as i64;
Utc.timestamp_opt(secs, 0).single()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parse_hg_date_handles_seconds() {
let arr = vec![Value::from(1_700_000_000_f64), Value::from(0_f64)];
let dt = parse_hg_date(&arr).expect("parses");
assert_eq!(dt.timestamp(), 1_700_000_000);
}
#[test]
fn log_url_uses_revset_author_query() {
let src = HgSource {
client: Client::new(),
writer: Arc::new(NoopWriter),
state: Arc::new(NoopState),
config: HgConfig::default(),
};
let url = src.log_url("mozilla-central", "thijssen");
assert!(url.contains("json-log?rev=author(thijssen)"));
assert!(url.contains("revcount=500"));
}
// Tiny stub impls just so we can construct an HgSource for unit tests.
struct NoopWriter;
#[async_trait]
impl EventWriter for NoopWriter {
async fn upsert_events(
&self,
_events: &[Event],
) -> Result<usize, moments_core::StoreError> {
Ok(0)
}
async fn upsert_repo_languages(
&self,
_languages: &[moments_entities::RepoLanguage],
) -> Result<usize, moments_core::StoreError> {
Ok(0)
}
}
struct NoopState;
#[async_trait]
impl PollerStateStore for NoopState {
async fn load(
&self,
_source: &str,
) -> Result<Option<moments_core::PollerState>, moments_core::StoreError> {
Ok(None)
}
async fn save(
&self,
_source: &str,
_etag: Option<&str>,
_last_modified: Option<DateTime<Utc>>,
) -> Result<(), moments_core::StoreError> {
Ok(())
}
async fn touch(&self, _source: &str) -> Result<(), moments_core::StoreError> {
Ok(())
}
}
}

312
crates/moments-data/src/lib.rs
View File

@@ -1,9 +1,15 @@
pub mod bugzilla;
pub mod gitea;
pub mod github;
pub mod github_repo;
pub mod github_search;
pub mod hg;
use async_trait::async_trait;
use chrono::{DateTime, Utc};
use moments_core::{EventReader, EventWriter, PollerState, PollerStateStore, StoreError};
use moments_entities::{Event, EventQuery, Source, SourceSummary};
use chrono::NaiveDate;
use moments_entities::{DailyCount, Event, EventQuery, HourlyAvg, LanguageDailyCount, ProjectSummary, RepoLanguage, Source, SourceSummary};
use sqlx::Row;
use sqlx::postgres::{PgPool, PgPoolOptions};
use std::str::FromStr;
@@ -43,19 +49,36 @@ impl EventReader for PgStore {
let rows = sqlx::query(
r#"
SELECT id, source, action, occurred_at, payload
SELECT id, source, action, occurred_at, public, payload
FROM events
WHERE ($1::timestamptz IS NULL OR occurred_at >= $1)
AND ($2::timestamptz IS NULL OR occurred_at < $2)
AND ($3::text[] IS NULL OR source = ANY($3))
AND ($4::bool OR public = true)
AND ($6::text IS NULL OR (CASE source
WHEN 'github' THEN COALESCE(
payload->'repo'->>'name',
payload->'repository'->>'full_name',
payload->>'_repo'
)
WHEN 'gitea' THEN COALESCE(
payload->'repo'->>'full_name',
payload->'repo'->>'name'
)
WHEN 'hg' THEN payload->>'_repo'
WHEN 'bugzilla' THEN payload->>'product'
ELSE NULL
END) = $6)
ORDER BY occurred_at DESC
LIMIT $4
LIMIT $5
"#,
)
.bind(query.from)
.bind(query.to)
.bind(sources.as_deref())
.bind(query.include_private)
.bind(query.limit as i64)
.bind(query.repo.as_deref())
.fetch_all(&self.pool)
.await
.map_err(map_err)?;
@@ -68,13 +91,14 @@ impl EventReader for PgStore {
source: Source::from_str(&source_str).map_err(map_err)?,
action: r.try_get("action").map_err(map_err)?,
occurred_at: r.try_get("occurred_at").map_err(map_err)?,
public: r.try_get("public").map_err(map_err)?,
payload: r.try_get("payload").map_err(map_err)?,
})
})
.collect()
}
async fn source_summaries(&self) -> Result<Vec<SourceSummary>, StoreError> {
async fn source_summaries(&self, include_private: bool) -> Result<Vec<SourceSummary>, StoreError> {
let rows = sqlx::query(
r#"
SELECT source,
@@ -82,10 +106,12 @@ impl EventReader for PgStore {
MIN(occurred_at) AS earliest,
MAX(occurred_at) AS latest
FROM events
WHERE $1::bool OR public = true
GROUP BY source
ORDER BY source
"#,
)
.bind(include_private)
.fetch_all(&self.pool)
.await
.map_err(map_err)?;
@@ -105,6 +131,245 @@ impl EventReader for PgStore {
})
.collect()
}
async fn list_projects(&self) -> Result<Vec<ProjectSummary>, StoreError> {
let rows = sqlx::query(
r#"
SELECT source, repo, host,
SUM(commits)::bigint AS commit_count,
SUM(issues)::bigint AS issue_count,
SUM(prs)::bigint AS pr_count,
MIN(occurred_at) AS first_activity,
MAX(occurred_at) AS last_activity
FROM (
SELECT source, occurred_at,
CASE source
WHEN 'github' THEN COALESCE(
payload->'repo'->>'name',
payload->'repository'->>'full_name',
payload->>'_repo'
)
WHEN 'gitea' THEN COALESCE(
payload->'repo'->>'full_name',
payload->'repo'->>'name'
)
WHEN 'hg' THEN payload->>'_repo'
WHEN 'bugzilla' THEN payload->>'product'
ELSE NULL
END AS repo,
CASE source
WHEN 'github' THEN 'github.com'
WHEN 'gitea' THEN COALESCE(payload->>'_host', 'git.lair.cafe')
WHEN 'hg' THEN COALESCE(payload->>'_host', 'hg-edge.mozilla.org')
WHEN 'bugzilla' THEN 'bugzilla.mozilla.org'
ELSE 'unknown'
END AS host,
CASE WHEN action IN ('Commit', 'PushEvent', 'commit_repo') THEN 1 ELSE 0 END AS commits,
CASE WHEN action IN ('Issue', 'IssuesEvent') THEN 1 ELSE 0 END AS issues,
CASE WHEN action IN ('PullRequest', 'PullRequestEvent') THEN 1 ELSE 0 END AS prs
FROM events
WHERE public = true
) sub
WHERE repo IS NOT NULL AND repo != ''
GROUP BY source, repo, host
ORDER BY MAX(occurred_at) DESC
"#,
)
.fetch_all(&self.pool)
.await
.map_err(map_err)?;
rows.into_iter()
.map(|r| {
let source_str: String = r.try_get("source").map_err(map_err)?;
Ok(ProjectSummary {
source: Source::from_str(&source_str).map_err(map_err)?,
repo: r.try_get("repo").map_err(map_err)?,
host: r.try_get("host").map_err(map_err)?,
commit_count: r.try_get::<i64, _>("commit_count").map_err(map_err).unwrap_or(0),
issue_count: r.try_get::<i64, _>("issue_count").map_err(map_err).unwrap_or(0),
pr_count: r.try_get::<i64, _>("pr_count").map_err(map_err).unwrap_or(0),
first_activity: r.try_get("first_activity").map_err(map_err)?,
last_activity: r.try_get("last_activity").map_err(map_err)?,
})
})
.collect()
}
async fn daily_counts(&self, from: NaiveDate, to: NaiveDate, include_private: bool) -> Result<Vec<DailyCount>, StoreError> {
let rows = sqlx::query(
r#"
SELECT d::date AS date,
COUNT(e.id)::bigint AS count
FROM generate_series($1::date, $2::date, '1 day') d
LEFT JOIN events e
ON e.occurred_at >= (d::date || 'T00:00:00Z')::timestamptz
AND e.occurred_at < ((d::date + 1) || 'T00:00:00Z')::timestamptz
AND ($3::bool OR e.public = true)
GROUP BY d::date
ORDER BY d::date
"#,
)
.bind(from)
.bind(to)
.bind(include_private)
.fetch_all(&self.pool)
.await
.map_err(map_err)?;
rows.into_iter()
.map(|r| {
Ok(DailyCount {
date: r.try_get("date").map_err(map_err)?,
count: r.try_get("count").map_err(map_err)?,
})
})
.collect()
}
async fn language_daily_counts(&self, from: NaiveDate, to: NaiveDate, include_private: bool) -> Result<Vec<LanguageDailyCount>, StoreError> {
let rows = sqlx::query(
r#"
SELECT date, language, color,
ROUND(SUM(weight))::bigint AS commits
FROM (
SELECT d::date AS date,
rl.language,
COALESCE(rl.color,
(SELECT color FROM repo_languages
WHERE language = rl.language AND color IS NOT NULL
LIMIT 1)
) AS color,
rl.bytes::float / NULLIF(rt.total, 0) AS weight
FROM generate_series($1::date, $2::date, '1 day') d
JOIN events e
ON e.occurred_at >= (d::date || 'T00:00:00Z')::timestamptz
AND e.occurred_at < ((d::date + 1) || 'T00:00:00Z')::timestamptz
AND ($3::bool OR e.public = true)
AND e.action IN ('Commit', 'PushEvent', 'commit_repo')
JOIN repo_languages rl
ON rl.source = e.source
AND rl.repo = CASE e.source
WHEN 'github' THEN COALESCE(
e.payload->'repo'->>'name',
e.payload->'repository'->>'full_name',
e.payload->>'_repo'
)
WHEN 'gitea' THEN COALESCE(
e.payload->'repo'->>'full_name',
e.payload->'repo'->>'name'
)
ELSE NULL
END
JOIN LATERAL (
SELECT SUM(bytes)::float AS total
FROM repo_languages r2
WHERE r2.source = rl.source AND r2.repo = rl.repo
) rt ON true
) weighted
GROUP BY date, language, color
HAVING ROUND(SUM(weight)) > 0
ORDER BY date, commits DESC
"#,
)
.bind(from)
.bind(to)
.bind(include_private)
.fetch_all(&self.pool)
.await
.map_err(map_err)?;
rows.into_iter()
.map(|r| {
Ok(LanguageDailyCount {
date: r.try_get("date").map_err(map_err)?,
language: r.try_get("language").map_err(map_err)?,
color: r.try_get("color").map_err(map_err)?,
commits: r.try_get("commits").map_err(map_err)?,
})
})
.collect()
}
async fn hourly_avgs(
&self,
from: NaiveDate,
to: NaiveDate,
tz: &str,
include_private: bool,
) -> Result<Vec<HourlyAvg>, StoreError> {
// GREATEST guards against from > to (returns NaN-via-div-by-zero
// otherwise). EXTRACT(hour FROM tz-shifted timestamp) buckets each
// event into the user's local hour rather than UTC, so the chart
// matches the labels they'd see on a clock.
let rows = sqlx::query(
r#"
WITH params AS (
SELECT GREATEST(($2::date - $1::date + 1), 1)::float8 AS day_count
),
bucketed AS (
SELECT EXTRACT(hour FROM (occurred_at AT TIME ZONE $3))::int AS hour
FROM events
WHERE occurred_at >= ($1::date::timestamp AT TIME ZONE 'UTC')
AND occurred_at < (($2::date + 1)::timestamp AT TIME ZONE 'UTC')
AND ($4::bool OR public = true)
)
SELECT g.h::int AS hour,
(COUNT(b.hour)::float8 / (SELECT day_count FROM params)) AS avg
FROM generate_series(0, 23) AS g(h)
LEFT JOIN bucketed b ON b.hour = g.h
GROUP BY g.h
ORDER BY g.h
"#,
)
.bind(from)
.bind(to)
.bind(tz)
.bind(include_private)
.fetch_all(&self.pool)
.await
.map_err(map_err)?;
rows.into_iter()
.map(|r| {
Ok(HourlyAvg {
hour: r.try_get("hour").map_err(map_err)?,
avg: r.try_get("avg").map_err(map_err)?,
})
})
.collect()
}
async fn repo_languages(&self) -> Result<Vec<RepoLanguage>, StoreError> {
let rows = sqlx::query(
r#"
SELECT source, repo, language, bytes,
COALESCE(color,
(SELECT color FROM repo_languages r2
WHERE r2.language = repo_languages.language AND r2.color IS NOT NULL
LIMIT 1)
) AS color
FROM repo_languages
ORDER BY repo, bytes DESC
"#,
)
.fetch_all(&self.pool)
.await
.map_err(map_err)?;
rows.into_iter()
.map(|r| {
let source_str: String = r.try_get("source").map_err(map_err)?;
Ok(RepoLanguage {
source: Source::from_str(&source_str).map_err(map_err)?,
repo: r.try_get("repo").map_err(map_err)?,
language: r.try_get("language").map_err(map_err)?,
bytes: r.try_get("bytes").map_err(map_err)?,
color: r.try_get("color").map_err(map_err)?,
})
})
.collect()
}
}
#[async_trait]
@@ -187,12 +452,13 @@ impl EventWriter for PgStore {
for ev in events {
let n = sqlx::query(
r#"
INSERT INTO events (id, source, action, occurred_at, payload)
VALUES ($1, $2, $3, $4, $5)
INSERT INTO events (id, source, action, occurred_at, public, payload)
VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (id) DO UPDATE
SET source = EXCLUDED.source,
action = EXCLUDED.action,
occurred_at = EXCLUDED.occurred_at,
public = EXCLUDED.public,
payload = EXCLUDED.payload
"#,
)
@@ -200,6 +466,7 @@ impl EventWriter for PgStore {
.bind(ev.source.as_str())
.bind(&ev.action)
.bind(ev.occurred_at)
.bind(ev.public)
.bind(&ev.payload)
.execute(&mut *tx)
.await
@@ -210,4 +477,37 @@ impl EventWriter for PgStore {
tx.commit().await.map_err(map_err)?;
Ok(inserted)
}
async fn upsert_repo_languages(&self, languages: &[RepoLanguage]) -> Result<usize, StoreError> {
if languages.is_empty() {
return Ok(0);
}
let mut tx = self.pool.begin().await.map_err(map_err)?;
let mut count = 0usize;
for lang in languages {
let n = sqlx::query(
r#"
INSERT INTO repo_languages (source, repo, language, bytes, color, fetched_at)
VALUES ($1, $2, $3, $4, $5, now())
ON CONFLICT (source, repo, language) DO UPDATE
SET bytes = EXCLUDED.bytes,
color = EXCLUDED.color,
fetched_at = EXCLUDED.fetched_at
"#,
)
.bind(lang.source.as_str())
.bind(&lang.repo)
.bind(&lang.language)
.bind(lang.bytes)
.bind(&lang.color)
.execute(&mut *tx)
.await
.map_err(map_err)?
.rows_affected();
count += n as usize;
}
tx.commit().await.map_err(map_err)?;
Ok(count)
}
}

132
crates/moments-entities/src/lib.rs
View File

@@ -54,6 +54,10 @@ pub struct Event {
pub source: Source,
pub action: String,
pub occurred_at: DateTime<Utc>,
/// True when the upstream marks this event as visible to anyone (e.g.
/// GitHub's top-level `public` flag). The DB stores everything; the API
/// uses this to gate what gets surfaced on the public timeline.
pub public: bool,
pub payload: serde_json::Value,
}
@@ -63,6 +67,11 @@ pub struct EventQuery {
pub from: Option<DateTime<Utc>>,
pub to: Option<DateTime<Utc>>,
pub sources: Option<Vec<Source>>,
/// Filter to events matching a specific repo (matched against payload).
pub repo: Option<String>,
/// When false (default), only `public = true` rows are returned. The API
/// pins this to false today; a future authenticated path can flip it.
pub include_private: bool,
pub limit: u32,
}
@@ -74,3 +83,126 @@ pub struct SourceSummary {
pub earliest: Option<DateTime<Utc>>,
pub latest: Option<DateTime<Utc>>,
}
/// Per-day event count for the contribution graph.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DailyCount {
pub date: chrono::NaiveDate,
pub count: i64,
}
/// Average events per day at a given hour of the day, computed in a
/// caller-supplied IANA timezone. 24 entries (0..=23).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct HourlyAvg {
pub hour: i32,
pub avg: f64,
}
/// Per-repo activity rollup for the dashboard.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ProjectSummary {
pub repo: String,
pub source: Source,
pub host: String,
pub commit_count: i64,
pub issue_count: i64,
pub pr_count: i64,
pub first_activity: Option<DateTime<Utc>>,
pub last_activity: Option<DateTime<Utc>>,
}
/// Per-language daily commit count for the language stream graph.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LanguageDailyCount {
pub date: chrono::NaiveDate,
pub language: String,
pub color: Option<String>,
pub commits: i64,
}
/// Per-repo language breakdown from the forge.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RepoLanguage {
pub source: Source,
pub repo: String,
pub language: String,
pub bytes: i64,
pub color: Option<String>,
}
// ---------------------------------------------------------------------
// Presentation shape — what `GET /v1/events` actually returns.
// The API reshapes raw payloads into these so the frontend stays dumb.
// ---------------------------------------------------------------------
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TimelineItem {
pub id: String,
pub source: Source,
pub action: String,
pub occurred_at: DateTime<Utc>,
pub icon: TimelineIcon,
/// Primary headline. Mixed plain text + inline links so the UI can
/// render the right anchors without parsing.
pub title: Vec<TitleSegment>,
pub subtitle: Option<Vec<TitleSegment>>,
pub body: Option<TimelineBody>,
}
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
#[serde(tag = "kind", rename_all = "lowercase")]
pub enum TitleSegment {
Text { text: String },
Link { text: String, url: String },
}
impl TitleSegment {
pub fn text(s: impl Into<String>) -> Self {
Self::Text { text: s.into() }
}
pub fn link(text: impl Into<String>, url: impl Into<String>) -> Self {
Self::Link {
text: text.into(),
url: url.into(),
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "lowercase")]
pub enum TimelineBody {
Markdown { text: String },
Commits { commits: Vec<CommitSummary> },
Links { items: Vec<TitleSegment> },
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CommitSummary {
pub sha: String,
pub short_sha: String,
pub message: String,
pub url: String,
pub author: Option<String>,
}
/// UI icon hint. The frontend maps these to its own icon set; new variants
/// here require a frontend update but never break existing renders (the UI
/// falls back to the generic icon for unknown values).
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "kebab-case")]
pub enum TimelineIcon {
GitPush,
GitCommit,
GitMerge,
GitFork,
GitBranchCreate,
GitBranchDelete,
PullRequest,
Issue,
Comment,
Star,
Release,
Bug,
Generic,
}

171
crates/moments-worker/src/main.rs
View File

@@ -4,7 +4,12 @@ use clap::Parser;
use moments_core::{EventSource, run_poller};
use moments_data::{
PgStore,
bugzilla::{BugzillaConfig, BugzillaSource},
gitea::{GiteaConfig, GiteaSource},
github::{GithubConfig, GithubSource},
github_repo::{GithubRepoConfig, GithubRepoSource},
github_search::{GithubSearchConfig, GithubSearchSource},
hg::{HgConfig, HgSource},
};
use reqwest::Client;
use tracing::info;
@@ -22,9 +27,81 @@ struct Args {
#[arg(long, env = "GITHUB_TOKEN")]
github_token: Option<String>,
/// Seconds between poll attempts per source.
/// Seconds between Events-API polls (live feed, last 90 days).
#[arg(long, env = "POLL_INTERVAL_SECS", default_value = "600")]
interval_secs: u64,
/// Seconds between Search-API polls (historical issue/PR backfill).
/// Defaults to 24h — this is a backfill, not a live feed.
#[arg(long, env = "SEARCH_POLL_INTERVAL_SECS", default_value = "86400")]
search_interval_secs: u64,
/// Seconds between per-repo commit enumeration polls (full history backfill).
/// Defaults to weekly — expensive initial scan, cheap afterwards.
#[arg(long, env = "REPO_POLL_INTERVAL_SECS", default_value = "604800")]
repo_interval_secs: u64,
#[arg(long, env = "GITEA_HOST", default_value = "git.lair.cafe")]
gitea_host: String,
#[arg(long, env = "GITEA_USER", default_value = "grenade")]
gitea_user: String,
#[arg(long, env = "GITEA_TOKEN")]
gitea_token: Option<String>,
/// Seconds between Gitea activity-feed polls.
#[arg(long, env = "GITEA_POLL_INTERVAL_SECS", default_value = "600")]
gitea_interval_secs: u64,
#[arg(long, env = "HG_HOST", default_value = "hg-edge.mozilla.org")]
hg_host: String,
/// Comma-separated repo groups to scan. Repos within each group are
/// discovered via `/{group}/?style=json`.
#[arg(
long,
env = "HG_GROUPS",
value_delimiter = ',',
default_value = "build,integration"
)]
hg_groups: Vec<String>,
/// Comma-separated individual repos to scan (e.g. `mozilla-central`).
#[arg(
long,
env = "HG_REPOS",
value_delimiter = ',',
default_value = "mozilla-central"
)]
hg_repos: Vec<String>,
/// Comma-separated author substrings for `author()` revset queries.
#[arg(
long,
env = "HG_AUTHOR_TERMS",
value_delimiter = ',',
default_value = "rthijssen,grenade"
)]
hg_author_terms: Vec<String>,
/// Seconds between hg pushlog scans (defaults to 24h — historical data).
#[arg(long, env = "HG_POLL_INTERVAL_SECS", default_value = "86400")]
hg_interval_secs: u64,
#[arg(long, env = "BUGZILLA_HOST", default_value = "bugzilla.mozilla.org")]
bugzilla_host: String,
#[arg(long, env = "BUGZILLA_EMAIL", default_value = "rthijssen@mozilla.com")]
bugzilla_email: String,
/// Optional bugzilla API key. Without one, only public bugs are returned.
#[arg(long, env = "BUGZILLA_API_KEY")]
bugzilla_api_key: Option<String>,
/// Seconds between bugzilla creator-query polls (defaults to 24h).
#[arg(long, env = "BUGZILLA_POLL_INTERVAL_SECS", default_value = "86400")]
bugzilla_interval_secs: u64,
}
#[tokio::main]
@@ -50,18 +127,108 @@ async fn main() -> anyhow::Result<()> {
},
)) as Arc<dyn EventSource>;
let github_search = Arc::new(GithubSearchSource::new(
http.clone(),
store.clone(),
store.clone(),
GithubSearchConfig {
user: args.github_user.clone(),
token: args.github_token.clone(),
..Default::default()
},
)) as Arc<dyn EventSource>;
let github_repo = Arc::new(GithubRepoSource::new(
http.clone(),
store.clone(),
store.clone(),
GithubRepoConfig {
user: args.github_user.clone(),
token: args.github_token.clone(),
..Default::default()
},
)) as Arc<dyn EventSource>;
let gitea = Arc::new(GiteaSource::new(
http.clone(),
store.clone(),
store.clone(),
GiteaConfig {
host: args.gitea_host.clone(),
user: args.gitea_user.clone(),
token: args.gitea_token.clone(),
..Default::default()
},
)) as Arc<dyn EventSource>;
let hg = Arc::new(HgSource::new(
http.clone(),
store.clone(),
store.clone(),
HgConfig {
host: args.hg_host.clone(),
author_terms: args.hg_author_terms.clone(),
groups: args.hg_groups.clone(),
repos: args.hg_repos.clone(),
},
)) as Arc<dyn EventSource>;
let bugzilla = Arc::new(BugzillaSource::new(
http.clone(),
store.clone(),
store.clone(),
BugzillaConfig {
host: args.bugzilla_host.clone(),
creator_email: args.bugzilla_email.clone(),
api_key: args.bugzilla_api_key.clone(),
..Default::default()
},
)) as Arc<dyn EventSource>;
info!(
github_user = args.github_user,
interval_secs = args.interval_secs,
gitea_host = args.gitea_host,
gitea_user = args.gitea_user,
hg_host = args.hg_host,
hg_groups = ?args.hg_groups,
hg_repos = ?args.hg_repos,
hg_author_terms = ?args.hg_author_terms,
bugzilla_host = args.bugzilla_host,
bugzilla_email = args.bugzilla_email,
events_interval_secs = args.interval_secs,
search_interval_secs = args.search_interval_secs,
repo_interval_secs = args.repo_interval_secs,
gitea_interval_secs = args.gitea_interval_secs,
hg_interval_secs = args.hg_interval_secs,
bugzilla_interval_secs = args.bugzilla_interval_secs,
"worker started"
);
let interval = Duration::from_secs(args.interval_secs);
let search_interval = Duration::from_secs(args.search_interval_secs);
let repo_interval = Duration::from_secs(args.repo_interval_secs);
let gitea_interval = Duration::from_secs(args.gitea_interval_secs);
let hg_interval = Duration::from_secs(args.hg_interval_secs);
let bugzilla_interval = Duration::from_secs(args.bugzilla_interval_secs);
let github_task = tokio::spawn(async move { run_poller(github, interval).await });
let github_search_task =
tokio::spawn(async move { run_poller(github_search, search_interval).await });
let github_repo_task =
tokio::spawn(async move { run_poller(github_repo, repo_interval).await });
let gitea_task = tokio::spawn(async move { run_poller(gitea, gitea_interval).await });
let hg_task = tokio::spawn(async move { run_poller(hg, hg_interval).await });
let bugzilla_task =
tokio::spawn(async move { run_poller(bugzilla, bugzilla_interval).await });
tokio::signal::ctrl_c().await?;
info!("shutdown signal received");
github_task.abort();
github_search_task.abort();
github_repo_task.abort();
gitea_task.abort();
hg_task.abort();
bugzilla_task.abort();
Ok(())
}

127
readme.md
View File

@@ -1,41 +1,136 @@
# moments
Personal activity timeline for [rob.tn](https://rob.tn). Polls public sources (GitHub, Gitea, hg-edge.mozilla.org, bugzilla.mozilla.org), stores raw payloads in Postgres, and serves a reshaped timeline to a static React frontend.
personal activity timeline and portfolio site. polls public sources (github, gitea, mercurial, bugzilla), stores raw payloads in postgres, and serves a dashboard + project detail views to a react frontend.
Successor to the now-defunct [grenade-events-react](https://github.com/grenade/grenade-events-react), which depended on MongoDB Stitch (retired by MongoDB in September 2022).
successor to the now-defunct [grenade-events-react](https://github.com/grenade/grenade-events-react), which depended on mongodb stitch (retired by mongodb in september 2022).
## Layout
## layout
```
crates/
moments-entities/ # types and DTOs
moments-core/ # ingestion + reshape logic
moments-data/ # postgres adapter + migrations
moments-api/ # axum read-only HTTP API (binary)
moments-entities/ # types and dtos (event, source, project/daily summaries)
moments-core/ # ingestion traits, presentation reshape, poller loop
moments-data/ # postgres adapter, migrations, all event-source impls
moments-api/ # axum read-only http api + forge proxy + og image (binary)
moments-worker/ # ingestion daemon (binary)
ui/ # vite + react + swc + ts frontend
ui/ # vite + react + swc + typescript frontend
asset/ # systemd, nginx, firewalld, manifest.yml
script/deploy.sh
script/
deploy.sh # manifest-driven deploy to prod
hg-ingest.sh # one-shot local hg clone + psql ingest
certify.sh # letsencrypt cert management
teardown.sh # service removal
db-perms.sh # postgres role + ident setup
```
Architectural conventions follow [grenade/architecture/generic.md](https://git.lair.cafe/grenade/architecture/src/branch/main/generic.md).
architectural conventions follow [grenade/architecture/generic.md](https://git.lair.cafe/grenade/architecture/src/branch/main/generic.md).
## Local development
## data sources
| source | impl | endpoint | notes |
|--------|------|----------|-------|
| github events | `github.rs` | `/users/{user}/events` | last 90 days, etag-optimised polling |
| github search | `github_search.rs` | `/search/commits` + `/search/issues` | historical backfill, 1000-result cap |
| github repo | `github_repo.rs` | `/user/repos` + `/repos/{o}/{r}/commits` | full commit history, no cap, weekly poll |
| gitea | `gitea.rs` | user + org activity feeds | auto-discovers orgs, filters by user |
| mercurial | `hg.rs` | `json-log?rev=author()` | revset-based, one-shot backfill then skip |
| bugzilla | `bugzilla.rs` | `/rest/bug?creator=` | mozilla bugzilla |
hg repos are archived (mozilla retired hg). the worker skips hg after the first successful scan. for bulk ingestion, `script/hg-ingest.sh` clones repos locally and inserts via psql, avoiding rate limits on hg-edge.mozilla.org.
## frontend routes
| path | page | description |
|------|------|-------------|
| `/` or `/dash` | dashboard | contribution graphs (daily + all-time weekly) + ranked project cards with forge icons and language info |
| `/activity` | timeline | filterable activity feed with source toggles, date range slider, and event limit |
| `/activity/:timespan` | timeline | pre-filtered by date (`YYYY-MM-DD`) or range (`YYYY-MM-DD..YYYY-MM-DD`) |
| `/project/:source/*` | project detail | repo readme, language breakdown bar, per-repo activity timeline |
| `/cv` | resume | loaded from github gist, markdown-rendered |
shared layout provides nav header (dash, activity, cv + external links) and footer across all routes.
## api endpoints
| method | path | description |
|--------|------|-------------|
| GET | `/v1/healthz` | liveness probe |
| GET | `/v1/events?from=&to=&source=&repo=&limit=` | reshaped timeline items |
| GET | `/v1/sources` | per-source summary (count, earliest, latest) |
| GET | `/v1/projects` | per-repo aggregated stats (commits, issues, prs, date range) |
| GET | `/v1/activity/daily?from=&to=` | per-day event counts for contribution graphs |
| GET | `/v1/forge/{source}/*?host=` | proxy to github/gitea apis (avoids cors) |
| GET | `/v1/og/contributions.png` | server-rendered contribution graph as png (resvg) |
the og image endpoint renders the all-time weekly contribution graph as svg, rasterizes to png via resvg, and serves it with a 1-hour cache. used as the `og:image` meta tag for social media previews.
## local development
```sh
cargo build --workspace
cargo run -p moments-api # serves on 127.0.0.1:8080
cargo run -p moments-worker # one-shot ingest tick (until --interval is wired up)
cargo run -p moments-worker # starts all pollers
cd ui && npm install && npm run dev # vite dev server on :5173
```
The API expects a Postgres reachable at `DATABASE_URL`. For magrathea, that's an mTLS connection using the host cert. For local dev against a throwaway database:
the api expects a postgres reachable at `DATABASE_URL`. in production this is an mtls connection using the host cert. for local dev against a throwaway database:
```sh
DATABASE_URL=postgres://localhost/moments cargo run -p moments-api
```
Migrations live in `crates/moments-data/migrations/` and run automatically on API startup.
migrations live in `crates/moments-data/migrations/` and run automatically on worker startup. the api connects as `moments_ro` and never runs migrations — the worker (as `moments_rw`) is the schema owner.
## Deployment
## deployment
See `asset/manifest.yml` and `script/deploy.sh`.
```sh
./script/deploy.sh <env> all # api + worker + web
./script/deploy.sh <env> api worker # subset
./script/deploy.sh <env> default # api + web only (worker untouched)
./script/deploy.sh <env> all --dry-run
```
concrete hosts, ports, and the site's `server_name` live in `asset/manifest.yml`. the shape of the deployment:
| component | notes |
|-----------|-------|
| api | binds the port from `api.config.bind`; firewalld service `moments-api` |
| worker | no listening port; pollers only |
| web | per-site nginx ingress; `/api/*` reverse-proxies to the api host |
| db | postgres mtls, passwordless |
postgres roles `moments_rw` and `moments_ro` must exist on the primary, with `pg_ident.conf.d/<host>.conf` mapping the api host's fqdn to `moments_ro` and the worker host's fqdn to `moments_rw`. see `asset/sql/bootstrap-moments.sql`, `asset/postgres/ident.conf.tmpl`, and `script/db-perms.sh`.
secrets are resolved at deploy time via `pass`. the mapping of env-var name to pass-store path lives under `worker.secrets` in `manifest.yml`; `deploy.sh` iterates the map, fetches each secret, and substitutes the matching `{{NAME}}` placeholder in `worker.env.tmpl`.
## environment variables
### worker
| variable | default | description |
|----------|---------|-------------|
| `DATABASE_URL` | required | postgres connection string |
| `GITHUB_USER` | `grenade` | github username |
| `GITHUB_TOKEN` | optional | github pat for higher rate limits + private events |
| `POLL_INTERVAL_SECS` | `600` | github events api poll interval |
| `SEARCH_POLL_INTERVAL_SECS` | `86400` | github search backfill interval |
| `REPO_POLL_INTERVAL_SECS` | `604800` | github per-repo commit enumeration (weekly) |
| `GITEA_HOST` | `git.lair.cafe` | gitea instance hostname |
| `GITEA_USER` | `grenade` | gitea username |
| `GITEA_TOKEN` | optional | gitea token for org discovery |
| `GITEA_POLL_INTERVAL_SECS` | `600` | gitea activity feed poll interval |
| `HG_HOST` | `hg-edge.mozilla.org` | mercurial host |
| `HG_GROUPS` | `build,integration` | hg repo groups to discover |
| `HG_REPOS` | `mozilla-central` | individual hg repos |
| `HG_AUTHOR_TERMS` | `rthijssen,grenade` | author substrings for revset queries |
| `HG_POLL_INTERVAL_SECS` | `86400` | hg poll interval (skips after first scan) |
| `BUGZILLA_HOST` | `bugzilla.mozilla.org` | bugzilla instance |
| `BUGZILLA_EMAIL` | `rthijssen@mozilla.com` | bugzilla creator email filter |
| `BUGZILLA_POLL_INTERVAL_SECS` | `86400` | bugzilla poll interval |
### api
| variable | default | description |
|----------|---------|-------------|
| `DATABASE_URL` | required | postgres connection string (read-only role) |
| `BIND_ADDR` | `127.0.0.1:8080` | api listen address |

17
script/certify.sh Normal file
View File

@@ -0,0 +1,17 @@
#!/usr/bin/env bash
tld=rob.tn
fqdn=${tld}
sudo certbot certonly \
-m ops@${tld} \
--agree-tos \
--no-eff-email \
--noninteractive \
--cert-name ${fqdn} \
--expand \
--allow-subset-of-names \
--key-type ecdsa \
--dns-cloudflare \
--dns-cloudflare-credentials /root/.cloudflare/${tld} \
--dns-cloudflare-propagation-seconds 60 \
-d ${fqdn}

63
script/db-perms.sh Executable file
View File

@@ -0,0 +1,63 @@
#!/usr/bin/env bash
#
# Idempotently add cert_cn → role mappings to pg_ident.conf.d on the moments
# postgres primary and standby, then reload postgres so the changes take
# effect. Re-running is a no-op (no duplicate lines, no spurious reload).
#
# Run from a workstation with ssh access to both pg hosts. This script ssh's
# out; do NOT run it on magrathea/frankie directly.
set -euo pipefail
api_host=nikola.kosherinata.internal
worker_host=frootmig.kosherinata.internal
pg_hosts=(
magrathea.kosherinata.internal
frankie.kosherinata.internal
)
# Each (cert_cn host, role) pair becomes one cert_cn line in
# pg_ident.conf.d/<cert_cn host>.conf on every pg host listed above.
mapping_pairs=(
"$api_host" moments_ro
"$worker_host" moments_rw
)
ident_dir=/var/lib/pgsql/18/data/pg_ident.conf.d
for pg_host in "${pg_hosts[@]}"; do
printf '==> %s\n' "$pg_host"
ssh -o BatchMode=yes "$pg_host" "sudo bash -s -- ${ident_dir@Q} ${mapping_pairs[@]@Q}" <<'REMOTE_EOF'
set -euo pipefail
ident_dir="$1"; shift
changed=0
while [[ $# -gt 0 ]]; do
cert_cn_host="$1"
role="$2"
shift 2
line="cert_cn ${cert_cn_host} ${role}"
file="${ident_dir}/${cert_cn_host}.conf"
# The heredoc runs as root via sudo bash, so [[ -f ]] and grep are fine
# without dropping privs. tee --append runs as postgres so a newly-created
# file lands with the conventional postgres:postgres ownership.
if [[ -f "$file" ]] && grep --fixed-strings --line-regexp --quiet -- "$line" "$file"; then
printf ' present: %s\n' "$line"
else
printf '%s\n' "$line" | sudo -u postgres tee --append "$file" >/dev/null
printf ' added: %s\n' "$line"
changed=1
fi
done
if (( changed )); then
systemctl reload postgresql-18
echo " reloaded postgresql-18"
else
echo " no changes; reload skipped"
fi
REMOTE_EOF
done

551
script/deploy.sh Executable file
View File

@@ -0,0 +1,551 @@
#!/usr/bin/env bash
#
# moments deployment script.
#
# ./script/deploy.sh <environment> [component...]
# ./script/deploy.sh prod api worker web
# ./script/deploy.sh prod all
#
# Builds artifacts locally, resolves secrets from `pass`, renders config
# templates, rsyncs everything to the target hosts, and reloads systemd /
# nginx / firewalld / SELinux state idempotently.
set -euo pipefail
shopt -s nullglob
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
manifest="${repo_root}/asset/manifest.yml"
dry_run=0
usage() {
cat <<EOF >&2
usage: $(basename "$0") <environment> [component...] [--dry-run]
$(basename "$0") prod api worker web
$(basename "$0") prod all
$(basename "$0") prod default # api + web (worker isn't restarted unless asked)
EOF
exit 2
}
log() { printf '\033[1;34m[deploy]\033[0m %s\n' "$*" >&2; }
warn() { printf '\033[1;33m[deploy]\033[0m %s\n' "$*" >&2; }
die() { printf '\033[1;31m[deploy]\033[0m %s\n' "$*" >&2; exit 1; }
run() {
if (( dry_run )); then
printf '\033[2m[dry-run]\033[0m %s\n' "$*" >&2
else
"$@"
fi
}
ssh_run() {
local host="$1"; shift
if (( dry_run )); then
printf '\033[2m[dry-run]\033[0m ssh %s -- %s\n' "$host" "$*" >&2
else
ssh -o BatchMode=yes "$host" "$@"
fi
}
# Ensure /tmp on the remote is world-writable + sticky (mode 1777). Some
# hosts in this fleet have had /tmp reset to root-owned 0755 by an
# unrelated configuration step, which silently breaks the rsync of the
# deploy stage dir under our unprivileged user. Check the mode first so a
# correctly-configured host doesn't incur a needless sudo call.
ensure_tmp_writable() {
local host="$1"
if (( dry_run )); then
printf '\033[2m[dry-run]\033[0m ssh %s -- stat /tmp; chmod 1777 if needed\n' "$host" >&2
return 0
fi
local mode
mode="$(ssh -o BatchMode=yes "$host" 'stat -c %a /tmp')" || {
warn "could not stat /tmp on $host"
return 1
}
if [[ "$mode" != "1777" ]]; then
warn "/tmp on $host is mode $mode; fixing to 1777"
ssh -o BatchMode=yes "$host" 'sudo chmod 1777 /tmp' || {
warn "failed to chmod /tmp on $host"
return 1
}
fi
}
[[ $# -ge 1 ]] || usage
environment="$1"; shift
components=()
while [[ $# -gt 0 ]]; do
case "$1" in
--dry-run) dry_run=1 ;;
*) components+=("$1") ;;
esac
shift
done
[[ -f "$manifest" ]] || die "manifest not found: $manifest"
command -v yq >/dev/null 2>&1 || die "yq is required"
command -v pass >/dev/null 2>&1 || die "pass is required"
command -v rsync >/dev/null 2>&1 || die "rsync is required"
command -v cargo >/dev/null 2>&1 || die "cargo is required"
command -v podman >/dev/null 2>&1 || die "podman is required (used for the deploy build container)"
# Rust binaries are built inside a Debian container so the resulting ELF
# links against an older glibc than this workstation's. Building natively
# on f44 (glibc 2.43) produces binaries that won't load on f42 / f43
# servers — the dynamic loader refuses them outright. Debian bookworm's
# glibc 2.36 is older than every Fedora release we deploy to, so its
# binaries are forward-compatible.
#
# The artifacts land in target/deploy/release/ so a native `cargo build`
# in this checkout (for tests, clippy, dev runs) doesn't compete with
# the container for incremental state, and vice-versa.
rust_build_image="docker.io/library/rust:1-bookworm"
rust_target_dir="${repo_root}/target/deploy"
# Resolve component list ----------------------------------------------------
env_path=".environments.${environment}"
yq --exit-status "${env_path}" "$manifest" >/dev/null \
|| die "environment '$environment' not found in manifest"
mapfile -t all_components < <(yq --raw-output "${env_path}.components | keys | .[]" "$manifest")
if [[ ${#components[@]} -eq 0 ]]; then
usage
fi
case "${components[0]:-}" in
all) components=("${all_components[@]}") ;;
default) components=(api web) ;;
esac
# Build artifacts -----------------------------------------------------------
needs_rust=0
needs_web=0
for c in "${components[@]}"; do
case "$c" in
api|worker) needs_rust=1 ;;
web) needs_web=1 ;;
esac
done
if (( needs_rust )); then
log "cargo build --release in ${rust_build_image} (api, worker)"
install --directory "$rust_target_dir"
# Named volumes cache the cargo registry and git index across runs so
# subsequent builds don't re-fetch every crate. CARGO_TARGET_DIR
# redirects build output into the host-mounted target/deploy.
# :Z relabels the bind mount for SELinux on Fedora hosts.
run podman run --rm \
--volume "${repo_root}:/workspace:Z" \
--volume moments-deploy-cargo-registry:/usr/local/cargo/registry \
--volume moments-deploy-cargo-git:/usr/local/cargo/git \
--workdir /workspace \
--env CARGO_TARGET_DIR=/workspace/target/deploy \
"$rust_build_image" \
cargo build --release --bin moments-api --bin moments-worker
fi
if (( needs_web )); then
log "vite build (ui)"
run sh -c "cd '${repo_root}/ui' && pnpm install --frozen-lockfile && pnpm run build"
fi
# Per-component deploy ------------------------------------------------------
deploy_api() {
local host="$1"
log "api -> $host"
local bind
bind="$(yq --raw-output "${env_path}.components.api.config.bind" "$manifest")"
[[ -n "$bind" && "$bind" != "null" ]] || die "api.config.bind missing in manifest"
[[ "$bind" == *:* ]] \
|| die "api.config.bind must be host:port form: '$bind'"
local api_port
api_port="${bind##*:}"
[[ "$api_port" =~ ^[0-9]+$ ]] \
|| die "api.config.bind port is not numeric: '$api_port'"
if (( dry_run )); then
printf '\033[2m[dry-run]\033[0m render api.env (HOSTNAME=%s, BIND=%s) + firewalld svc (port=%s) + units, stage to %s:/tmp/, install via heredoc, run sysusers/restorecon/semanage/systemctl on %s\n' \
"$host" "$bind" "$api_port" "$host" "$host" >&2
return 0
fi
local fqdn="$host"
local stage
stage="$(mktemp --directory)"
trap "rm --recursive --force '$stage'" RETURN
install --directory \
"$stage/etc/moments" \
"$stage/etc/systemd/system" \
"$stage/etc/sysusers.d" \
"$stage/etc/firewalld/services" \
"$stage/usr/local/bin"
local rendered
rendered="$(<"${repo_root}/asset/config/api.env.tmpl")"
rendered=${rendered//'{{HOSTNAME}}'/$fqdn}
rendered=${rendered//'{{BIND}}'/$bind}
printf '%s\n' "$rendered" > "$stage/etc/moments/api.env"
rendered="$(<"${repo_root}/asset/systemd/moments-api-cert.path")"
rendered=${rendered//'{{HOSTNAME}}'/$fqdn}
printf '%s\n' "$rendered" > "$stage/etc/systemd/system/moments-api-cert.path"
rendered="$(<"${repo_root}/asset/firewalld/moments-api.xml.tmpl")"
rendered=${rendered//'{{API_PORT}}'/$api_port}
printf '%s\n' "$rendered" > "$stage/etc/firewalld/services/moments-api.xml"
chmod 0644 "$stage/etc/firewalld/services/moments-api.xml"
install --mode=0644 "${repo_root}/asset/systemd/moments-api.service" "$stage/etc/systemd/system/"
install --mode=0644 "${repo_root}/asset/systemd/moments-api-cert-reload.service" "$stage/etc/systemd/system/"
install --mode=0644 "${repo_root}/asset/systemd/moments.sysusers.conf" "$stage/etc/sysusers.d/moments.conf"
install --mode=0755 "${rust_target_dir}/release/moments-api" "$stage/usr/local/bin/moments-api"
chmod 0640 "$stage/etc/moments/api.env"
# Stage to a tmpdir on the remote, then `install` each file at its final
# path via the heredoc. Never rsync into /, since rsync of staged parent
# dirs (etc/, usr/, ...) can leak ownership, ACLs and xattrs onto the
# live system dirs.
local remote_stage="/tmp/moments-deploy.api.${$}.${RANDOM}"
ensure_tmp_writable "$host" || return 1
rsync \
--archive \
--hard-links \
--numeric-ids \
--rsh='ssh -o BatchMode=yes' \
"$stage/" \
"${host}:${remote_stage}/"
ssh_run "$host" "sudo bash -s -- ${remote_stage@Q} ${api_port@Q}" <<'REMOTE_EOF'
set -euo pipefail
remote_stage="$1"
api_port="$2"
trap 'rm --recursive --force "$remote_stage"' EXIT
fqdn="$(hostname --fqdn)"
install --owner=root --group=root --mode=0644 \
"$remote_stage/etc/sysusers.d/moments.conf" \
/etc/sysusers.d/moments.conf
systemd-sysusers /etc/sysusers.d/moments.conf
install --directory --owner=root --group=moments --mode=0750 /etc/moments
install --directory --owner=moments --group=moments --mode=0750 /var/lib/moments
install --owner=root --group=moments --mode=0640 \
"$remote_stage/etc/moments/api.env" \
/etc/moments/api.env
install --owner=root --group=root --mode=0644 \
"$remote_stage/etc/systemd/system/moments-api.service" \
/etc/systemd/system/moments-api.service
install --owner=root --group=root --mode=0644 \
"$remote_stage/etc/systemd/system/moments-api-cert.path" \
/etc/systemd/system/moments-api-cert.path
install --owner=root --group=root --mode=0644 \
"$remote_stage/etc/systemd/system/moments-api-cert-reload.service" \
/etc/systemd/system/moments-api-cert-reload.service
install --owner=root --group=root --mode=0644 \
"$remote_stage/etc/firewalld/services/moments-api.xml" \
/etc/firewalld/services/moments-api.xml
install --owner=root --group=root --mode=0755 \
"$remote_stage/usr/local/bin/moments-api" \
/usr/local/bin/moments-api
# Grant the moments user read access to the host private key for the
# postgres mTLS connection.
setfacl --modify=u:moments:r "/etc/pki/tls/private/${fqdn}.pem" || true
# Idempotent label: --add fails if the port is already labelled (we suppress
# that one stderr line); --modify is then a no-op or fixes a stale type.
semanage port --add --type=http_port_t --proto=tcp "$api_port" 2>/dev/null \
|| semanage port --modify --type=http_port_t --proto=tcp "$api_port"
firewall-cmd --reload
zone="$(firewall-cmd --get-default-zone)"
if ! firewall-cmd --zone="$zone" --query-service=moments-api >/dev/null 2>&1; then
firewall-cmd --permanent --zone="$zone" --add-service=moments-api
firewall-cmd --zone="$zone" --add-service=moments-api
fi
restorecon -Rv /usr/local/bin/moments-api /etc/moments /var/lib/moments
systemctl daemon-reload
systemctl enable --now moments-api-cert.path
systemctl enable --now moments-api.service
systemctl restart moments-api.service
# Quietly retry while the service binds; only show curl's diagnostics if
# every attempt fails. The journalctl tail on failure is the verbose source.
for i in 1 2 3 4 5 6 7 8 9 10; do
if curl --fail --silent "http://${fqdn}:${api_port}/v1/healthz" >/dev/null 2>&1; then
echo "moments-api healthy"
exit 0
fi
sleep 1
done
echo "moments-api did not become healthy" >&2
curl --fail --silent --show-error "http://${fqdn}:${api_port}/v1/healthz" >/dev/null || true
journalctl --unit=moments-api.service --lines=50 --no-pager >&2
exit 1
REMOTE_EOF
}
deploy_worker() {
local host="$1"
log "worker -> $host"
# Manifest entries under `worker.secrets` map env-var name -> pass store path.
# The script fetches each via `pass` and substitutes the matching {{NAME}}
# placeholder in worker.env.tmpl. Adding a new secret is then a manifest +
# template change; no script edit required.
local -a secret_lines secret_keys
mapfile -t secret_lines < <(yq --raw-output \
"${env_path}.components.worker.secrets // {} | to_entries | .[] | \"\(.key)=\(.value)\"" \
"$manifest")
local line
for line in "${secret_lines[@]}"; do
[[ -n "$line" ]] && secret_keys+=("${line%%=*}")
done
if (( dry_run )); then
printf '\033[2m[dry-run]\033[0m render worker.env (HOSTNAME=%s, secrets [%s] from pass) + units, stage to %s:/tmp/, install via heredoc, run sysusers/restorecon/systemctl on %s\n' \
"$host" "${secret_keys[*]:-none}" "$host" "$host" >&2
return 0
fi
local fqdn="$host"
local stage
stage="$(mktemp --directory)"
trap "rm --recursive --force '$stage'" RETURN
install --directory \
"$stage/etc/moments" \
"$stage/etc/systemd/system" \
"$stage/etc/sysusers.d" \
"$stage/usr/local/bin"
# Render templates in-memory so secrets never appear on a command line
# (sed would expose them to anything that can read /proc/<pid>/cmdline).
local rendered
rendered="$(<"${repo_root}/asset/config/worker.env.tmpl")"
rendered=${rendered//'{{HOSTNAME}}'/$fqdn}
local key pass_path value
for line in "${secret_lines[@]}"; do
[[ -z "$line" ]] && continue
key="${line%%=*}"
pass_path="${line#*=}"
if pass show "$pass_path" >/dev/null 2>&1; then
value="$(pass show "$pass_path")"
else
warn "no secret in pass at '${pass_path}' for ${key}; worker will run without ${key}"
value=""
fi
rendered=${rendered//"{{${key}}}"/$value}
done
printf '%s\n' "$rendered" > "$stage/etc/moments/worker.env"
rendered="$(<"${repo_root}/asset/systemd/moments-worker-cert.path")"
rendered=${rendered//'{{HOSTNAME}}'/$fqdn}
printf '%s\n' "$rendered" > "$stage/etc/systemd/system/moments-worker-cert.path"
install --mode=0644 "${repo_root}/asset/systemd/moments-worker.service" "$stage/etc/systemd/system/"
install --mode=0644 "${repo_root}/asset/systemd/moments-worker-cert-reload.service" "$stage/etc/systemd/system/"
install --mode=0644 "${repo_root}/asset/systemd/moments.sysusers.conf" "$stage/etc/sysusers.d/moments.conf"
install --mode=0755 "${rust_target_dir}/release/moments-worker" "$stage/usr/local/bin/moments-worker"
chmod 0640 "$stage/etc/moments/worker.env"
# Stage to a tmpdir on the remote, then `install` each file at its final
# path via the heredoc. Never rsync into /.
local remote_stage="/tmp/moments-deploy.worker.${$}.${RANDOM}"
ensure_tmp_writable "$host" || return 1
rsync \
--archive \
--hard-links \
--numeric-ids \
--rsh='ssh -o BatchMode=yes' \
"$stage/" \
"${host}:${remote_stage}/"
ssh_run "$host" "sudo bash -s -- ${remote_stage@Q}" <<'REMOTE_EOF'
set -euo pipefail
remote_stage="$1"
trap 'rm --recursive --force "$remote_stage"' EXIT
fqdn="$(hostname --fqdn)"
install --owner=root --group=root --mode=0644 \
"$remote_stage/etc/sysusers.d/moments.conf" \
/etc/sysusers.d/moments.conf
systemd-sysusers /etc/sysusers.d/moments.conf
install --directory --owner=root --group=moments --mode=0750 /etc/moments
install --directory --owner=moments --group=moments --mode=0750 /var/lib/moments
install --owner=root --group=moments --mode=0640 \
"$remote_stage/etc/moments/worker.env" \
/etc/moments/worker.env
install --owner=root --group=root --mode=0644 \
"$remote_stage/etc/systemd/system/moments-worker.service" \
/etc/systemd/system/moments-worker.service
install --owner=root --group=root --mode=0644 \
"$remote_stage/etc/systemd/system/moments-worker-cert.path" \
/etc/systemd/system/moments-worker-cert.path
install --owner=root --group=root --mode=0644 \
"$remote_stage/etc/systemd/system/moments-worker-cert-reload.service" \
/etc/systemd/system/moments-worker-cert-reload.service
install --owner=root --group=root --mode=0755 \
"$remote_stage/usr/local/bin/moments-worker" \
/usr/local/bin/moments-worker
setfacl --modify=u:moments:r "/etc/pki/tls/private/${fqdn}.pem" || true
restorecon -Rv /usr/local/bin/moments-worker /etc/moments /var/lib/moments
systemctl daemon-reload
systemctl enable --now moments-worker-cert.path
systemctl enable --now moments-worker.service
systemctl restart moments-worker.service
if ! systemctl is-active --quiet moments-worker.service; then
journalctl --unit=moments-worker.service --lines=50 --no-pager >&2
exit 1
fi
echo "moments-worker active"
REMOTE_EOF
}
deploy_web() {
local host="$1"
log "web -> $host"
local server_name web_root api_upstream
server_name="$(yq --raw-output "${env_path}.components.web.config.server_name" "$manifest")"
web_root="$(yq --raw-output "${env_path}.components.web.config.root" "$manifest")"
api_upstream="$(yq --raw-output "${env_path}.components.web.config.api_upstream" "$manifest")"
[[ -n "$server_name" && "$server_name" != "null" ]] || die "web.config.server_name missing in manifest"
[[ -n "$web_root" && "$web_root" != "null" ]] || die "web.config.root missing in manifest"
[[ -n "$api_upstream" && "$api_upstream" != "null" ]] || die "web.config.api_upstream missing in manifest"
[[ "$web_root" == /* ]] \
|| die "web.config.root must be an absolute path: '$web_root'"
[[ "$api_upstream" == http://* || "$api_upstream" == https://* ]] \
|| die "web.config.api_upstream must be a http(s) URL: '$api_upstream'"
local api_upstream_scheme api_upstream_addr api_upstream_port
api_upstream_scheme="${api_upstream%%://*}"
api_upstream_addr="${api_upstream#*://}"
[[ "$api_upstream_addr" == *:* ]] \
|| die "web.config.api_upstream must include an explicit port: '$api_upstream'"
api_upstream_port="${api_upstream_addr##*:}"
[[ "$api_upstream_port" =~ ^[0-9]+$ ]] \
|| die "extracted upstream port is not numeric: '$api_upstream_port'"
local site_conf_path="/etc/nginx/conf.d/${server_name}.conf"
if (( dry_run )); then
printf '\033[2m[dry-run]\033[0m render %s (server_name=%s, docroot=%s, upstream=%s://%s) + rsync ui/dist/ to %s:%s/, run nginx -t/reload on %s\n' \
"$site_conf_path" "$server_name" "$web_root" \
"$api_upstream_scheme" "$api_upstream_addr" \
"$host" "$web_root" "$host" >&2
return 0
fi
local stage
stage="$(mktemp --directory)"
trap "rm --recursive --force '$stage'" RETURN
install --directory "${stage}${web_root}" "$stage/etc/nginx/conf.d"
rsync --archive "${repo_root}/ui/dist/" "${stage}${web_root}/"
local rendered
rendered="$(<"${repo_root}/asset/nginx/site.conf.tmpl")"
rendered=${rendered//'{{SERVER_NAME}}'/$server_name}
rendered=${rendered//'{{DOCROOT}}'/$web_root}
rendered=${rendered//'{{API_UPSTREAM_SCHEME}}'/$api_upstream_scheme}
rendered=${rendered//'{{API_UPSTREAM_ADDR}}'/$api_upstream_addr}
printf '%s\n' "$rendered" > "${stage}${site_conf_path}"
chmod 0644 "${stage}${site_conf_path}"
# Both targets are leaf paths (the docroot itself, and a single named
# file) so rsync does not traverse /var or /etc parents — `--chown` is
# enough; -A/-X are intentionally absent.
rsync \
--archive \
--hard-links \
--numeric-ids \
--chown root:root \
--rsh='ssh -o BatchMode=yes' \
--rsync-path 'sudo rsync' \
--delete \
"${stage}${web_root}/" \
"${host}:${web_root}/"
rsync \
--archive \
--hard-links \
--numeric-ids \
--chown root:root \
--rsh='ssh -o BatchMode=yes' \
--rsync-path 'sudo rsync' \
"${stage}${site_conf_path}" \
"${host}:${site_conf_path}"
ssh_run "$host" "sudo bash -s -- ${web_root@Q} ${site_conf_path@Q} ${api_upstream_port@Q}" <<'REMOTE_EOF'
set -euo pipefail
web_root="$1"
site_conf_path="$2"
api_upstream_port="$3"
# Allow nginx to make outbound connections to the moments-api upstream
# across the WG mesh.
setsebool -P httpd_can_network_connect on
# Idempotent label: --add fails if the port is already labelled (we suppress
# that one stderr line); --modify is then a no-op or fixes a stale type.
semanage port --add --type=http_port_t --proto=tcp "$api_upstream_port" 2>/dev/null \
|| semanage port --modify --type=http_port_t --proto=tcp "$api_upstream_port"
restorecon -Rv "$web_root" "$site_conf_path"
if ! nginx -t; then
echo "nginx config check failed" >&2
exit 1
fi
systemctl reload nginx
echo "nginx reloaded"
REMOTE_EOF
}
# Dispatch ------------------------------------------------------------------
failed=()
for component in "${components[@]}"; do
mapfile -t hosts < <(yq --raw-output "${env_path}.components.${component}.hosts[]" "$manifest")
for host in "${hosts[@]}"; do
case "$component" in
api) deploy_api "$host" || failed+=("api@$host") ;;
worker) deploy_worker "$host" || failed+=("worker@$host") ;;
web) deploy_web "$host" || failed+=("web@$host") ;;
*) warn "unknown component: $component" ;;
esac
done
done
if [[ ${#failed[@]} -gt 0 ]]; then
die "failed: ${failed[*]}"
fi
log "deploy complete"

141
script/hg-ingest.sh Executable file
View File

@@ -0,0 +1,141 @@
#!/usr/bin/env bash
#
# One-shot hg changeset ingestion via local clones.
#
# Bare-clones each hg repo, extracts changesets matching author terms,
# and inserts them into the moments database. Sets poller_state so the
# worker won't re-scan.
#
# Requirements: hg (mercurial), psql, jq
#
# Usage:
# DATABASE_URL="postgres://..." ./script/hg-ingest.sh
#
set -euo pipefail
DATABASE_URL="${DATABASE_URL:-postgres://moments_rw@magrathea.kosherinata.internal:5432/moments?sslmode=verify-full&sslrootcert=/etc/pki/ca-trust/source/anchors/root-internal.pem&sslcert=/etc/pki/tls/misc/$(hostname -f).pem&sslkey=/etc/pki/tls/private/$(hostname -f).pem}"
HG_HOST="${HG_HOST:-hg-edge.mozilla.org}"
WORK_DIR="${HG_WORK_DIR:-$HOME/hg}"
# Repos to clone (groups are expanded inline)
REPOS=(
mozilla-central
integration/mozilla-inbound
integration/autoland
integration/fx-team
integration/b2g-inbound
build/puppet
build/tools
build/buildbot
build/buildbot-configs
build/slave_health
build/mozharness
build/braindump
build/cloud-tools
build/compare-locales
build/nagios-core
build/partner-repacks
build/preproduction
build/rpm-sources
build/talos
build/tupperware
build/ash-mozharness
build/autoland
build/opsi-package-sources
)
# Author terms — matched case-insensitively against changeset author fields
AUTHOR_TERMS=("rthijssen" "grenade")
: "${DATABASE_URL:?DATABASE_URL must be set}"
mkdir -p "$WORK_DIR"
total=0
CLONE_DIR="$WORK_DIR/clone"
CACHE_DIR="$WORK_DIR/cache"
mkdir -p "$CACHE_DIR"
cd "$WORK_DIR"
for repo in "${REPOS[@]}"; do
cache_file="$CACHE_DIR/$(echo "$repo" | tr '/' '_').tsv"
# Skip repos already cached (re-run safe)
if [ -f "$cache_file" ]; then
echo "[hg-ingest] $repo: using cached results"
else
# Remove any previous clone to keep only one on disk
rm -rf "$CLONE_DIR"
echo "[hg-ingest] cloning $repo"
if ! hg clone --noupdate "https://$HG_HOST/$repo" "$CLONE_DIR"; then
echo "[hg-ingest] clone failed: $repo (skipping)"
continue
fi
# Build revset: author(term1) or author(term2) ...
revset=""
for term in "${AUTHOR_TERMS[@]}"; do
if [ -z "$revset" ]; then
revset="author('$term')"
else
revset="$revset or author('$term')"
fi
done
# Extract matching changesets to cache file
hg log -R "$CLONE_DIR" -r "$revset" \
--template '{node}\t{author}\t{date|hgdate}\t{desc|firstline}\n' \
> "$cache_file" || true
# Free disk immediately
rm -rf "$CLONE_DIR"
fi
# Ingest cached results into the database
count=0
while IFS=$'\t' read -r node author date_raw desc; do
[ -z "$node" ] && continue
# {date|hgdate} outputs "timestamp offset" — take just the timestamp
date_ts="${date_raw%% *}"
# Build ISO timestamp from unix epoch
occurred_at=$(date -u -d "@${date_ts}" '+%Y-%m-%dT%H:%M:%SZ')
event_id="hg:${repo}:${node}"
# Build payload JSON (jq handles all escaping)
payload=$(jq -n \
--arg node "$node" \
--arg user "$author" \
--arg desc "$desc" \
--arg repo "$repo" \
--arg host "$HG_HOST" \
'{node: $node, user: $user, desc: $desc, _repo: $repo, _host: $host}')
# Upsert into events table
psql "$DATABASE_URL" -q -c "
INSERT INTO events (id, source, action, occurred_at, public, payload)
VALUES (\$\$${event_id}\$\$, 'hg', 'Commit', '${occurred_at}', true, \$\$${payload}\$\$::jsonb)
ON CONFLICT (id) DO NOTHING;
"
count=$((count + 1))
done < "$cache_file"
if [ "$count" -gt 0 ]; then
echo "[hg-ingest] $repo: $count changesets ingested"
fi
total=$((total + count))
done
# Mark poller state so the worker skips hg
psql "$DATABASE_URL" -q -c "
INSERT INTO poller_state (source, last_fetched)
VALUES ('hg', now())
ON CONFLICT (source) DO UPDATE SET last_fetched = now();
"
echo "[hg-ingest] done. total: $total changesets"

288
script/teardown.sh Executable file
View File

@@ -0,0 +1,288 @@
#!/usr/bin/env bash
#
# moments teardown script.
#
# ./script/teardown.sh <environment> <host> [component...] [--dry-run]
# ./script/teardown.sh prod anjie.kosherinata.internal api worker
# ./script/teardown.sh prod oolon.kosherinata.internal web --remove-docroot
# ./script/teardown.sh prod anjie.kosherinata.internal all --dry-run
#
# Removes moments unit files, binaries, env files, firewalld service +
# definition, SELinux port label, and (when no moments component env files
# remain) the shared /etc/moments + /var/lib/moments dirs and the sysusers
# entry. Idempotent — safe to re-run.
#
# Notes:
# - The host argument is explicit on purpose: you typically tear down on
# hosts you've already removed from manifest.components.<c>.hosts.
# - Manifest is still read for env-wide config (api port, server_name,
# docroot path), so $environment must still resolve.
# - The `moments` user/group is intentionally NOT removed: any leftover
# file owned by it would become orphan-owned. Run `userdel moments`
# manually if you're certain there are none.
# - Web docroot is left intact unless --remove-docroot is given.
set -euo pipefail
shopt -s nullglob
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
manifest="${repo_root}/asset/manifest.yml"
dry_run=0
remove_docroot=0
usage() {
cat <<EOF >&2
usage: $(basename "$0") <environment> <host> [component...] [--dry-run] [--remove-docroot]
$(basename "$0") prod anjie.kosherinata.internal api worker
$(basename "$0") prod oolon.kosherinata.internal web --remove-docroot
$(basename "$0") prod anjie.kosherinata.internal all
components: api | worker | web | all
EOF
exit 2
}
log() { printf '\033[1;34m[teardown]\033[0m %s\n' "$*" >&2; }
warn() { printf '\033[1;33m[teardown]\033[0m %s\n' "$*" >&2; }
die() { printf '\033[1;31m[teardown]\033[0m %s\n' "$*" >&2; exit 1; }
ssh_run() {
local host="$1"; shift
if (( dry_run )); then
printf '\033[2m[dry-run]\033[0m ssh %s -- %s\n' "$host" "$*" >&2
else
ssh -o BatchMode=yes "$host" "$@"
fi
}
[[ $# -ge 2 ]] || usage
environment="$1"; shift
target_host="$1"; shift
components=()
while [[ $# -gt 0 ]]; do
case "$1" in
--dry-run) dry_run=1 ;;
--remove-docroot) remove_docroot=1 ;;
*) components+=("$1") ;;
esac
shift
done
[[ -f "$manifest" ]] || die "manifest not found: $manifest"
command -v yq >/dev/null 2>&1 || die "yq is required"
env_path=".environments.${environment}"
yq --exit-status "${env_path}" "$manifest" >/dev/null \
|| die "environment '$environment' not found in manifest"
if [[ ${#components[@]} -eq 0 ]]; then
usage
fi
if [[ "${components[0]:-}" == "all" ]]; then
components=(api worker web)
fi
teardown_api() {
local host="$1"
log "api -> $host"
local bind api_port=""
bind="$(yq --raw-output "${env_path}.components.api.config.bind" "$manifest")"
if [[ -n "$bind" && "$bind" != "null" && "$bind" == *:* ]]; then
api_port="${bind##*:}"
[[ "$api_port" =~ ^[0-9]+$ ]] || api_port=""
fi
if (( dry_run )); then
printf '\033[2m[dry-run]\033[0m stop+disable moments-api units, remove unit files, /etc/moments/api.env, /usr/local/bin/moments-api, firewalld svc moments-api, SELinux label tcp/%s on %s\n' \
"${api_port:-<unknown>}" "$host" >&2
return 0
fi
ssh_run "$host" "sudo bash -s -- ${api_port@Q}" <<'REMOTE_EOF'
set -euo pipefail
api_port="$1"
# Stop + disable units. `disable --now` quietly does nothing on a unit that
# isn't loaded, but emits non-zero exit on some systemd versions when the
# file is already gone — swallow that so re-runs are clean.
for unit in moments-api.service moments-api-cert.path moments-api-cert-reload.service; do
systemctl disable --now "$unit" 2>/dev/null || true
done
rm --force \
/etc/systemd/system/moments-api.service \
/etc/systemd/system/moments-api-cert.path \
/etc/systemd/system/moments-api-cert-reload.service
systemctl daemon-reload
rm --force /etc/moments/api.env /usr/local/bin/moments-api
# Firewalld: remove service from default zone, then drop service definition.
zone="$(firewall-cmd --get-default-zone)"
if firewall-cmd --zone="$zone" --query-service=moments-api >/dev/null 2>&1; then
firewall-cmd --permanent --zone="$zone" --remove-service=moments-api
firewall-cmd --zone="$zone" --remove-service=moments-api 2>/dev/null || true
fi
rm --force /etc/firewalld/services/moments-api.xml
firewall-cmd --reload
# SELinux: remove the port label, if we know which port. --delete fails when
# the port wasn't user-labelled — that's fine, swallow it.
if [[ -n "$api_port" ]]; then
semanage port --delete --proto=tcp "$api_port" 2>/dev/null || true
fi
echo "moments-api torn down"
REMOTE_EOF
}
teardown_worker() {
local host="$1"
log "worker -> $host"
if (( dry_run )); then
printf '\033[2m[dry-run]\033[0m stop+disable moments-worker units, remove unit files, /etc/moments/worker.env, /usr/local/bin/moments-worker on %s\n' \
"$host" >&2
return 0
fi
ssh_run "$host" "sudo bash -s" <<'REMOTE_EOF'
set -euo pipefail
for unit in moments-worker.service moments-worker-cert.path moments-worker-cert-reload.service; do
systemctl disable --now "$unit" 2>/dev/null || true
done
rm --force \
/etc/systemd/system/moments-worker.service \
/etc/systemd/system/moments-worker-cert.path \
/etc/systemd/system/moments-worker-cert-reload.service
systemctl daemon-reload
rm --force /etc/moments/worker.env /usr/local/bin/moments-worker
echo "moments-worker torn down"
REMOTE_EOF
}
teardown_web() {
local host="$1"
log "web -> $host"
local server_name web_root
server_name="$(yq --raw-output "${env_path}.components.web.config.server_name" "$manifest")"
web_root="$(yq --raw-output "${env_path}.components.web.config.root" "$manifest")"
[[ -n "$server_name" && "$server_name" != "null" ]] || die "web.config.server_name missing in manifest"
[[ -n "$web_root" && "$web_root" != "null" ]] || die "web.config.root missing in manifest"
[[ "$web_root" == /* ]] || die "web.config.root must be an absolute path: '$web_root'"
# Refuse to recursively remove a shallow or system path even if the
# manifest says so.
if (( remove_docroot )); then
case "$web_root" in
/|/bin|/bin/*|/boot|/boot/*|/dev|/dev/*|/etc|/etc/*|/home|/home/*|/lib|/lib/*|/lib64|/lib64/*|/proc|/proc/*|/root|/root/*|/run|/run/*|/sbin|/sbin/*|/srv|/srv/*|/sys|/sys/*|/tmp|/tmp/*|/usr|/usr/*|/var|/var/lib|/var/log|/var/run|/var/spool|/var/www)
die "refusing to recursively remove a system path: '$web_root'"
;;
esac
# Require at least three path components (e.g. /var/www/<site>) to
# rule out things like /opt or /srv directly.
[[ "$web_root" =~ ^/[^/]+/[^/]+/[^/]+ ]] \
|| die "refusing to recursively remove a path with fewer than 3 components: '$web_root'"
fi
local site_conf_path="/etc/nginx/conf.d/${server_name}.conf"
if (( dry_run )); then
if (( remove_docroot )); then
printf '\033[2m[dry-run]\033[0m remove %s, recursively remove %s, nginx -t/reload on %s\n' \
"$site_conf_path" "$web_root" "$host" >&2
else
printf '\033[2m[dry-run]\033[0m remove %s, nginx -t/reload on %s (docroot %s left intact; pass --remove-docroot to also clear it)\n' \
"$site_conf_path" "$host" "$web_root" >&2
fi
return 0
fi
ssh_run "$host" "sudo bash -s -- ${site_conf_path@Q} ${web_root@Q} ${remove_docroot@Q}" <<'REMOTE_EOF'
set -euo pipefail
site_conf_path="$1"
web_root="$2"
remove_docroot="$3"
rm --force "$site_conf_path"
if nginx -t 2>&1; then
systemctl reload nginx
echo "nginx reloaded without ${site_conf_path}"
else
echo "nginx -t failed AFTER removing ${site_conf_path}; check other site configs" >&2
exit 1
fi
if [[ "$remove_docroot" == "1" && -d "$web_root" ]]; then
rm --recursive --force "$web_root"
echo "removed docroot ${web_root}"
fi
REMOTE_EOF
}
teardown_shared() {
local host="$1"
log "shared (post-component cleanup) -> $host"
if (( dry_run )); then
printf '\033[2m[dry-run]\033[0m if no api.env/worker.env remain: remove /etc/sysusers.d/moments.conf and rmdir /etc/moments + /var/lib/moments on %s (moments user left in place)\n' \
"$host" >&2
return 0
fi
ssh_run "$host" "sudo bash -s" <<'REMOTE_EOF'
set -euo pipefail
# If any component env still exists, leave shared state alone — another
# moments component is still using /etc/moments and the moments user.
if [[ -e /etc/moments/api.env || -e /etc/moments/worker.env ]]; then
echo "moments env files still present; leaving /etc/moments + /var/lib/moments + sysusers entry in place"
exit 0
fi
# rmdir refuses non-empty dirs — defensive against unknown stragglers.
rmdir /etc/moments 2>/dev/null || true
rmdir /var/lib/moments 2>/dev/null || true
rm --force /etc/sysusers.d/moments.conf
echo "shared state cleared (where empty); moments user/group intentionally left in place"
REMOTE_EOF
}
# Dispatch ------------------------------------------------------------------
failed=()
did_app=0
for component in "${components[@]}"; do
case "$component" in
api) teardown_api "$target_host" || failed+=("api@$target_host") ;;
worker) teardown_worker "$target_host" || failed+=("worker@$target_host") ;;
web) teardown_web "$target_host" || failed+=("web@$target_host") ;;
*) warn "unknown component: $component" ;;
esac
case "$component" in
api|worker) did_app=1 ;;
esac
done
# Shared cleanup runs after api/worker teardown. It's a no-op if either
# component still has its env file present on the host.
if (( did_app )); then
teardown_shared "$target_host" || failed+=("shared@$target_host")
fi
if [[ ${#failed[@]} -gt 0 ]]; then
die "failed: ${failed[*]}"
fi
log "teardown complete on $target_host"

74
ui/index.html Normal file
View File

@@ -0,0 +1,74 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>rob.tn</title>
<meta
name="description"
content="a timeline of open source contributions across github, gitea, and mozilla hg. ranked projects, language trends, and commit activity since 2012."
/>
<meta
property="og:title"
content="rob thijssen: developer activity and contribution history"
/>
<meta
property="og:description"
content="a timeline of open source contributions across github, gitea, and mozilla hg. ranked projects, language trends, and commit activity since 2012."
/>
<meta
property="og:image"
content="https://rob.tn/api/v1/og/contributions.png"
width="1200"
height="630"
/>
<meta property="og:type" content="website" />
<meta property="og:url" content="https://rob.tn/" />
<meta property="og:site_name" content="rob.tn" />
<meta property="og:locale" content="en_US" />
<meta property="og:logo" content="https://rob.tn/icon-512.png" />
<meta name="twitter:card" content="summary_large_image" />
<meta
name="twitter:image"
content="https://rob.tn/api/v1/og/contributions.png"
width="1200"
height="630"
/>
<link rel="icon" type="image/x-icon" href="/favicon.ico" />
<link
rel="icon"
type="image/png"
sizes="16x16"
href="/favicon-16.png"
/>
<link
rel="icon"
type="image/png"
sizes="32x32"
href="/favicon-32.png"
/>
<link
rel="icon"
type="image/png"
sizes="48x48"
href="/favicon-48.png"
/>
<link rel="apple-touch-icon" href="/apple-touch-icon.png" />
<link
rel="icon"
type="image/png"
sizes="192x192"
href="/icon-192.png"
/>
<link
rel="icon"
type="image/png"
sizes="512x512"
href="/icon-512.png"
/>
</head>
<body>
<div id="root"></div>
<script type="module" src="/src/main.tsx"></script>
</body>
</html>

35
ui/package.json Normal file
View File

@@ -0,0 +1,35 @@
{
"name": "moments-ui",
"private": true,
"version": "0.1.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "tsc -b && vite build",
"preview": "vite preview",
"lint": "tsc --noEmit"
},
"dependencies": {
"@tanstack/react-query": "^5.62.0",
"bootstrap": "^5.3.3",
"rc-slider": "^11.1.7",
"react": "^19.0.0",
"react-bootstrap": "^2.10.6",
"react-bootstrap-icons": "^1.11.4",
"react-dom": "^19.0.0",
"react-markdown": "^9.0.1",
"react-router-dom": "^7.14.2",
"react-vertical-timeline-component": "^3.6.0",
"rehype-raw": "^7.0.0",
"rehype-sanitize": "^6.0.0",
"remark-gfm": "^4.0.1"
},
"devDependencies": {
"@types/react": "^19.0.0",
"@types/react-dom": "^19.0.0",
"@types/react-vertical-timeline-component": "^3.3.6",
"@vitejs/plugin-react-swc": "^3.7.2",
"typescript": "~5.7.0",
"vite": "^6.0.0"
}
}

2400
ui/pnpm-lock.yaml generated Normal file
View File

File diff suppressed because it is too large Load Diff

BIN
ui/public/apple-touch-icon.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 37 KiB

BIN
ui/public/bean.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 134 KiB

BIN
ui/public/favicon-16.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.5 KiB

BIN
ui/public/favicon-32.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.4 KiB

BIN
ui/public/favicon-48.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.4 KiB

BIN
ui/public/favicon.ico Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

23
ui/public/gitea.svg Normal file
View File

@@ -0,0 +1,23 @@
<?xml version="1.0" encoding="utf-8" ?>
<svg
xmlns="http://www.w3.org/2000/svg"
aria-label="Gitea"
role="img"
viewBox="0 0 512 512"
>
<rect rx="15%" height="512" width="512" fill="#ffffff" />
<path
d="M419 150c-98 7-186 2-276-1-27 0-63 19-61 67 3 75 71 82 99 83 3 14 35 62 59 65h104c63-5 109-213 75-214zm-311 67c-3-21 7-42 42-42 3 39 10 61 22 96-32-5-59-15-64-54z"
fill="#592"
/>
<path d="m293 152v70" stroke="#ffffff" stroke-width="9" />
<g transform="rotate(25.7 496 -423)" stroke-width="7" fill="#592">
<path d="M561 246h97" stroke="#592" />
<rect x="561" y="246" width="97" height="97" rx="16" fill="#ffffff" />
<path d="M592 245v75" stroke="#592" />
<path d="M592 273c45 0 38-5 38 48" fill="none" stroke="#592" />
<circle cx="592" cy="320" r="10" />
<circle cx="630" cy="320" r="10" />
<circle cx="592" cy="273" r="10" />
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 951 B

14
ui/public/github.svg Normal file
View File

@@ -0,0 +1,14 @@
<?xml version="1.0" encoding="utf-8" ?>
<svg
fill="#ffffff"
width="800px"
height="800px"
viewBox="0 0 24 24"
role="img"
xmlns="http://www.w3.org/2000/svg"
>
<title>github</title>
<path
d="M12 .297c-6.63 0-12 5.373-12 12 0 5.303 3.438 9.8 8.205 11.385.6.113.82-.258.82-.577 0-.285-.01-1.04-.015-2.04-3.338.724-4.042-1.61-4.042-1.61C4.422 18.07 3.633 17.7 3.633 17.7c-1.087-.744.084-.729.084-.729 1.205.084 1.838 1.236 1.838 1.236 1.07 1.835 2.809 1.305 3.495.998.108-.776.417-1.305.76-1.605-2.665-.3-5.466-1.332-5.466-5.93 0-1.31.465-2.38 1.235-3.22-.135-.303-.54-1.523.105-3.176 0 0 1.005-.322 3.3 1.23.96-.267 1.98-.399 3-.405 1.02.006 2.04.138 3 .405 2.28-1.552 3.285-1.23 3.285-1.23.645 1.653.24 2.873.12 3.176.765.84 1.23 1.91 1.23 3.22 0 4.61-2.805 5.625-5.475 5.92.42.36.81 1.096.81 2.22 0 1.606-.015 2.896-.015 3.286 0 .315.21.69.825.57C20.565 22.092 24 17.592 24 12.297c0-6.627-5.373-12-12-12"
/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 956 B

BIN
ui/public/icon-192.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

BIN
ui/public/icon-512.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 218 KiB

BIN
ui/public/icon-maskable-512.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 153 KiB

30
ui/public/mozilla.svg Normal file
View File

@@ -0,0 +1,30 @@
<?xml version="1.0" standalone="no" ?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg
width="800px"
height="800px"
viewBox="-10 -5 1034 1034"
xmlns="http://www.w3.org/2000/svg"
xmlns:xlink="http://www.w3.org/1999/xlink"
version="1.1"
>
<path
fill="#ffffff"
d="M411 237q-31 1 -64 10q-27 8 -54 22q-20 9 -38 20l-14 10q-50 33 -120 94q-67 57 -108 100q8 1 23 -5q12 -4 18 -5l99 -27l-32 26q-47 38 -69 59q-39 36 -52 62q11 -4 42 -19t46 -21q26 -11 40 -12l-18 16q-33 30 -49 46q-28 27 -38 44q1 1 1 4v2l10 -6q33 -20 52 -29
q32 -16 47 -18q-77 83 -89 125q19 -13 55 -32.5t55 -26.5q-3 8 -15 23q-9 12 -12.5 18.5t-15.5 21.5t-17 23q-8 14 -9 25q30 -29 73 -53q-4 10 -15 27q-10 15 -14 23q-7 13 -9 26q9 -10 27 -18q11 -5 33 -12l11 -4l-4 4q-27 21 -42 39l8 -1q2 -1 9 1l6 1l-32 13
q143 188 395 166l-13 -28q10 -18 19 -63t19 -66q16 -34 48 -51l25 -14q24 -14 37 -17q18 -4 49 2q8 1 32 7q35 9 52 12q30 6 46 2l10 -5q36 -22 58 -39q24 5 34.5 1.5t18.5 -16.5l5 -14q18 -50 28 -72l1 -15q-7 -21 -11 -75q-2 -32 -5 -36q-13 -18 -45 -34q-20 -10 -67 -27
q-54 -20 -79 -33q-43 -21 -66 -47q8 -25 -5 -51q-10 -19 -32 -40q-25 -18 -63 -16q-24 1 -64 12q-30 -6 -86 -24q-29 -9 -44 -14q-8 -1 -18 -1h-4zM410 285q22 0 64 12q27 7 41 9q-1 4 -6.5 8t-6.5 7l-4 6q-4 5 -5.5 8.5t0.5 7.5l32 -12q13 -4 38 -12q32 -11 48 -14
q28 -6 51 -3l24 31q-24 -20 -55 -15q-19 3 -56 20q-33 15 -49 18q-11 2 -33 8q-17 5 -25 7q29 22 75 17q3 15 18 34.5t29 22.5q-1 -6 -7 -20.5t-7 -22.5q-3 -14 4 -25l9 -6q17 -10 26 -13q16 -5 27 4q-10 1 -26 11l-8 5q-6 4 -8.5 14t-0.5 14q7 20 39 33q36 14 90 14
q-3 -4 -16 -18.5t-18 -21.5q-9 -11 -8 -17q30 30 81 55q30 14 89 35q35 12 49 18q22 10 31 19q1 5 -13 22q-7 9 -9 13q-4 7 -2 10q-3 7 7 22q5 8 22 29l3 4q-3 -25 -1.5 -36.5t6 -9.5t8.5 14.5t3 28.5q0 19 -7 37l-35 -8q-47 -3 -84 -15q-33 -9 -66 -28q-21 -12 -65 -42
l-30 -20q-27 -18 -79 -60l-14 -11l-50 -19q14 17 36 41q20 22 25 30q7 12 5 24.5t-16 36.5l-17 -31l-7 -55q-11 40 -13 66q-2 36 13 62q17 30 58 49l50 7q-22 -18 -30 -29q-12 -16 -8 -31q15 30 78 55q35 14 105 32l24 6q15 6 24 15q-7 6 -13 9q-4 2 -13 4.5t-15 4.5l-28 -9
q-36 -11 -51 -18q-26 -11 -36 -25q4 6 -27 14q-18 4 -63 13q-29 5 -37 7q-13 3 -3 4q-35 -1 -72 -23q-33 -19 -61 -50l-9 -3q0 17 11 35q6 11 22.5 32t22.5 31l38 17l-10 6q-16 10 -23 16q-13 10 -19 21l-2 6q-5 9 -6 14q-2 8 1 14q21 -23 61 -33l-16 25q2 18 -9 49
q-2 -5 -5 -10q-47 -20 -71 -39q6 20 25 44q16 21 37 38q-5 12 -12 23q-36 -25 -54 -42q5 9 11 27q9 24 15 34q-133 -10 -217 -82l60 -34q-10 2 -47 11l-30 8l-7 -8l24 -12q29 -14 42 -22q23 -14 29 -26v0l-6 2l-61 3q-4 -13 0 -27q2 -9 10.5 -24.5t9.5 -23.5q2 -13 -7 -25
q-35 -46 -49 -80q-20 -46 -15 -90l1 2q7 15 13 20l2 -17q2 -18 5 -26v1q6 20 11 29q8 16 21 24l1 -2q-14 -50 -2 -95q13 -51 56 -78q-6 2 -23 4q-26 4 -41 10q2 -1 5 -10q11 -29 23 -39q6 -4 15 -13q11 -10 19 -15q24 -19 43 -28q26 -14 59 -19q8 -1 16 -1h4zM425 332
q-28 0 -60 12l-13 10q-14 17 -19 26q-10 14 -12 29l3 4q13 -16 37 -28q0 18 9.5 33.5t25.5 24.5h3q-3 -19 -1 -39.5t9 -37.5q7 -6 23 -14q18 -10 25 -17q-14 -4 -30 -3zM625 391v0q8 0 15 3q-6 3 -8 11q-1 4 0 7q-11 -8 -11 -21h4zM462 392q2 7 14 22l9 11q-28 2 -60 11
q5 4 14.5 9t13.5 9l-16 6q-21 7 -31 11q-16 8 -28 19l5 2q31 -6 66 -5.5t67 7.5l3 -2l-25 -46l38 -8q-39 -37 -70 -46zM662 406q8 7 15 16q-11 3 -22 0q5 -3 7 -10v-6zM599 430q13 39 37 41q33 6 62 1q-12 -7 -39 -13q-22 -6 -32 -10q-17 -7 -28 -19zM368 492
q-23 17 -32.5 42t-5.5 53q2 7 1 17q-1 6 -4 17q-4 17 -3 25q0 13 8 22q22 25 63 55l-2 -8q-35 -41 -46 -74l17 6q36 13 55 16q-6 -10 -22 -29q-28 -34 -36 -53q-14 -32 -4 -62q0 -5 6 -12q7 -10 5 -15zM728 621q2 0 5 1q8 2 9 4q-3 2 -8.5 7t-8.5 6l1 -6q0 -9 2 -12z
M768 642v0q4 0 9 4h1l-13 13l1 -17h2zM798 653v0l10 6q-1 0 -6 7t-5.5 6.5t-0.5 -6.5q0 -13 2 -13zM829 667v0l9 2q-1 1 -4 7q-6 11 -11 11l4 -8v-9q0 -3 2 -3zM861 673l12 3l-13 22zM896 679q1 0 8 2l4 2l-15 11q-1 -9 -0.5 -12t3.5 -3zM933 685q5 0 7 2q0 5 -7 12
q-4 4 -5 7v-6q2 -10 0 -15h5z"
/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 3.8 KiB

14
ui/public/robots.txt Normal file
View File

@@ -0,0 +1,14 @@
User-agent: *
Allow: /
User-agent: facebookexternalhit
Allow: /
User-agent: Twitterbot
Allow: /
User-agent: LinkedInBot
Allow: /
User-agent: WhatsApp
Allow: /

187
ui/src/App.css Normal file
View File

@@ -0,0 +1,187 @@
body {
background-color: #2c3e50;
color: #ecf0f1;
text-transform: lowercase;
}
.container {
color: #ecf0f1;
}
a {
color: #ff4081;
text-decoration: none;
}
a:hover {
color: #ff80ab;
text-decoration: underline;
}
.hot-pink,
a.hot-pink {
color: #ff4081;
}
/* react-vertical-timeline-component date label sits in the gutter — readable
against the dark backdrop. */
.vertical-timeline-element-date {
color: #ecf0f1 !important;
opacity: 0.8;
}
.vertical-timeline-element-content {
color: #2c3e50;
}
.vertical-timeline-element-content h4.vertical-timeline-element-title {
font-size: 0.85rem;
}
.vertical-timeline-element-content h5.vertical-timeline-element-subtitle {
font-size: 0.75rem;
}
.vertical-timeline-element-content p,
.vertical-timeline-element-content ul,
.vertical-timeline-element-content li,
.vertical-timeline-element-content code {
font-size: 0.75rem;
}
.vertical-timeline-element-content a {
color: #1565c0;
}
.site-header h1 {
font-size: 1.75rem;
}
.site-header nav a {
color: #ecf0f1;
opacity: 0.7;
}
.site-header nav a:hover {
color: #ff4081;
opacity: 1;
text-decoration: none;
}
.site-header nav a.active {
color: #ff4081;
opacity: 1;
}
.nav-divider {
opacity: 0.3;
}
.graph-label {
fill: #ecf0f1;
font-size: 9px;
opacity: 0.6;
}
.graph-cell {
cursor: pointer;
transition: opacity 0.15s;
}
.graph-cell:hover {
opacity: 0.8;
stroke: #ecf0f1;
stroke-width: 1;
}
.project-card {
background: rgba(255, 255, 255, 0.05);
border: 1px solid rgba(255, 255, 255, 0.1);
border-radius: 6px;
height: 100%;
}
.project-card h5 {
font-size: 0.9rem;
}
.forge-icon {
width: 16px;
height: 16px;
margin-right: 6px;
vertical-align: -2px;
opacity: 0.7;
}
.project-card a {
color: #ff4081;
}
.project-card .text-muted {
color: rgba(236, 240, 241, 0.5) !important;
font-size: 0.7rem;
}
.project-card h5,
.project-card .text-muted,
.project-card span {
color: #ecf0f1;
}
.language-bar {
display: flex;
height: 6px;
border-radius: 3px;
overflow: hidden;
}
.language-bar-segment {
min-width: 2px;
}
.language-dot {
display: inline-block;
width: 8px;
height: 8px;
border-radius: 50%;
margin-right: 4px;
}
.project-readme {
background: rgba(255, 255, 255, 0.05);
border: 1px solid rgba(255, 255, 255, 0.1);
border-radius: 6px;
padding: 1.5rem;
font-size: 0.85rem;
line-height: 1.5;
}
.project-readme h1,
.project-readme h2,
.project-readme h3 {
font-size: 1.1rem;
margin-top: 1rem;
}
.project-readme pre {
background: rgba(0, 0, 0, 0.2);
padding: 0.75rem;
border-radius: 4px;
overflow-x: auto;
}
.project-readme code {
font-size: 0.8rem;
}
.project-readme img {
max-width: 100%;
}
.site-footer {
margin-top: 3rem;
padding: 1rem 0;
font-size: 0.75rem;
opacity: 0.6;
text-align: center;
}

27
ui/src/App.tsx Normal file
View File

@@ -0,0 +1,27 @@
import { Routes, Route } from 'react-router-dom';
import 'bootstrap/dist/css/bootstrap.min.css';
import 'rc-slider/assets/index.css';
import 'react-vertical-timeline-component/style.min.css';
import './App.css';
import { Layout } from './components/Layout';
import { DashPage } from './pages/DashPage';
import { TimelineHome } from './pages/TimelineHome';
import { ProjectPage } from './pages/ProjectPage';
import { CvPage } from './pages/CvPage';
export default function App() {
return (
<Routes>
<Route element={<Layout />}>
<Route index element={<DashPage />} />
<Route path="/dash" element={<DashPage />} />
<Route path="/activity" element={<TimelineHome />} />
<Route path="/activity/:timespan" element={<TimelineHome />} />
<Route path="/project/:source/*" element={<ProjectPage />} />
<Route path="/cv" element={<CvPage />} />
</Route>
</Routes>
);
}

185
ui/src/api/client.ts Normal file
View File

@@ -0,0 +1,185 @@
// Wire types mirror the moments-entities types serialised by the API.
// Hand-maintained for now; if drift becomes a problem, generate them
// from the Rust crate via ts-rs or specta.
export type Source = 'github' | 'gitea' | 'hg' | 'bugzilla';
export type TitleSegment =
| { kind: 'text'; text: string }
| { kind: 'link'; text: string; url: string };
export interface CommitSummary {
sha: string;
short_sha: string;
message: string;
url: string;
author: string | null;
}
export type TimelineBody =
| { kind: 'markdown'; text: string }
| { kind: 'commits'; commits: CommitSummary[] }
| { kind: 'links'; items: TitleSegment[] };
export type TimelineIcon =
| 'git-push'
| 'git-commit'
| 'git-merge'
| 'git-fork'
| 'git-branch-create'
| 'git-branch-delete'
| 'pull-request'
| 'issue'
| 'comment'
| 'star'
| 'release'
| 'bug'
| 'generic';
export interface TimelineItem {
id: string;
source: Source;
action: string;
occurred_at: string;
icon: TimelineIcon;
title: TitleSegment[];
subtitle: TitleSegment[] | null;
body: TimelineBody | null;
}
export interface SourceSummary {
source: Source;
count: number;
earliest: string | null;
latest: string | null;
}
export interface ProjectSummary {
repo: string;
source: Source;
host: string;
commit_count: number;
issue_count: number;
pr_count: number;
first_activity: string | null;
last_activity: string | null;
}
export interface DailyCount {
date: string;
count: number;
}
export interface HourlyAvg {
hour: number;
avg: number;
}
export interface LanguageDailyCount {
date: string;
language: string;
color: string | null;
commits: number;
}
export interface EventQuery {
from?: Date;
to?: Date;
sources?: Source[];
repo?: string;
limit?: number;
}
const API_BASE = '/api/v1';
/** Decode base64 content as UTF-8 (atob only handles Latin-1). */
function decodeBase64Utf8(b64: string): string {
const bytes = Uint8Array.from(atob(b64), (c) => c.charCodeAt(0));
return new TextDecoder().decode(bytes);
}
export async function fetchEvents(q: EventQuery): Promise<TimelineItem[]> {
const params = new URLSearchParams();
if (q.from) params.set('from', q.from.toISOString());
if (q.to) params.set('to', q.to.toISOString());
if (q.sources && q.sources.length > 0) {
params.set('source', q.sources.join(','));
}
if (q.repo) params.set('repo', q.repo);
if (q.limit) params.set('limit', String(q.limit));
const resp = await fetch(`${API_BASE}/events?${params}`);
if (!resp.ok) throw new Error(`events: HTTP ${resp.status}`);
return resp.json();
}
export async function fetchSources(): Promise<SourceSummary[]> {
const resp = await fetch(`${API_BASE}/sources`);
if (!resp.ok) throw new Error(`sources: HTTP ${resp.status}`);
return resp.json();
}
export async function fetchDailyCounts(from: string, to: string): Promise<DailyCount[]> {
const resp = await fetch(`${API_BASE}/activity/daily?from=${from}&to=${to}`);
if (!resp.ok) throw new Error(`daily-counts: HTTP ${resp.status}`);
return resp.json();
}
export async function fetchHourlyAvgs(from: string, to: string, tz: string): Promise<HourlyAvg[]> {
const qs = new URLSearchParams({ from, to, tz });
const resp = await fetch(`${API_BASE}/activity/hourly?${qs}`);
if (!resp.ok) throw new Error(`hourly-avgs: HTTP ${resp.status}`);
return resp.json();
}
export async function fetchLanguageDailyCounts(from: string, to: string): Promise<LanguageDailyCount[]> {
const resp = await fetch(`${API_BASE}/languages/daily?from=${from}&to=${to}`);
if (!resp.ok) throw new Error(`language-daily-counts: HTTP ${resp.status}`);
return resp.json();
}
export interface RepoLanguageEntry {
source: Source;
repo: string;
language: string;
bytes: number;
color: string | null;
}
export async function fetchRepoLanguages(): Promise<RepoLanguageEntry[]> {
const resp = await fetch(`${API_BASE}/languages/repos`);
if (!resp.ok) throw new Error(`repo-languages: HTTP ${resp.status}`);
return resp.json();
}
export async function fetchProjects(): Promise<ProjectSummary[]> {
const resp = await fetch(`${API_BASE}/projects`);
if (!resp.ok) throw new Error(`projects: HTTP ${resp.status}`);
return resp.json();
}
/** Fetch repo README as raw markdown via the forge proxy. */
export async function fetchReadme(source: Source, host: string, repo: string): Promise<string | null> {
if (source === 'github') {
const resp = await fetch(`${API_BASE}/forge/github/repos/${repo}/readme`);
if (!resp.ok) return null;
const data = await resp.json();
if (data.encoding === 'base64' && data.content) {
return decodeBase64Utf8(data.content);
}
return data.content ?? null;
}
if (source === 'gitea') {
for (const name of ['README.md', 'readme.md', 'Readme.md']) {
const resp = await fetch(`${API_BASE}/forge/gitea/repos/${repo}/contents/${name}?host=${encodeURIComponent(host)}`);
if (!resp.ok) continue;
const data = await resp.json();
if (data.encoding === 'base64' && data.content) {
return decodeBase64Utf8(data.content);
}
if (data.content) return data.content;
}
return null;
}
return null;
}

82
ui/src/api/cv.ts Normal file
View File

@@ -0,0 +1,82 @@
// Fetches the CV gist at runtime and returns the parsed config + file
// list. The legacy implementation (cv/src/App.js) hits the same endpoint
// and relies entirely on the inline `content` field — no per-file raw_url
// fetches. We do the same: one request, dedup'd via TanStack Query.
const GIST_OWNER = 'grenade';
const GIST_ID = '8e487477663c8e57c7bf31e8371f454a';
const GIST_API_URL = `https://api.github.com/gists/${GIST_ID}`;
const GIST_RAW_BASE = `https://gist.githubusercontent.com/${GIST_OWNER}/${GIST_ID}/raw`;
export const CV_PHOTO_URL = `${GIST_RAW_BASE}/rob.png`;
const CONFIG_FILENAME = 'cv-config.json';
export type SectionPlacement = 'body' | 'nav';
export type SortDirection = 'ascending' | 'descending';
export interface CvSectionConfig {
name: string;
filename_prefix: string;
order: number;
show_section_name: boolean;
placement: SectionPlacement;
sort?: {
on: 'filename';
direction: SortDirection;
};
}
export interface CvConfig {
sections: CvSectionConfig[];
}
export interface GistFile {
filename: string;
type: string;
language: string | null;
raw_url: string;
size: number;
truncated: boolean;
content: string;
}
interface GistResponse {
files: Record<string, GistFile>;
}
export interface CvData {
config: CvConfig;
files: Record<string, GistFile>;
}
export async function fetchCv(): Promise<CvData> {
const resp = await fetch(GIST_API_URL);
if (!resp.ok) {
throw new Error(`gist: HTTP ${resp.status} ${resp.statusText}`);
}
const gist = (await resp.json()) as GistResponse;
const cfgFile = gist.files[CONFIG_FILENAME];
if (!cfgFile) {
throw new Error(`gist: missing ${CONFIG_FILENAME}`);
}
const config = JSON.parse(cfgFile.content) as CvConfig;
return { config, files: gist.files };
}
// Pick out the gist files whose names start with the given prefix, applying
// the section's sort order. Mirrors the legacy filter at cv/src/App.js:67-68.
export function filesForSection(
data: CvData,
section: CvSectionConfig,
): GistFile[] {
const matches = Object.keys(data.files)
.filter((name) => name.startsWith(section.filename_prefix))
.sort();
if (section.sort?.direction === 'descending') {
matches.reverse();
}
return matches.map((name) => data.files[name]);
}

464
ui/src/components/ContributionGraph.tsx Normal file
View File

@@ -0,0 +1,464 @@
import { useMemo } from "react";
import { useQuery } from "@tanstack/react-query";
import { useNavigate } from "react-router-dom";
import {
fetchDailyCounts,
fetchLanguageDailyCounts,
fetchProjects,
fetchSources,
} from "../api/client";
const CELL_SIZE = 12;
const GAP = 3;
const RADIUS = CELL_SIZE / 2;
const ROWS = 7;
const LEFT_LABEL_WIDTH = 28;
const TOP_LABEL_HEIGHT = 16;
const DAY_LABELS = ["", "mon", "", "wed", "", "fri", ""];
const MONTH_LABELS = [
"jan",
"feb",
"mar",
"apr",
"may",
"jun",
"jul",
"aug",
"sep",
"oct",
"nov",
"dec",
];
const EMPTY_COLOR = "rgba(255,255,255,0.05)";
const FALLBACK_COLOR = "#39d353";
/** Daily contribution graph — last 1 year, one circle per day. */
export function ContributionGraph() {
const to = new Date();
const from = new Date(to);
from.setFullYear(from.getFullYear() - 1);
const fromStr = fmt(from);
const toStr = fmt(to);
const dailyQ = useQuery({
queryKey: ["daily-counts", fromStr, toStr],
queryFn: () => fetchDailyCounts(fromStr, toStr),
staleTime: 5 * 60_000,
});
const langQ = useQuery({
queryKey: ["language-daily", fromStr, toStr],
queryFn: () => fetchLanguageDailyCounts(fromStr, toStr),
staleTime: 5 * 60_000,
});
const projectsQ = useQuery({
queryKey: ["projects"],
queryFn: fetchProjects,
staleTime: 60_000,
});
const repoCount = useMemo(() => {
if (!projectsQ.data) return 0;
const fromMs = from.getTime();
const toMs = to.getTime();
return projectsQ.data.filter((p) => {
const first = p.first_activity
? new Date(p.first_activity).getTime()
: Infinity;
const last = p.last_activity ? new Date(p.last_activity).getTime() : 0;
return last >= fromMs && first <= toMs;
}).length;
}, [projectsQ.data]);
const navigate = useNavigate();
// Build map of date → dominant language color
const dayColorMap = useMemo(() => {
return buildDominantColorMap(langQ.data ?? []);
}, [langQ.data]);
const { weeks, monthMarkers, thresholds, totalCount } = useMemo(() => {
const counts = dailyQ.data ?? [];
const countMap = new Map(counts.map((d) => [d.date, d.count]));
const start = new Date(from);
start.setDate(start.getDate() - start.getDay());
const weeks: { date: string; count: number; col: number; row: number }[][] =
[];
const monthMarkers: { col: number; label: string }[] = [];
let col = 0;
let prevMonth = -1;
const cursor = new Date(start);
while (cursor <= to) {
const week: (typeof weeks)[0] = [];
for (let row = 0; row < ROWS; row++) {
const dateStr = fmt(cursor);
const count = countMap.get(dateStr) ?? 0;
week.push({ date: dateStr, count, col, row });
if (row === 0) {
const m = cursor.getMonth();
if (m !== prevMonth) {
monthMarkers.push({ col, label: MONTH_LABELS[m] });
prevMonth = m;
}
}
cursor.setDate(cursor.getDate() + 1);
}
weeks.push(week);
col++;
}
const nonZero = counts
.map((d) => d.count)
.filter((c) => c > 0)
.sort((a, b) => a - b);
const thresholds = computeThresholds(nonZero);
const totalCount = counts.reduce((sum, d) => sum + d.count, 0);
return { weeks, monthMarkers, thresholds, totalCount };
}, [dailyQ.data]);
const cols = weeks.length;
const svgWidth = LEFT_LABEL_WIDTH + cols * (CELL_SIZE + GAP);
const svgHeight = TOP_LABEL_HEIGHT + ROWS * (CELL_SIZE + GAP);
if (dailyQ.isLoading)
return <p style={{ fontSize: "0.8rem" }}>loading contribution graph...</p>;
if (dailyQ.isError) return null;
return (
<div className="contribution-graph mb-3">
<p style={{ fontSize: "0.8rem", opacity: 0.6 }}>
{new Intl.NumberFormat().format(totalCount)} contributions
{repoCount > 0 && `, across ${repoCount} repositories, `}
in the last year
</p>
<div>
<svg
viewBox={`0 0 ${svgWidth} ${svgHeight}`}
width="100%"
className="d-block"
>
{DAY_LABELS.map((label, i) =>
label ? (
<text
key={i}
x={LEFT_LABEL_WIDTH - 6}
y={TOP_LABEL_HEIGHT + i * (CELL_SIZE + GAP) + CELL_SIZE / 2}
textAnchor="end"
dominantBaseline="central"
className="graph-label"
>
{label}
</text>
) : null,
)}
{monthMarkers.map(({ col, label }, i) => (
<text
key={i}
x={LEFT_LABEL_WIDTH + col * (CELL_SIZE + GAP) + RADIUS}
y={10}
textAnchor="middle"
className="graph-label"
>
{label}
</text>
))}
{weeks.flatMap((week) =>
week.map(({ date, count, col, row }) => (
<circle
key={date}
cx={LEFT_LABEL_WIDTH + col * (CELL_SIZE + GAP) + RADIUS}
cy={TOP_LABEL_HEIGHT + row * (CELL_SIZE + GAP) + RADIUS}
r={RADIUS - 1}
fill={
count === 0
? EMPTY_COLOR
: (dayColorMap.get(date) ?? FALLBACK_COLOR)
}
opacity={count === 0 ? 1 : opacityFor(count, thresholds)}
className="graph-cell"
onClick={() => navigate(`/activity/${date}`)}
>
<title>{`${date}: ${count} ${count === 1 ? "contribution" : "contributions"}`}</title>
</circle>
)),
)}
</svg>
</div>
</div>
);
}
/** All-time monthly contribution graph — years on X axis, months on Y axis. */
export function AllTimeGraph() {
const sourcesQ = useQuery({
queryKey: ["sources"],
queryFn: fetchSources,
staleTime: 60_000,
});
const earliest = useMemo(() => {
if (!sourcesQ.data) return null;
const dates = sourcesQ.data
.map((s) => s.earliest)
.filter((d): d is string => d != null)
.map((d) => new Date(d));
return dates.length > 0
? new Date(Math.min(...dates.map((d) => d.getTime())))
: null;
}, [sourcesQ.data]);
const projectsQ = useQuery({
queryKey: ["projects"],
queryFn: fetchProjects,
staleTime: 60_000,
});
const repoCount = projectsQ.data?.length ?? 0;
const to = new Date();
const from = earliest ?? new Date(to.getFullYear() - 5, 0, 1);
const fromStr = fmt(from);
const toStr = fmt(to);
const dailyQ = useQuery({
queryKey: ["daily-counts-alltime", fromStr, toStr],
queryFn: () => fetchDailyCounts(fromStr, toStr),
enabled: !!earliest,
staleTime: 10 * 60_000,
});
const langQ = useQuery({
queryKey: ["language-daily-alltime", fromStr, toStr],
queryFn: () => fetchLanguageDailyCounts(fromStr, toStr),
enabled: !!earliest,
staleTime: 10 * 60_000,
});
const navigate = useNavigate();
// Aggregate daily language data to month level: pick the language with most commits
const monthColorMap = useMemo(() => {
const entries = langQ.data ?? [];
if (entries.length === 0) return new Map<string, string>();
const map = new Map<
string,
Map<string, { commits: number; color: string }>
>();
for (const e of entries) {
const key = e.date.slice(0, 7); // YYYY-MM
if (!map.has(key)) map.set(key, new Map());
const langMap = map.get(key)!;
const cur = langMap.get(e.language);
if (cur) {
cur.commits += e.commits;
} else {
langMap.set(e.language, {
commits: e.commits,
color: e.color ?? FALLBACK_COLOR,
});
}
}
const result = new Map<string, string>();
for (const [key, langMap] of map) {
let best = { commits: 0, color: FALLBACK_COLOR };
for (const v of langMap.values()) {
if (v.commits > best.commits) best = v;
}
result.set(key, best.color);
}
return result;
}, [langQ.data]);
const { years, monthGrid, thresholds, totalCount } = useMemo(() => {
const counts = dailyQ.data ?? [];
if (counts.length === 0)
return { years: [], monthGrid: [], thresholds: [1, 2, 3], totalCount: 0 };
const countMap = new Map(counts.map((d) => [d.date, d.count]));
const startYear = from.getFullYear();
const endYear = to.getFullYear();
const years: number[] = [];
for (let yr = startYear; yr <= endYear; yr++) years.push(yr);
// Build a 12 x years grid of monthly totals
const monthGrid: {
year: number;
month: number;
count: number;
monthStart: string;
monthEnd: string;
monthKey: string;
}[][] = [];
for (let m = 0; m < 12; m++) {
const row: (typeof monthGrid)[0] = [];
for (const yr of years) {
const monthStart = new Date(yr, m, 1);
const monthEnd = new Date(yr, m + 1, 0); // last day of month
const monthKey = `${yr}-${String(m + 1).padStart(2, "0")}`;
// Don't include months entirely outside our data range
if (monthStart > to || monthEnd < from) {
row.push({
year: yr,
month: m,
count: 0,
monthStart: fmt(monthStart),
monthEnd: fmt(monthEnd),
monthKey,
});
continue;
}
let total = 0;
const cursor = new Date(monthStart);
while (cursor <= monthEnd && cursor <= to) {
total += countMap.get(fmt(cursor)) ?? 0;
cursor.setDate(cursor.getDate() + 1);
}
row.push({
year: yr,
month: m,
count: total,
monthStart: fmt(monthStart),
monthEnd: fmt(monthEnd),
monthKey,
});
}
monthGrid.push(row);
}
const allCounts = monthGrid.flat().map((c) => c.count);
const nonZero = allCounts.filter((c) => c > 0).sort((a, b) => a - b);
const thresholds = computeThresholds(nonZero);
const totalCount = counts.reduce((sum, d) => sum + d.count, 0);
return { years, monthGrid, thresholds, totalCount };
}, [dailyQ.data]);
if (!earliest || dailyQ.isLoading) return null;
if (dailyQ.isError) return null;
if (years.length === 0) return null;
const monthLabelWidth = 28;
const topLabelHeight = 16;
const numCols = years.length;
const svgWidth = monthLabelWidth + numCols * (CELL_SIZE + GAP);
const svgHeight = topLabelHeight + 12 * (CELL_SIZE + GAP);
return (
<div className="contribution-graph mb-4">
<p style={{ fontSize: "0.8rem", opacity: 0.6 }}>
{new Intl.NumberFormat().format(totalCount)} contributions
{repoCount > 0 && `, across ${repoCount} repos, `}
since {fmt(from).split("-")[0]}
</p>
<div>
<svg
viewBox={`0 0 ${svgWidth} ${svgHeight}`}
width="100%"
className="d-block"
>
{/* Year labels along the top */}
{years.map((year, colIdx) => (
<text
key={year}
x={monthLabelWidth + colIdx * (CELL_SIZE + GAP) + RADIUS}
y={10}
textAnchor="middle"
className="graph-label"
>
{String(year).slice(2)}
</text>
))}
{/* Month labels along the left */}
{MONTH_LABELS.map((label, rowIdx) => (
<text
key={rowIdx}
x={monthLabelWidth - 6}
y={topLabelHeight + rowIdx * (CELL_SIZE + GAP) + CELL_SIZE / 2}
textAnchor="end"
dominantBaseline="central"
className="graph-label"
>
{label}
</text>
))}
{/* Monthly contribution circles */}
{monthGrid.map((row, rowIdx) =>
row.map(
({ year, count, monthStart, monthEnd, monthKey }, colIdx) => (
<circle
key={`${year}-${rowIdx}`}
cx={monthLabelWidth + colIdx * (CELL_SIZE + GAP) + RADIUS}
cy={topLabelHeight + rowIdx * (CELL_SIZE + GAP) + RADIUS}
r={RADIUS - 1}
fill={
count === 0
? EMPTY_COLOR
: (monthColorMap.get(monthKey) ?? FALLBACK_COLOR)
}
opacity={count === 0 ? 1 : opacityFor(count, thresholds)}
className="graph-cell"
onClick={() =>
navigate(`/activity/${monthStart}..${monthEnd}`)
}
>
<title>{`${MONTH_LABELS[rowIdx]} ${year}: ${count} ${count === 1 ? "contribution" : "contributions"}`}</title>
</circle>
),
),
)}
</svg>
</div>
</div>
);
}
function fmt(d: Date): string {
return d.toISOString().slice(0, 10);
}
/** Build a map of date → dominant (highest commit count) language color. */
function buildDominantColorMap(
entries: {
date: string;
language: string;
color: string | null;
commits: number;
}[],
): Map<string, string> {
const map = new Map<string, { commits: number; color: string }>();
for (const e of entries) {
const cur = map.get(e.date);
if (!cur || e.commits > cur.commits) {
map.set(e.date, { commits: e.commits, color: e.color ?? FALLBACK_COLOR });
}
}
const result = new Map<string, string>();
for (const [date, { color }] of map) {
result.set(date, color);
}
return result;
}
/** Map count to opacity (0.3 – 1.0) based on quartile thresholds. */
function opacityFor(count: number, thresholds: number[]): number {
if (count <= thresholds[0]) return 0.35;
if (count <= thresholds[1]) return 0.55;
if (count <= thresholds[2]) return 0.75;
return 1;
}
function computeThresholds(sorted: number[]): number[] {
if (sorted.length === 0) return [1, 2, 3];
const p = (pct: number) =>
sorted[Math.min(Math.floor(pct * sorted.length), sorted.length - 1)];
return [p(0.25), p(0.5), p(0.75)];
}

200
ui/src/components/ContributionStats.tsx Normal file
View File

@@ -0,0 +1,200 @@
import { useMemo } from 'react';
import { useQuery } from '@tanstack/react-query';
import { fetchDailyCounts, fetchHourlyAvgs, fetchSources } from '../api/client';
export function ContributionStats() {
const sourcesQ = useQuery({
queryKey: ['sources'],
queryFn: fetchSources,
staleTime: 60_000,
});
const earliest = useMemo(() => {
if (!sourcesQ.data) return null;
const dates = sourcesQ.data
.map((s) => s.earliest)
.filter((d): d is string => d != null)
.map((d) => new Date(d));
return dates.length > 0 ? new Date(Math.min(...dates.map((d) => d.getTime()))) : null;
}, [sourcesQ.data]);
const to = new Date();
const from = earliest ?? new Date(to.getFullYear() - 5, 0, 1);
const fromStr = fmt(from);
const toStr = fmt(to);
const dailyQ = useQuery({
queryKey: ['daily-counts-alltime', fromStr, toStr],
queryFn: () => fetchDailyCounts(fromStr, toStr),
enabled: !!earliest,
staleTime: 10 * 60_000,
});
// Bucket hour-of-day in the user's local timezone so the chart matches
// the clock they see. Browser may report e.g. "Europe/Helsinki"; fall
// back to UTC if the resolver returns something the server won't
// accept (it validates the string before binding).
const tz = useMemo(
() => Intl.DateTimeFormat().resolvedOptions().timeZone || 'UTC',
[],
);
const hourlyQ = useQuery({
queryKey: ['hourly-avgs-alltime', fromStr, toStr, tz],
queryFn: () => fetchHourlyAvgs(fromStr, toStr, tz),
enabled: !!earliest,
staleTime: 10 * 60_000,
});
const stats = useMemo(() => {
const counts = dailyQ.data ?? [];
if (counts.length === 0) return null;
// Build a set of dates with contributions
const countMap = new Map(counts.map((d) => [d.date, d.count]));
// Current streak (consecutive days ending today or yesterday with contributions)
let currentStreak = 0;
const cursor = new Date(to);
// Allow today to have 0 (day isn't over yet) — start from yesterday if today is 0
if ((countMap.get(fmt(cursor)) ?? 0) > 0) {
currentStreak = 1;
cursor.setDate(cursor.getDate() - 1);
} else {
cursor.setDate(cursor.getDate() - 1);
if ((countMap.get(fmt(cursor)) ?? 0) > 0) {
currentStreak = 1;
cursor.setDate(cursor.getDate() - 1);
}
}
if (currentStreak > 0) {
while ((countMap.get(fmt(cursor)) ?? 0) > 0) {
currentStreak++;
cursor.setDate(cursor.getDate() - 1);
}
}
// Longest streak
let longestStreak = 0;
let streak = 0;
const sorted = [...counts].sort((a, b) => a.date.localeCompare(b.date));
for (let i = 0; i < sorted.length; i++) {
if (sorted[i].count > 0) {
streak++;
if (streak > longestStreak) longestStreak = streak;
} else {
streak = 0;
}
}
// Busiest day
const busiest = sorted.reduce((best, d) => (d.count > best.count ? d : best), sorted[0]);
// Day-of-week averages
const dayTotals = [0, 0, 0, 0, 0, 0, 0];
const dayCounts = [0, 0, 0, 0, 0, 0, 0];
for (const d of sorted) {
const dow = new Date(d.date + 'T00:00:00').getDay();
dayTotals[dow] += d.count;
dayCounts[dow]++;
}
const dayNames = ['sun', 'mon', 'tue', 'wed', 'thu', 'fri', 'sat'];
const dayAvgs = dayNames.map((name, i) => ({
name,
avg: dayCounts[i] > 0 ? dayTotals[i] / dayCounts[i] : 0,
}));
const maxAvg = Math.max(...dayAvgs.map((d) => d.avg));
// Total active days
const activeDays = sorted.filter((d) => d.count > 0).length;
return { currentStreak, longestStreak, busiest, dayAvgs, maxAvg, activeDays };
}, [dailyQ.data]);
const hourly = useMemo(() => {
const data = hourlyQ.data ?? [];
if (data.length === 0) return null;
const byHour = new Array(24).fill(0);
for (const { hour, avg } of data) {
if (hour >= 0 && hour < 24) byHour[hour] = avg;
}
const max = Math.max(...byHour);
return { hours: byHour, max };
}, [hourlyQ.data]);
if (!stats) return null;
return (
<div>
<p style={{ fontSize: '0.8rem', opacity: 0.6 }}>contribution stats</p>
<div className="d-flex flex-column gap-2" style={{ fontSize: '0.8rem' }}>
<div className="d-flex justify-content-between">
<span style={{ opacity: 0.7 }}>current streak</span>
<span>{stats.currentStreak} {stats.currentStreak === 1 ? 'day' : 'days'}</span>
</div>
<div className="d-flex justify-content-between">
<span style={{ opacity: 0.7 }}>longest streak</span>
<span>{stats.longestStreak} {stats.longestStreak === 1 ? 'day' : 'days'}</span>
</div>
<div className="d-flex justify-content-between">
<span style={{ opacity: 0.7 }}>busiest day</span>
<span>{stats.busiest.count} on {stats.busiest.date}</span>
</div>
<div className="d-flex justify-content-between">
<span style={{ opacity: 0.7 }}>active days</span>
<span>{stats.activeDays.toLocaleString()}</span>
</div>
<div className="mt-1">
<span style={{ opacity: 0.7, fontSize: '0.75rem' }}>avg by weekday</span>
<div className="d-flex align-items-end gap-1 mt-1" style={{ height: 64 }}>
{stats.dayAvgs.map(({ name, avg }) => (
<div key={name} className="d-flex flex-column align-items-center" style={{ flex: 1, height: '100%', justifyContent: 'flex-end' }}>
<span style={{ fontSize: '0.65rem', opacity: 0.6, marginBottom: 2 }}>{avg.toFixed(1)}</span>
<div style={{ width: '100%', maxWidth: 20, borderRadius: 3, background: 'rgba(255,255,255,0.05)', flex: 1, display: 'flex', flexDirection: 'column', justifyContent: 'flex-end' }}>
<div
style={{
height: stats.maxAvg > 0 ? `${(avg / stats.maxAvg) * 100}%` : '0%',
borderRadius: 3,
backgroundColor: '#39d353',
opacity: 0.7,
}}
/>
</div>
<span style={{ fontSize: '0.65rem', opacity: 0.7, marginTop: 2 }}>{name}</span>
</div>
))}
</div>
</div>
{hourly && (
<div className="mt-3">
<span style={{ opacity: 0.7, fontSize: '0.75rem' }}>avg by hour ({tz})</span>
<div className="d-flex align-items-end gap-1 mt-1" style={{ height: 64 }}>
{hourly.hours.map((avg, h) => (
<div key={h} className="d-flex flex-column align-items-center" style={{ flex: 1, height: '100%', justifyContent: 'flex-end' }}>
<div style={{ width: '100%', borderRadius: 2, background: 'rgba(255,255,255,0.05)', flex: 1, display: 'flex', flexDirection: 'column', justifyContent: 'flex-end' }}>
<div
style={{
height: hourly.max > 0 ? `${(avg / hourly.max) * 100}%` : '0%',
borderRadius: 2,
backgroundColor: '#39d353',
opacity: 0.7,
}}
title={`${h.toString().padStart(2, '0')}:00 — ${avg.toFixed(2)}/day`}
/>
</div>
<span style={{ fontSize: '0.6rem', opacity: 0.7, marginTop: 2, minHeight: '0.7rem' }}>
{h % 4 === 0 ? h.toString().padStart(2, '0') : ''}
</span>
</div>
))}
</div>
</div>
)}
</div>
</div>
);
}
function fmt(d: Date): string {
return d.toISOString().slice(0, 10);
}

98
ui/src/components/Filters.tsx Normal file
View File

@@ -0,0 +1,98 @@
import Col from 'react-bootstrap/Col';
import Form from 'react-bootstrap/Form';
import Row from 'react-bootstrap/Row';
import Slider from 'rc-slider';
import type { Source, SourceSummary } from '../api/client';
const ALL_SOURCES: Source[] = ['github', 'gitea', 'hg', 'bugzilla'];
interface Props {
enabledSources: Record<Source, boolean>;
onSourceToggle: (s: Source, on: boolean) => void;
rangeMin: number;
rangeMax: number;
rangeValue: [number, number];
onRangeChange: (v: [number, number]) => void;
limit: number;
onLimitChange: (n: number) => void;
summaries: SourceSummary[] | undefined;
}
export function Filters({
enabledSources,
onSourceToggle,
rangeMin,
rangeMax,
rangeValue,
onRangeChange,
limit,
onLimitChange,
summaries,
}: Props) {
const summaryFor = (src: Source) => summaries?.find((s) => s.source === src);
return (
<>
<Row className="mb-3">
<Col md={6}>
{ALL_SOURCES.map((src) => {
const sum = summaryFor(src);
const label = sum ? `${src} (${sum.count})` : src;
return (
<Form.Check
key={src}
type="switch"
id={`source-${src}`}
label={label}
checked={enabledSources[src]}
disabled={!sum || sum.count === 0}
onChange={(e) => onSourceToggle(src, e.target.checked)}
/>
);
})}
</Col>
<Col md={6}>
<label style={{ fontSize: '70%' }}>
number of activities to display: {limit}
</label>
<Slider
value={limit}
min={10}
max={1000}
onChange={(v) => onLimitChange(Array.isArray(v) ? v[0] : v)}
/>
</Col>
</Row>
<Row className="mb-3">
<Col>
<Slider
range
allowCross={false}
value={rangeValue}
min={rangeMin}
max={rangeMax}
onChange={(v) => {
if (Array.isArray(v) && v.length === 2) {
onRangeChange([v[0], v[1]]);
}
}}
/>
<p className="text-center" style={{ fontSize: '85%' }}>
<em>{formatDate(rangeValue[0])}</em> to{' '}
<em>{formatDate(rangeValue[1])}</em>
</p>
</Col>
</Row>
</>
);
}
function formatDate(ts: number): string {
return new Date(ts)
.toLocaleDateString('en-GB', {
year: 'numeric',
month: 'long',
day: 'numeric',
})
.toLowerCase();
}

35
ui/src/components/LanguageBar.tsx Normal file
View File

@@ -0,0 +1,35 @@
export function LanguageBar({ languages, colorMap, compact }: {
languages: Record<string, number>;
colorMap: Record<string, string>;
compact?: boolean;
}) {
const total = Object.values(languages).reduce((a, b) => a + b, 0);
if (total === 0) return null;
const sorted = Object.entries(languages).sort(([, a], [, b]) => b - a);
return (
<div className={compact ? 'mb-1' : 'mt-2'}>
<div className="language-bar">
{sorted.map(([lang, bytes]) => (
<div
key={lang}
className="language-bar-segment"
style={{ width: `${(bytes / total) * 100}%`, backgroundColor: colorMap[lang] ?? '#8b8b8b' }}
title={`${lang} ${((bytes / total) * 100).toFixed(1)}%`}
/>
))}
</div>
{!compact && (
<div className="d-flex flex-wrap gap-2 mt-1" style={{ fontSize: '0.75rem' }}>
{sorted.slice(0, 8).map(([lang, bytes]) => (
<span key={lang}>
<span className="language-dot" style={{ backgroundColor: colorMap[lang] ?? '#8b8b8b' }} />
{lang} {((bytes / total) * 100).toFixed(1)}%
</span>
))}
</div>
)}
</div>
);
}

198
ui/src/components/LanguageStreamGraph.tsx Normal file
View File

@@ -0,0 +1,198 @@
import { useMemo } from 'react';
import { useQuery } from '@tanstack/react-query';
import { fetchLanguageDailyCounts } from '../api/client';
const HEIGHT = 160;
const LABEL_HEIGHT = 16;
/** Language stream graph — stacked area showing language usage over time. */
export function LanguageStreamGraph() {
const to = new Date();
const from = new Date(to);
from.setFullYear(from.getFullYear() - 1);
const fromStr = fmt(from);
const toStr = fmt(to);
const langQ = useQuery({
queryKey: ['language-daily', fromStr, toStr],
queryFn: () => fetchLanguageDailyCounts(fromStr, toStr),
staleTime: 5 * 60_000,
});
const { languages, paths, legendItems } = useMemo(() => {
const raw = langQ.data ?? [];
if (raw.length === 0)
return { weeks: [], languages: [], paths: [], legendItems: [] };
// Aggregate daily counts into weekly buckets
const colorMap = new Map<string, string>();
const weeklyMap = new Map<string, Map<string, number>>();
for (const d of raw) {
if (d.color) colorMap.set(d.language, d.color);
// Bucket to ISO week (Monday-based, keyed by Monday date)
const dt = new Date(d.date + 'T00:00:00Z');
const day = dt.getUTCDay();
const monday = new Date(dt);
monday.setUTCDate(monday.getUTCDate() - ((day + 6) % 7));
const weekKey = monday.toISOString().slice(0, 10);
if (!weeklyMap.has(weekKey)) weeklyMap.set(weekKey, new Map());
const langs = weeklyMap.get(weekKey)!;
langs.set(d.language, (langs.get(d.language) ?? 0) + d.commits);
}
const weeks = [...weeklyMap.keys()].sort();
// Rank languages by total commits to pick top N + "other"
const totals = new Map<string, number>();
for (const langs of weeklyMap.values()) {
for (const [lang, count] of langs) {
totals.set(lang, (totals.get(lang) ?? 0) + count);
}
}
const ranked = [...totals.entries()].sort(([, a], [, b]) => b - a);
const topN = 8;
const topLangs = ranked.slice(0, topN).map(([l]) => l);
const hasOther = ranked.length > topN;
const languages = hasOther ? [...topLangs, 'Other'] : topLangs;
// Build stacked data per week
const stacked: number[][] = weeks.map((wk) => {
const langs = weeklyMap.get(wk)!;
const values = topLangs.map((l) => langs.get(l) ?? 0);
if (hasOther) {
let other = 0;
for (const [l, c] of langs) {
if (!topLangs.includes(l)) other += c;
}
values.push(other);
}
return values;
});
// Compute stream layout (centered baseline)
const maxTotal = Math.max(...stacked.map((row) => row.reduce((a, b) => a + b, 0)), 1);
const chartHeight = HEIGHT - LABEL_HEIGHT;
// For each week, compute y0 (centered) then stack upward
const layerCount = languages.length;
const y0s: number[][] = [];
const y1s: number[][] = [];
for (let w = 0; w < weeks.length; w++) {
const total = stacked[w].reduce((a, b) => a + b, 0);
const scaledTotal = (total / maxTotal) * chartHeight;
let baseline = (chartHeight - scaledTotal) / 2 + LABEL_HEIGHT;
const wy0: number[] = [];
const wy1: number[] = [];
for (let l = 0; l < layerCount; l++) {
const h = (stacked[w][l] / maxTotal) * chartHeight;
wy0.push(baseline);
baseline += h;
wy1.push(baseline);
}
y0s.push(wy0);
y1s.push(wy1);
}
// Build SVG paths for each language layer using smooth curves
const xFor = (w: number) =>
weeks.length > 1 ? (w / (weeks.length - 1)) * 100 : 50;
const paths = languages.map((_, l) => {
if (weeks.length === 0) return '';
const topPts = weeks.map((_, w) => [xFor(w), y0s[w][l]] as [number, number]);
const bottomPts = weeks
.map((_, w) => [xFor(w), y1s[w][l]] as [number, number])
.reverse();
return `M${topPts[0][0]},${topPts[0][1]} ${smoothLine(topPts)} L${bottomPts[0][0]},${bottomPts[0][1]} ${smoothLine(bottomPts)} Z`;
});
// Default colors for "Other" and fallback
const FALLBACK_COLORS = [
'#e34c26', '#563d7c', '#3178c6', '#dea584',
'#f1e05a', '#89e051', '#00ADD8', '#438eff',
];
const legendItems = languages.map((lang, i) => ({
language: lang,
color:
lang === 'Other'
? 'rgba(255,255,255,0.2)'
: colorMap.get(lang) ?? FALLBACK_COLORS[i % FALLBACK_COLORS.length],
total: ranked[i]?.[1] ?? 0,
}));
return { weeks, languages, paths, legendItems };
}, [langQ.data]);
if (langQ.isLoading) return <p style={{ fontSize: '0.8rem' }}>loading language graph...</p>;
if (langQ.isError || languages.length === 0) return null;
return (
<div className="contribution-graph mb-4">
<p style={{ fontSize: '0.8rem', opacity: 0.6 }}>languages by commit activity</p>
<svg viewBox={`0 0 100 ${HEIGHT}`} width="100%" preserveAspectRatio="none" className="d-block" style={{ height: `${HEIGHT}px` }}>
{paths.map((d, i) => (
<path
key={legendItems[i].language}
d={d}
fill={legendItems[i].color}
opacity={0.85}
>
<title>{legendItems[i].language}</title>
</path>
))}
</svg>
<div className="d-flex flex-wrap gap-2 mt-1" style={{ fontSize: '0.75rem' }}>
{legendItems.map(({ language, color }) => (
<span key={language} className="d-flex align-items-center gap-1">
<span
style={{
width: 10,
height: 10,
borderRadius: '50%',
backgroundColor: color,
display: 'inline-block',
}}
/>
{language}
</span>
))}
</div>
</div>
);
}
function fmt(d: Date): string {
return d.toISOString().slice(0, 10);
}
/** Convert a series of points into smooth cubic bezier curve commands.
* Uses Catmull-Rom to Bezier conversion with tension 0.5. */
function smoothLine(pts: [number, number][]): string {
if (pts.length < 2) return '';
if (pts.length === 2)
return `L${pts[1][0]},${pts[1][1]}`;
const commands: string[] = [];
for (let i = 1; i < pts.length; i++) {
const p0 = pts[Math.max(i - 2, 0)];
const p1 = pts[i - 1];
const p2 = pts[i];
const p3 = pts[Math.min(i + 1, pts.length - 1)];
const t = 0.5;
const cp1x = p1[0] + (p2[0] - p0[0]) * t / 3;
const cp1y = p1[1] + (p2[1] - p0[1]) * t / 3;
const cp2x = p2[0] - (p3[0] - p1[0]) * t / 3;
const cp2y = p2[1] - (p3[1] - p1[1]) * t / 3;
commands.push(`C${cp1x},${cp1y} ${cp2x},${cp2y} ${p2[0]},${p2[1]}`);
}
return commands.join(' ');
}

42
ui/src/components/Layout.tsx Normal file
View File

@@ -0,0 +1,42 @@
import { NavLink, Outlet } from 'react-router-dom';
import Container from 'react-bootstrap/Container';
const externalLinks = [
{ url: 'https://linkedin.com/in/thijssen/', label: 'linkedin' },
{ url: 'https://stackoverflow.com/users/68115/grenade', label: 'stackoverflow' },
{ url: 'https://github.com/grenade', label: 'github' },
{ url: 'https://git.lair.cafe/grenade', label: 'gitea' },
];
export function Layout() {
return (
<>
<Container className="py-4">
<header className="site-header d-flex flex-wrap justify-content-between align-items-center mb-4">
<h1 className="mb-0">hi, i'm rob</h1>
<nav className="d-flex flex-wrap gap-3 align-items-center">
<NavLink to="/" end>dash</NavLink>
<NavLink to="/activity">activity</NavLink>
<NavLink to="/cv">cv</NavLink>
<span className="nav-divider">|</span>
{externalLinks.map((el) => (
<a
key={el.url}
href={el.url}
target="_blank"
rel="noopener noreferrer"
>
{el.label}
</a>
))}
</nav>
</header>
<Outlet />
</Container>
<footer className="site-footer">
no cookies are set or read by this site, which is why no consent banner
is shown.
</footer>
</>
);
}

90
ui/src/components/TimelineEntry.tsx Normal file
View File

@@ -0,0 +1,90 @@
import ReactMarkdown from 'react-markdown';
import { VerticalTimelineElement } from 'react-vertical-timeline-component';
import type { TimelineBody, TimelineItem, TitleSegment } from '../api/client';
import { colorFor, iconFor } from '../lib/icon';
interface Props {
item: TimelineItem;
}
export function TimelineEntry({ item }: Props) {
const Icon = iconFor(item.icon);
const date = formatDate(item.occurred_at);
return (
<VerticalTimelineElement
date={date}
iconStyle={{ background: colorFor(item.icon), color: '#fff' }}
icon={<Icon />}
>
<h4 className="vertical-timeline-element-title">
{renderSegments(item.title)}
</h4>
{item.subtitle && (
<h5 className="vertical-timeline-element-subtitle">
{renderSegments(item.subtitle)}
</h5>
)}
{item.body && <Body body={item.body} />}
</VerticalTimelineElement>
);
}
function Body({ body }: { body: TimelineBody }) {
switch (body.kind) {
case 'markdown':
return <ReactMarkdown>{body.text}</ReactMarkdown>;
case 'commits':
return (
<ul style={{ listStyle: 'none', paddingLeft: 0 }}>
{body.commits.map((c) => (
<li key={c.sha}>
<a href={c.url} target="_blank" rel="noopener noreferrer">
<code>{c.short_sha}</code>
</a>{' '}
{c.message}
</li>
))}
</ul>
);
case 'links':
return (
<ul>
{body.items.map((seg, i) => (
<li key={i}>{renderSegment(seg, i)}</li>
))}
</ul>
);
}
}
function renderSegments(segments: TitleSegment[]) {
return segments.map((seg, i) => renderSegment(seg, i));
}
function renderSegment(seg: TitleSegment, i: number) {
if (seg.kind === 'link') {
return (
<a key={i} href={seg.url} target="_blank" rel="noopener noreferrer">
{seg.text}
</a>
);
}
return <span key={i}>{seg.text}</span>;
}
function formatDate(iso: string): string {
const d = new Date(iso);
const date = d
.toLocaleDateString('en-GB', {
weekday: 'long',
year: 'numeric',
month: 'long',
day: 'numeric',
})
.toLowerCase();
const time = d
.toLocaleTimeString('en-GB', { timeZoneName: 'short' })
.toLowerCase();
return `${date} — ${time}`;
}

60
ui/src/components/TopLanguages.tsx Normal file
View File

@@ -0,0 +1,60 @@
import { useMemo } from 'react';
import { useQuery } from '@tanstack/react-query';
import { fetchRepoLanguages } from '../api/client';
const MAX_LANGS = 14;
export function TopLanguages() {
const langsQ = useQuery({
queryKey: ['repo-languages'],
queryFn: fetchRepoLanguages,
staleTime: 10 * 60_000,
});
const ranked = useMemo(() => {
if (!langsQ.data) return [];
const totals = new Map<string, { bytes: number; color: string }>();
for (const e of langsQ.data) {
const cur = totals.get(e.language);
if (cur) {
cur.bytes += e.bytes;
} else {
totals.set(e.language, { bytes: e.bytes, color: e.color ?? '#8b8b8b' });
}
}
return [...totals.entries()]
.sort(([, a], [, b]) => b.bytes - a.bytes)
.slice(0, MAX_LANGS);
}, [langsQ.data]);
if (!langsQ.data || ranked.length === 0) return null;
const maxBytes = ranked[0][1].bytes;
return (
<div>
<p style={{ fontSize: '0.8rem', opacity: 0.6 }}>
top languages by code volume
</p>
<div className="d-flex flex-column gap-1">
{ranked.map(([lang, { bytes, color }]) => (
<div key={lang} className="d-flex align-items-center gap-2" style={{ fontSize: '0.75rem' }}>
<span style={{ width: 70, textAlign: 'right', opacity: 0.8, flexShrink: 0 }}>{lang}</span>
<div style={{ flex: 1, height: 10, borderRadius: 3, background: 'rgba(255,255,255,0.05)' }}>
<div
style={{
width: `${(bytes / maxBytes) * 100}%`,
height: '100%',
borderRadius: 3,
backgroundColor: color,
opacity: 0.85,
}}
/>
</div>
</div>
))}
</div>
</div>
);
}

22
ui/src/components/cv/CvHeader.tsx Normal file
View File

@@ -0,0 +1,22 @@
import { Link } from 'react-router-dom';
import Image from 'react-bootstrap/Image';
import { CV_PHOTO_URL } from '../../api/cv';
export function CvHeader() {
return (
<div className="cv-header d-flex flex-column flex-md-row align-items-md-center gap-3 mb-4">
<Image
src={CV_PHOTO_URL}
alt="rob"
roundedCircle
className="cv-photo"
/>
<div className="flex-grow-1">
<h1 className="mb-1">curriculum vitae</h1>
<Link to="/" className="hot-pink">
← timeline
</Link>
</div>
</div>
);
}

66
ui/src/components/cv/CvSection.tsx Normal file
View File

@@ -0,0 +1,66 @@
import ReactMarkdown from 'react-markdown';
import Card from 'react-bootstrap/Card';
import { type CvSectionConfig, type GistFile } from '../../api/cv';
import { entryAnchorId } from '../../lib/cvDates';
interface Props {
section: CvSectionConfig;
files: GistFile[];
}
// Pipe-delimited fields (e.g. "email | phone | github, linkedin" in the
// contact section) become one paragraph per field, so each lands on its own
// line with a paragraph gap. Within each pipe-segment, comma-separated values
// are stacked with a soft line break (markdown ` \n` -> `<br/>`) so multiple
// emails / phones / urls each get their own line at a tighter spacing.
function splitPipes(content: string): string {
return content
.split('\n')
.map((line) => {
if (!line.includes(' | ')) return line;
return line
.split(' | ')
.map((segment) =>
segment.includes(', ') ? segment.split(', ').join(' \n') : segment,
)
.join('\n\n');
})
.join('\n');
}
// Renders a single section. Each .md file becomes its own block. When
// `show_section_name` is true (e.g. experience, education) the entries are
// wrapped in cards and given anchor ids so the timeline sidebar can deep-link
// to them; otherwise (e.g. summary, contact) they render as flat markdown.
export function CvSection({ section, files }: Props) {
return (
<section id={section.name} className="cv-section mb-4">
{section.show_section_name && <h2 className="cv-section-name">{section.name}</h2>}
{files.map((file) => {
const content = section.show_section_name
? file.content
: splitPipes(file.content);
if (section.show_section_name) {
return (
<div
key={file.filename}
id={entryAnchorId(section.name, file.content)}
className="cv-entry mb-3"
>
<Card className="cv-card">
<Card.Body>
<ReactMarkdown>{content}</ReactMarkdown>
</Card.Body>
</Card>
</div>
);
}
return (
<div key={file.filename} className="cv-entry">
<ReactMarkdown>{content}</ReactMarkdown>
</div>
);
})}
</section>
);
}

98
ui/src/components/cv/CvTimeline.tsx Normal file
View File

@@ -0,0 +1,98 @@
import type { ReactNode } from 'react';
import {
VerticalTimeline,
VerticalTimelineElement,
} from 'react-vertical-timeline-component';
import { type CvData, filesForSection } from '../../api/cv';
import { entryAnchorId, parseEntryHeader } from '../../lib/cvDates';
interface Props {
data: CvData;
}
// Tiny inline parser: turns "[text](url)" segments into <a> elements while
// leaving surrounding text alone. Used so the timeline title renders the
// company name as plain text and the linked website as an external link
// (matching how a markdown parser would render the same source).
function renderInlineLinks(text: string): ReactNode[] {
const parts: ReactNode[] = [];
const re = /\[([^\]]+)\]\(([^)]+)\)/g;
let last = 0;
let m: RegExpExecArray | null;
while ((m = re.exec(text)) !== null) {
if (m.index > last) parts.push(text.slice(last, m.index));
parts.push(
<a key={m.index} href={m[2]} target="_blank" rel="noopener noreferrer">
{m[1]}
</a>,
);
last = m.index + m[0].length;
}
if (last < text.length) parts.push(text.slice(last));
return parts;
}
// Sidebar timeline rendered from every body section that has `show_section_name`
// (i.e. timeline-eligible sections — experience and education in the current
// gist). Each element offers a small "→" link to the matching anchor in the
// body; the title and subtitle preserve any inline markdown links so they
// behave as proper external anchors.
export function CvTimeline({ data }: Props) {
const elements = data.config.sections
.filter((s) => s.placement === 'body' && s.show_section_name)
.flatMap((section) =>
filesForSection(data, section).map((file) => ({ section, file })),
);
if (elements.length === 0) return null;
return (
<div className="cv-timeline">
<h2 className="cv-section-name">timeline</h2>
<VerticalTimeline layout="1-column-left" lineColor="#ecf0f1">
{elements.map(({ section, file }) => {
const parsed = parseEntryHeader(file.content);
const anchor = `#${entryAnchorId(section.name, file.content)}`;
return (
<VerticalTimelineElement
key={file.filename}
date={parsed.interval.replace(/\s*\([^)]*\)/g, '')}
iconStyle={
parsed.iconUrl
? { background: '#ffffff', boxShadow: 'none' }
: { background: '#ff4081', color: '#ffffff' }
}
icon={
parsed.iconUrl ? (
<img
src={parsed.iconUrl}
alt={parsed.title}
style={{
width: '100%',
height: '100%',
objectFit: 'contain',
borderRadius: '50%',
padding: 4,
}}
/>
) : undefined
}
>
<h3 className="vertical-timeline-element-title">
{renderInlineLinks(parsed.titleMd)}
</h3>
{parsed.locationRoleMd && (
<h4 className="vertical-timeline-element-subtitle">
{renderInlineLinks(parsed.locationRoleMd)}
</h4>
)}
<a href={anchor} className="cv-timeline-anchor">
→ details
</a>
</VerticalTimelineElement>
);
})}
</VerticalTimeline>
</div>
);
}

91
ui/src/lib/cvDates.ts Normal file
View File

@@ -0,0 +1,91 @@
// Normalizes the date interval line of a CV entry. The legacy implementation
// at cv/src/App.js:139 chained .replace() calls per month name; this collapses
// that into a single regex pass.
const MONTH_ABBREV: Record<string, string> = {
january: 'jan',
february: 'feb',
march: 'mar',
april: 'apr',
// may stays "may"
june: 'jun',
july: 'jul',
august: 'aug',
september: 'sep',
october: 'oct',
november: 'nov',
december: 'dec',
};
const MONTH_RE = new RegExp(
`\\b(${Object.keys(MONTH_ABBREV).join('|')})\\b`,
'gi',
);
// Strip leading/trailing markdown # / whitespace, abbreviate months. Casing
// is left to the body-level `text-transform: lowercase` so a future toggle can
// flip it from a single place.
export function normalizeInterval(line: string): string {
return line
.replace(/^[\s#]+|[\s#]+$/g, '')
.replace(MONTH_RE, (m) => MONTH_ABBREV[m.toLowerCase()] ?? m);
}
// Strip markdown link syntax (e.g. "[text](url)") down to just the text.
export function stripMdLinks(s: string): string {
return s.replace(/\[([^\]]+)\]\([^)]+\)/g, '$1');
}
// Parse the title / location-role / interval triple out of a CV entry's
// markdown content. If the first line embeds a PNG URL (icon-style entry),
// indices shift down by one.
export interface ParsedHeader {
// Plain text, markdown links stripped — for alt= attributes and similar.
title: string;
locationRole: string;
// Markdown source with leading #s/whitespace stripped — for inline rendering
// so [text](url) links render as proper anchors.
titleMd: string;
locationRoleMd: string;
iconUrl: string | null;
interval: string;
}
export function parseEntryHeader(content: string): ParsedHeader {
const lines = content.split('\n');
const firstLine = lines[0] ?? '';
const hasIcon = firstLine.includes('.png');
const iconUrl = hasIcon
? (firstLine.match(/https:[^ )\]]+\.png/)?.[0] ?? null)
: null;
const titleLine = hasIcon ? (lines[1] ?? '') : firstLine;
const locRoleLine = hasIcon ? (lines[2] ?? '') : (lines[1] ?? '');
const intervalLine = hasIcon ? (lines[3] ?? '') : (lines[2] ?? '');
const titleMd = titleLine.replace(/^[\s#]+|[\s#]+$/g, '');
const locationRoleMd = locRoleLine.replace(/^[\s#]+|[\s#]+$/g, '');
return {
title: stripMdLinks(titleMd),
locationRole: stripMdLinks(locationRoleMd),
titleMd,
locationRoleMd,
iconUrl,
interval: normalizeInterval(intervalLine),
};
}
// Anchor id for an entry: combines the section name and a slug of the title
// line. Mirrors the legacy id format at cv/src/App.js:71.
export function entryAnchorId(sectionName: string, content: string): string {
const lines = content.split('\n');
const firstLine = lines[0] ?? '';
const titleLine = firstLine.includes('.png') ? (lines[1] ?? '') : firstLine;
const slug = stripMdLinks(titleLine.replace(/^[\s#]+|[\s#]+$/g, ''))
.toLowerCase()
.replace(/[^a-z0-9\s-]/g, '')
.replace(/\s+/g, '-')
.replace(/-+/g, '-');
return `${sectionName}-${slug}`;
}

56
ui/src/lib/icon.tsx Normal file
View File

@@ -0,0 +1,56 @@
import {
ArrowLeftRight,
ArrowUpCircle,
ArrowsAngleContract,
Bug,
ChatLeft,
CodeSquare,
DashCircle,
Diagram3,
ExclamationCircle,
PlusCircle,
StarFill,
Tag,
Wrench,
} from 'react-bootstrap-icons';
import type { TimelineIcon } from '../api/client';
const map: Record<TimelineIcon, typeof Wrench> = {
'git-push': ArrowUpCircle,
'git-commit': CodeSquare,
'git-merge': ArrowsAngleContract,
'git-fork': Diagram3,
'git-branch-create': PlusCircle,
'git-branch-delete': DashCircle,
'pull-request': ArrowLeftRight,
issue: ExclamationCircle,
comment: ChatLeft,
star: StarFill,
release: Tag,
bug: Bug,
generic: Wrench,
};
const colors: Record<TimelineIcon, string> = {
'git-push': '#2e7d32',
'git-commit': '#1565c0',
'git-merge': '#6a1b9a',
'git-fork': '#1565c0',
'git-branch-create': '#2e7d32',
'git-branch-delete': '#c62828',
'pull-request': '#1565c0',
issue: '#ef6c00',
comment: '#1565c0',
star: '#f9a825',
release: '#6a1b9a',
bug: '#c62828',
generic: '#546e7a',
};
export function iconFor(name: TimelineIcon) {
return map[name] ?? Wrench;
}
export function colorFor(name: TimelineIcon) {
return colors[name] ?? colors.generic;
}

24
ui/src/main.tsx Normal file
View File

@@ -0,0 +1,24 @@
import { StrictMode } from 'react';
import { createRoot } from 'react-dom/client';
import { BrowserRouter } from 'react-router-dom';
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
import App from './App';
const queryClient = new QueryClient({
defaultOptions: {
queries: {
staleTime: 30_000,
retry: 1,
},
},
});
createRoot(document.getElementById('root')!).render(
<StrictMode>
<QueryClientProvider client={queryClient}>
<BrowserRouter>
<App />
</BrowserRouter>
</QueryClientProvider>
</StrictMode>,
);

98
ui/src/pages/CvPage.css Normal file
View File

@@ -0,0 +1,98 @@
.cv-photo {
width: 96px;
height: 96px;
object-fit: cover;
}
.cv-section-name {
margin-bottom: 0.75rem;
border-bottom: 1px solid rgba(236, 241, 241, 0.2);
padding-bottom: 0.25rem;
}
.cv-card {
background-color: #34495e;
color: #ecf0f1;
border: 1px solid rgba(236, 241, 241, 0.1);
}
.cv-card a {
color: #ff80ab;
}
.cv-card a:hover {
color: #ff4081;
}
.cv-card img {
max-width: 96px;
max-height: 48px;
background-color: #ffffff;
padding: 4px;
border-radius: 4px;
margin-bottom: 0.75rem;
}
.cv-card h3 {
font-size: 1.25rem;
}
.cv-card h4 {
font-size: 1.05rem;
opacity: 0.9;
}
.cv-card h5 {
font-size: 0.95rem;
opacity: 0.75;
font-style: italic;
}
.cv-timeline .vertical-timeline-element-content {
background-color: #34495e;
color: #ecf0f1;
box-shadow: 0 3px 0 #ff4081;
}
.cv-timeline .vertical-timeline-element-content a {
color: #ff80ab;
}
.cv-timeline .vertical-timeline-element-content a:hover {
color: #ff4081;
}
.cv-timeline h3.vertical-timeline-element-title {
font-size: 0.95rem;
margin: 0;
}
.cv-timeline h4.vertical-timeline-element-subtitle {
font-size: 0.8rem;
opacity: 0.85;
margin: 0.15rem 0 0.4rem;
font-weight: normal;
}
.cv-timeline .cv-timeline-anchor {
font-size: 0.75rem;
opacity: 0.85;
float: right;
margin-left: 0.75rem;
}
.cv-timeline .vertical-timeline-element-content::before {
border-right-color: #34495e;
border-left-color: #34495e;
}
.cv-timeline .vertical-timeline-element-date {
color: #ecf0f1 !important;
opacity: 0.8;
}
@media (max-width: 991px) {
.cv-timeline {
margin-top: 2rem;
}
}

107
ui/src/pages/CvPage.tsx Normal file
View File

@@ -0,0 +1,107 @@
import { useEffect } from 'react';
import { useLocation } from 'react-router-dom';
import { useQuery } from '@tanstack/react-query';
import Alert from 'react-bootstrap/Alert';
import Col from 'react-bootstrap/Col';
import Row from 'react-bootstrap/Row';
import Spinner from 'react-bootstrap/Spinner';
import { fetchCv, filesForSection } from '../api/cv';
import { CvHeader } from '../components/cv/CvHeader';
import { CvSection } from '../components/cv/CvSection';
import { CvTimeline } from '../components/cv/CvTimeline';
import './CvPage.css';
export function CvPage() {
const { hash } = useLocation();
const cvQ = useQuery({
queryKey: ['cv-gist'],
queryFn: fetchCv,
staleTime: 5 * 60_000,
});
// Scroll to the anchored entry once the gist resolves and the section
// body has rendered its ids. Re-runs if the user changes the hash while
// already on the page.
useEffect(() => {
if (!cvQ.data || !hash) return;
const target = document.getElementById(hash.slice(1));
if (target) {
target.scrollIntoView({ behavior: 'smooth', block: 'start' });
}
}, [cvQ.data, hash]);
if (cvQ.isLoading) {
return (
<>
<CvHeader />
<div className="d-flex align-items-center gap-2">
<Spinner animation="border" role="status" size="sm" />
<span>loading cv…</span>
</div>
</>
);
}
if (cvQ.isError) {
const msg = (cvQ.error as Error).message;
const rateHint = /403|rate limit/i.test(msg)
? ' (github limits unauthenticated requests to 60/hour per ip — try again shortly)'
: '';
return (
<>
<CvHeader />
<Alert variant="danger">
<Alert.Heading>cv unavailable</Alert.Heading>
<p className="mb-2">
{msg}
{rateHint}
</p>
<button className="btn btn-outline-light" onClick={() => cvQ.refetch()}>
retry
</button>
</Alert>
</>
);
}
const data = cvQ.data!;
const bodySections = data.config.sections.filter((s) => s.placement === 'body');
const navSections = data.config.sections.filter((s) => s.placement === 'nav');
if (bodySections.length === 0 && navSections.length === 0) {
return (
<>
<CvHeader />
<Alert variant="warning">cv unavailable: no sections in config</Alert>
</>
);
}
return (
<>
<CvHeader />
<Row>
<Col lg={9} className="cv-body">
{bodySections.map((section) => (
<CvSection
key={section.name}
section={section}
files={filesForSection(data, section)}
/>
))}
</Col>
<Col lg={3} className="cv-sidebar">
{navSections.map((section) => (
<CvSection
key={section.name}
section={section}
files={filesForSection(data, section)}
/>
))}
<CvTimeline data={data} />
</Col>
</Row>
</>
);
}

141
ui/src/pages/DashPage.tsx Normal file
View File

@@ -0,0 +1,141 @@
import { useMemo } from 'react';
import { useQuery } from '@tanstack/react-query';
import { Link } from 'react-router-dom';
import Col from 'react-bootstrap/Col';
import Row from 'react-bootstrap/Row';
import { fetchProjects, fetchRepoLanguages, type ProjectSummary } from '../api/client';
import { LanguageBar } from '../components/LanguageBar';
import { ContributionGraph, AllTimeGraph } from '../components/ContributionGraph';
import { ContributionStats } from '../components/ContributionStats';
import { LanguageStreamGraph } from '../components/LanguageStreamGraph';
import { TopLanguages } from '../components/TopLanguages';
export function DashPage() {
const projectsQ = useQuery({
queryKey: ['projects'],
queryFn: fetchProjects,
refetchInterval: 60_000,
});
const langsQ = useQuery({
queryKey: ['repo-languages'],
queryFn: fetchRepoLanguages,
staleTime: 10 * 60_000,
});
const langsByRepo = useMemo(() => {
const map = new Map<string, Record<string, number>>();
for (const entry of langsQ.data ?? []) {
const key = `${entry.source}:${entry.repo}`;
if (!map.has(key)) map.set(key, {});
map.get(key)![entry.language] = entry.bytes;
}
return map;
}, [langsQ.data]);
const langColors = useMemo(() => {
const map: Record<string, string> = {};
for (const e of langsQ.data ?? []) {
if (e.color && !map[e.language]) map[e.language] = e.color;
}
return map;
}, [langsQ.data]);
const projects = projectsQ.data ?? [];
const ranked = rankProjects(projects);
return (
<>
<Row className="mb-3">
<Col>
<p>
i rarely say anything that warrants capital letters. a peek into the
projects i'm working on is below.
</p>
</Col>
</Row>
<ContributionGraph />
<LanguageStreamGraph />
<Row xs={1} md={2} lg={3} className="g-3 mb-3">
<Col>
<AllTimeGraph />
</Col>
<Col>
<TopLanguages />
</Col>
<Col>
<ContributionStats />
</Col>
</Row>
{projectsQ.isLoading && <p>loading...</p>}
{projectsQ.isError && (
<p>error: {(projectsQ.error as Error).message}</p>
)}
<Row xs={1} md={2} lg={3} className="g-3">
{ranked.map((p) => (
<Col key={`${p.source}:${p.repo}`}>
<ProjectCard project={p} langs={langsByRepo.get(`${p.source}:${p.repo}`) ?? null} colorMap={langColors} />
</Col>
))}
</Row>
</>
);
}
function ProjectCard({ project: p, langs, colorMap }: { project: ProjectSummary; langs: Record<string, number> | null; colorMap: Record<string, string> }) {
return (
<Link to={`/project/${p.source}/${p.repo}`} className="text-decoration-none">
<div className="project-card p-3">
<h5 className="mb-1"><img src={forgeIcon(p.source)} alt={p.source} className="forge-icon" />{p.repo}</h5>
{langs && <LanguageBar languages={langs} colorMap={colorMap} compact />}
<div className="d-flex gap-3" style={{ fontSize: '0.8rem' }}>
{p.commit_count > 0 && <span>{p.commit_count} commits</span>}
{p.issue_count > 0 && <span>{p.issue_count} issues</span>}
{p.pr_count > 0 && <span>{p.pr_count} prs</span>}
</div>
<div style={{ fontSize: '0.75rem', opacity: 0.6, marginTop: '0.25rem' }}>
{formatRange(p.first_activity, p.last_activity)}
</div>
</div>
</Link>
);
}
function forgeIcon(source: string): string {
switch (source) {
case 'github': return '/github.svg';
case 'gitea': return '/gitea.svg';
case 'hg': return '/mozilla.svg';
default: return '/github.svg';
}
}
function formatRange(first: string | null, last: string | null): string {
const fmt = (iso: string) =>
new Date(iso).toLocaleDateString('en-GB', { month: 'short', year: 'numeric' }).toLowerCase();
if (first && last) return `${fmt(first)} — ${fmt(last)}`;
if (last) return fmt(last);
return '';
}
function rankProjects(projects: ProjectSummary[]): ProjectSummary[] {
if (projects.length === 0) return [];
const now = Date.now();
const maxVolume = Math.max(...projects.map((p) => p.commit_count + p.issue_count + p.pr_count));
const oldest = Math.min(
...projects.map((p) => (p.last_activity ? new Date(p.last_activity).getTime() : 0)),
);
const range = now - oldest || 1;
return [...projects].sort((a, b) => score(b) - score(a));
function score(p: ProjectSummary): number {
if (p.commit_count >= 10000) return -1;
const volume = (p.commit_count + p.issue_count + p.pr_count) / (maxVolume || 1);
const recency = p.last_activity
? (new Date(p.last_activity).getTime() - oldest) / range
: 0;
return 0.6 * recency + 0.4 * volume;
}
}

182
ui/src/pages/ProjectPage.tsx Normal file
View File

@@ -0,0 +1,182 @@
import { useMemo } from 'react';
import { useParams } from 'react-router-dom';
import { useQuery } from '@tanstack/react-query';
import Col from 'react-bootstrap/Col';
import Row from 'react-bootstrap/Row';
import ReactMarkdown from 'react-markdown';
import rehypeRaw from 'rehype-raw';
import rehypeSanitize, { defaultSchema } from 'rehype-sanitize';
import remarkGfm from 'remark-gfm';
import { VerticalTimeline } from 'react-vertical-timeline-component';
import { fetchEvents, fetchReadme, fetchRepoLanguages, fetchProjects, type Source } from '../api/client';
import { LanguageBar } from '../components/LanguageBar';
import { TimelineEntry } from '../components/TimelineEntry';
export function ProjectPage() {
const { source, '*': repoPath } = useParams();
const repo = repoPath ?? '';
const projectsQ = useQuery({
queryKey: ['projects'],
queryFn: fetchProjects,
staleTime: 60_000,
});
const project = projectsQ.data?.find(
(p) => p.source === source && p.repo === repo,
);
const host = project?.host ?? '';
const eventsQ = useQuery({
queryKey: ['project-events', source, repo],
queryFn: () =>
fetchEvents({
sources: source ? [source as Source] : undefined,
repo,
limit: 500,
}),
refetchInterval: 60_000,
});
const readmeQ = useQuery({
queryKey: ['readme', source, host, repo],
queryFn: () => fetchReadme(source as Source, host, repo),
enabled: !!host && (source === 'github' || source === 'gitea'),
staleTime: 5 * 60_000,
});
const repoLangsQ = useQuery({
queryKey: ['repo-languages'],
queryFn: fetchRepoLanguages,
staleTime: 10 * 60_000,
});
const langs = useMemo(() => {
if (!repoLangsQ.data || !source) return null;
const entries = repoLangsQ.data.filter(
(e) => e.source === source && e.repo === repo,
);
if (entries.length === 0) return null;
const result: Record<string, number> = {};
for (const e of entries) result[e.language] = e.bytes;
return result;
}, [repoLangsQ.data, source, repo]);
const langColors = useMemo(() => {
const map: Record<string, string> = {};
for (const e of repoLangsQ.data ?? []) {
if (e.color && !map[e.language]) map[e.language] = e.color;
}
return map;
}, [repoLangsQ.data]);
const events = eventsQ.data ?? [];
return (
<>
<Row className="mb-3">
<Col>
<h2><a href={repoUrl(source ?? '', host, repo)} target="_blank" rel="noopener noreferrer"><img src={forgeIcon(source ?? '')} alt={source} className="forge-icon" style={{ width: 24, height: 24 }} /></a>{repo}</h2>
{langs && <LanguageBar languages={langs} colorMap={langColors} />}
</Col>
</Row>
{readmeQ.data && (
<Row className="mb-4">
<Col>
<div className="project-readme">
<ReactMarkdown
remarkPlugins={[remarkGfm]}
rehypePlugins={[rehypeRaw, [rehypeSanitize, readmeSanitizeSchema]]}
>
{readmeQ.data}
</ReactMarkdown>
</div>
</Col>
</Row>
)}
<Row>
<Col>
<p style={{ fontSize: '85%' }}>
{eventsQ.isLoading
? 'loading...'
: eventsQ.isError
? `error: ${(eventsQ.error as Error).message}`
: `${events.length} ${events.length === 1 ? 'activity' : 'activities'}`}
</p>
<VerticalTimeline>
{events.map((item) => (
<TimelineEntry key={item.id} item={item} />
))}
</VerticalTimeline>
</Col>
</Row>
</>
);
}
function repoUrl(source: string, host: string, repo: string): string {
switch (source) {
case 'github': return `https://github.com/${repo}`;
case 'gitea': return `https://${host}/${repo}`;
case 'hg': return `https://${host}/${repo}`;
default: return '#';
}
}
function forgeIcon(source: string): string {
switch (source) {
case 'github': return '/github.svg';
case 'gitea': return '/gitea.svg';
case 'hg': return '/mozilla.svg';
default: return '/github.svg';
}
}
// rehype-sanitize defaults are conservative — README authors lean on raw
// HTML for layout (centered headers, collapsible sections, image
// dimensions). Extend the schema to permit those tags/attributes while
// still blocking script-y or interactive content (iframe, object, etc.).
const readmeSanitizeSchema = {
...defaultSchema,
tagNames: [
...(defaultSchema.tagNames ?? []),
'details',
'summary',
'picture',
'source',
'kbd',
'sub',
'sup',
'mark',
'abbr',
'cite',
'figure',
'figcaption',
'center',
],
attributes: {
...defaultSchema.attributes,
'*': [
...((defaultSchema.attributes && defaultSchema.attributes['*']) || []),
'align',
'style',
],
a: [
...((defaultSchema.attributes && defaultSchema.attributes.a) || []),
'target',
'rel',
],
img: [
...((defaultSchema.attributes && defaultSchema.attributes.img) || []),
'width',
'height',
'align',
'srcset',
],
source: ['srcset', 'media', 'type'],
details: ['open'],
},
};

132
ui/src/pages/TimelineHome.tsx Normal file
View File

@@ -0,0 +1,132 @@
import { useMemo, useState } from 'react';
import { useParams } from 'react-router-dom';
import { useQuery } from '@tanstack/react-query';
import Col from 'react-bootstrap/Col';
import Row from 'react-bootstrap/Row';
import { VerticalTimeline } from 'react-vertical-timeline-component';
import { fetchDailyCounts, fetchEvents, fetchSources, type Source } from '../api/client';
import { Filters } from '../components/Filters';
import { TimelineEntry } from '../components/TimelineEntry';
const RANGE_MIN = new Date('2010-01-01T00:00:00Z').getTime();
const RANGE_MAX = Date.now();
function parseDate(s: string): number {
// Accept YYYY-MM-DD or full ISO datetime
const t = new Date(s.includes('T') ? s : s + 'T00:00:00Z').getTime();
return isNaN(t) ? NaN : t;
}
function endOfDay(s: string): number {
const t = new Date(s.includes('T') ? s : s + 'T23:59:59Z').getTime();
return isNaN(t) ? NaN : t;
}
function parseTimespan(timespan?: string): [number, number] | null {
if (!timespan) return null;
if (timespan.includes('..')) {
const [a, b] = timespan.split('..');
const from = parseDate(a);
const to = endOfDay(b);
if (!isNaN(from) && !isNaN(to)) return [from, to];
} else {
const from = parseDate(timespan);
const to = endOfDay(timespan);
if (!isNaN(from)) return [from, to];
}
return null;
}
export function TimelineHome() {
const { timespan } = useParams();
const [enabledSources, setEnabledSources] = useState<Record<Source, boolean>>({
github: true,
gitea: true,
hg: true,
bugzilla: true,
});
const [rangeValue, setRangeValue] = useState<[number, number]>(() => {
const parsed = parseTimespan(timespan);
if (parsed) return parsed;
const now = Date.now();
const thirtyDaysAgo = now - 30 * 24 * 60 * 60 * 1000;
return [thirtyDaysAgo, now];
});
const [limit, setLimit] = useState<number>(100);
const sourcesQ = useQuery({
queryKey: ['sources'],
queryFn: fetchSources,
refetchInterval: 60_000,
});
const activeSources = useMemo(
() =>
(Object.keys(enabledSources) as Source[]).filter((s) => enabledSources[s]),
[enabledSources],
);
const eventsQ = useQuery({
queryKey: ['events', rangeValue, activeSources, limit],
queryFn: () =>
fetchEvents({
from: new Date(rangeValue[0]),
to: new Date(rangeValue[1]),
sources: activeSources,
limit,
}),
refetchInterval: 60_000,
});
const events = eventsQ.data ?? [];
const fromStr = new Date(rangeValue[0]).toISOString().slice(0, 10);
const toStr = new Date(rangeValue[1]).toISOString().slice(0, 10);
const dailyQ = useQuery({
queryKey: ['daily-counts', fromStr, toStr],
queryFn: () => fetchDailyCounts(fromStr, toStr),
staleTime: 5 * 60_000,
});
const totalCount = useMemo(
() => (dailyQ.data ?? []).reduce((sum, d) => sum + d.count, 0),
[dailyQ.data],
);
const privateCount = totalCount - events.length;
return (
<>
<Filters
enabledSources={enabledSources}
onSourceToggle={(s, on) =>
setEnabledSources((prev) => ({ ...prev, [s]: on }))
}
rangeMin={RANGE_MIN}
rangeMax={RANGE_MAX}
rangeValue={rangeValue}
onRangeChange={setRangeValue}
limit={limit}
onLimitChange={setLimit}
summaries={sourcesQ.data}
/>
<Row>
<Col>
<p className="text-center" style={{ fontSize: '85%' }}>
{eventsQ.isLoading
? 'loading…'
: eventsQ.isError
? `error: ${(eventsQ.error as Error).message}`
: `showing ${events.length} public ${events.length === 1 ? 'activity' : 'activities'}${privateCount > 0 ? `, ${privateCount} private` : ''}`}
</p>
<VerticalTimeline>
{events.map((item) => (
<TimelineEntry key={item.id} item={item} />
))}
</VerticalTimeline>
</Col>
</Row>
</>
);
}

21
ui/tsconfig.app.json Normal file
View File

@@ -0,0 +1,21 @@
{
"compilerOptions": {
"target": "ES2022",
"lib": ["ES2022", "DOM", "DOM.Iterable"],
"module": "ESNext",
"moduleResolution": "bundler",
"jsx": "react-jsx",
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noFallthroughCasesInSwitch": true,
"skipLibCheck": true,
"isolatedModules": true,
"useDefineForClassFields": true,
"allowImportingTsExtensions": true,
"verbatimModuleSyntax": true,
"noEmit": true,
"resolveJsonModule": true
},
"include": ["src"]
}

7
ui/tsconfig.json Normal file
View File

@@ -0,0 +1,7 @@
{
"files": [],
"references": [
{ "path": "./tsconfig.app.json" },
{ "path": "./tsconfig.node.json" }
]
}

15
ui/tsconfig.node.json Normal file
View File

@@ -0,0 +1,15 @@
{
"compilerOptions": {
"target": "ES2022",
"lib": ["ES2023"],
"module": "ESNext",
"moduleResolution": "bundler",
"strict": true,
"skipLibCheck": true,
"isolatedModules": true,
"noEmit": true,
"allowImportingTsExtensions": true,
"verbatimModuleSyntax": true
},
"include": ["vite.config.ts"]
}

19
ui/vite.config.ts Normal file
View File

@@ -0,0 +1,19 @@
import { defineConfig } from 'vite';
import react from '@vitejs/plugin-react-swc';
// In dev, the UI is served by Vite at :5173 and proxies `/api/*` to the
// moments-api binary at :8080 (default). In prod, nginx serves the static
// build and reverse-proxies the same `/api/*` to the API backend, so the
// frontend's URL shape is identical in both environments.
export default defineConfig({
plugins: [react()],
server: {
proxy: {
'/api': {
target: 'http://localhost:8080',
changeOrigin: true,
rewrite: (path) => path.replace(/^\/api/, ''),
},
},
},
});