rob thijssen
815bfa7deb
All checks were successful
deploy / Build api + worker (static musl) (push) Successful in 5m24s
deploy / Deploy moments-worker to frootmig (push) Successful in 17s
deploy / Deploy moments-api to nikola (push) Successful in 24s
deploy / Build prerendered web (push) Successful in 4m36s
deploy / Deploy web to oolon (push) Successful in 23s
`events.public` was decided once, when a row was ingested, from whatever the forge reported at that moment — and every poller is incremental (the github events feed caps at 90 days, search at its top-1000 window, the per-repo scanner at a `since` cursor, the gitea feed at page 1 after the first run). Nothing ever revisited a repo, so flipping one to private upstream only relabelled whatever activity happened afterwards: its history kept serving commit messages, issue titles and the repo name indefinitely. The reverse flip was equally frozen. The github and gitea sources now run a reconciliation pass before ingesting. github reuses its existing repo discovery, which already re-reads `private`/`isPrivate` for everything reachable, and only spends a request on repos we're still exposing that discovery didn't return; gitea has no equivalent bulk endpoint, so it asks per repo, once a day rather than once a tick. Repos already hidden are skipped — they can't leak, and staying hidden is the safe direction to err in. A 404 counts as private (with the user's own token, a repo still in reach answers 200 even when private, so 404 means gone or transferred), while rate limits and transient errors flip nothing; the pass runs in both directions, so a spurious hide is undone by the next successful poll. That needs a repo key the worker can UPDATE against, hence `events.repo` — a stored generated column, and now the single definition of the payload -> repo mapping that list_events, list_projects, activity_summary and language_daily_counts each carried their own copy of. Consolidating them fixes an attribution gap along the way: /search/issues items carry neither `repo.name` nor `repository.full_name`, only `repository_url`, so every issue and PR backfilled through search resolved to NULL in all four queries. Those events now attach to their repo, which both makes them reconcilable and means they show up in /projects and /activity/summary. Also closes a leak that predates the flip problem: `/v1/languages/repos` had no visibility gate at all, and repo_languages is populated for every repo the worker discovers, private ones included. Repo names were on the wire (and baked into the prerendered HTML via the dehydrated query cache) regardless of what `events.public` said. Rather than a second visibility column to keep in sync, the response now derives it — a repo's languages are exposed exactly when at least one of its events is. Verified against postgres 16: the generated column extracts every payload shape the four sources produce (and NULLs a non-github `repository_url`), the reconciliation UPDATE flips all three github event shapes for a repo in one statement and is a no-op on re-run, the gitea host filter excludes other hosts while treating rows predating the `_host` stamp as local, and /v1/languages/repos drops a repo once its events go private. Closes #6
54 lines
2.3 KiB
SQL
54 lines
2.3 KiB
SQL
-- Materialise the repo each event belongs to.
|
|
--
|
|
-- Two problems, one column.
|
|
--
|
|
-- 1. Four read queries (list_events, list_projects, activity_summary,
|
|
-- language_daily_counts) each carried their own copy of the payload ->
|
|
-- repo CASE. A payload shape learned in one copy never reached the
|
|
-- others: /search/issues items, which carry neither `repo.name` nor
|
|
-- `repository.full_name`, resolved to NULL in all of them.
|
|
--
|
|
-- 2. `events.public` is stamped once, at ingest, from the visibility the
|
|
-- poller saw at the time — and every poller is incremental, so history
|
|
-- is never re-fetched. A repo flipped public -> private upstream kept
|
|
-- serving its old commit messages and repo name forever. Reconciling
|
|
-- that needs a repo key the worker can UPDATE against.
|
|
--
|
|
-- STORED rather than VIRTUAL: the reconciliation UPDATE and the language
|
|
-- visibility gate both filter on it, so it has to be indexable.
|
|
--
|
|
-- Every branch is IMMUTABLE (jsonb accessors, COALESCE, regex substring),
|
|
-- which is what a generated column requires.
|
|
|
|
ALTER TABLE events
|
|
ADD COLUMN repo TEXT GENERATED ALWAYS AS (
|
|
CASE source
|
|
WHEN 'github' THEN COALESCE(
|
|
-- events API
|
|
payload->'repo'->>'name',
|
|
-- /search/commits
|
|
payload->'repository'->>'full_name',
|
|
-- per-repo commit enumeration (stamped by the poller)
|
|
payload->>'_repo',
|
|
-- /search/issues: only an api URL to go on. The regex
|
|
-- yields NULL rather than a mangled string when the URL
|
|
-- shape is anything else.
|
|
substring(
|
|
payload->>'repository_url'
|
|
from '^https://api\.github\.com/repos/(.+)$'
|
|
)
|
|
)
|
|
WHEN 'gitea' THEN COALESCE(
|
|
payload->'repo'->>'full_name',
|
|
payload->'repo'->>'name'
|
|
)
|
|
WHEN 'hg' THEN payload->>'_repo'
|
|
WHEN 'bugzilla' THEN payload->>'product'
|
|
ELSE NULL
|
|
END
|
|
) STORED;
|
|
|
|
-- Serves the reconciliation UPDATE (source, repo), the exposure rollup it
|
|
-- reads first (source, repo, public), and the EXISTS gate on repo_languages.
|
|
CREATE INDEX events_source_repo_public ON events (source, repo, public);
|