rob thijssen 815bfa7deb fix: reconcile repo visibility instead of trusting the ingest-time flag ...

`events.public` was decided once, when a row was ingested, from whatever
the forge reported at that moment — and every poller is incremental (the
github events feed caps at 90 days, search at its top-1000 window, the
per-repo scanner at a `since` cursor, the gitea feed at page 1 after the
first run). Nothing ever revisited a repo, so flipping one to private
upstream only relabelled whatever activity happened afterwards: its
history kept serving commit messages, issue titles and the repo name
indefinitely. The reverse flip was equally frozen.

The github and gitea sources now run a reconciliation pass before
ingesting. github reuses its existing repo discovery, which already
re-reads `private`/`isPrivate` for everything reachable, and only spends
a request on repos we're still exposing that discovery didn't return;
gitea has no equivalent bulk endpoint, so it asks per repo, once a day
rather than once a tick. Repos already hidden are skipped — they can't
leak, and staying hidden is the safe direction to err in. A 404 counts
as private (with the user's own token, a repo still in reach answers 200
even when private, so 404 means gone or transferred), while rate limits
and transient errors flip nothing; the pass runs in both directions, so
a spurious hide is undone by the next successful poll.

That needs a repo key the worker can UPDATE against, hence `events.repo`
— a stored generated column, and now the single definition of the
payload -> repo mapping that list_events, list_projects,
activity_summary and language_daily_counts each carried their own copy
of. Consolidating them fixes an attribution gap along the way:
/search/issues items carry neither `repo.name` nor
`repository.full_name`, only `repository_url`, so every issue and PR
backfilled through search resolved to NULL in all four queries. Those
events now attach to their repo, which both makes them reconcilable and
means they show up in /projects and /activity/summary.

Also closes a leak that predates the flip problem: `/v1/languages/repos`
had no visibility gate at all, and repo_languages is populated for every
repo the worker discovers, private ones included. Repo names were on the
wire (and baked into the prerendered HTML via the dehydrated query
cache) regardless of what `events.public` said. Rather than a second
visibility column to keep in sync, the response now derives it — a
repo's languages are exposed exactly when at least one of its events is.

Verified against postgres 16: the generated column extracts every
payload shape the four sources produce (and NULLs a non-github
`repository_url`), the reconciliation UPDATE flips all three github
event shapes for a repo in one statement and is a no-op on re-run, the
gitea host filter excludes other hosts while treating rows predating the
`_host` stamp as local, and /v1/languages/repos drops a repo once its
events go private.

Closes #6

2026-08-15 19:41:33 +03:00

..

src

fix: reconcile repo visibility instead of trusting the ingest-time flag

2026-08-15 19:41:33 +03:00

Cargo.toml

feat(worker): add github events poller

2026-05-03 17:59:15 +03:00