`events.public` was decided once, when a row was ingested, from whatever
the forge reported at that moment — and every poller is incremental (the
github events feed caps at 90 days, search at its top-1000 window, the
per-repo scanner at a `since` cursor, the gitea feed at page 1 after the
first run). Nothing ever revisited a repo, so flipping one to private
upstream only relabelled whatever activity happened afterwards: its
history kept serving commit messages, issue titles and the repo name
indefinitely. The reverse flip was equally frozen.
The github and gitea sources now run a reconciliation pass before
ingesting. github reuses its existing repo discovery, which already
re-reads `private`/`isPrivate` for everything reachable, and only spends
a request on repos we're still exposing that discovery didn't return;
gitea has no equivalent bulk endpoint, so it asks per repo, once a day
rather than once a tick. Repos already hidden are skipped — they can't
leak, and staying hidden is the safe direction to err in. A 404 counts
as private (with the user's own token, a repo still in reach answers 200
even when private, so 404 means gone or transferred), while rate limits
and transient errors flip nothing; the pass runs in both directions, so
a spurious hide is undone by the next successful poll.
That needs a repo key the worker can UPDATE against, hence `events.repo`
— a stored generated column, and now the single definition of the
payload -> repo mapping that list_events, list_projects,
activity_summary and language_daily_counts each carried their own copy
of. Consolidating them fixes an attribution gap along the way:
/search/issues items carry neither `repo.name` nor
`repository.full_name`, only `repository_url`, so every issue and PR
backfilled through search resolved to NULL in all four queries. Those
events now attach to their repo, which both makes them reconcilable and
means they show up in /projects and /activity/summary.
Also closes a leak that predates the flip problem: `/v1/languages/repos`
had no visibility gate at all, and repo_languages is populated for every
repo the worker discovers, private ones included. Repo names were on the
wire (and baked into the prerendered HTML via the dehydrated query
cache) regardless of what `events.public` said. Rather than a second
visibility column to keep in sync, the response now derives it — a
repo's languages are exposed exactly when at least one of its events is.
Verified against postgres 16: the generated column extracts every
payload shape the four sources produce (and NULLs a non-github
`repository_url`), the reconciliation UPDATE flips all three github
event shapes for a repo in one statement and is a no-op on re-run, the
gitea host filter excludes other hosts while treating rows predating the
`_host` stamp as local, and /v1/languages/repos drops a repo once its
events go private.
Closes #6
The nightly refresh.yml and deploy.yml each substituted asset/nginx/
site.conf.tmpl with their own inline python. When bb2f5b1 templated the
listen line as {{WEB_LISTEN}} (moving the vhost behind oolon's stream SNI
router), it added the substitution to the template and deploy.yml but not
to refresh.yml. The daily refresh then rsynced a literal
`listen {{WEB_LISTEN}};` into /etc/nginx/conf.d/rob.tn.conf, `nginx -t`
failed for the whole edge, and — because the file is written into the live
conf.d before it is tested — every vhost's reload (including the step@
cert renewals) stayed frozen. Internal vhosts, cichlid.internal among
them, served certs that had expired days earlier while the renewed certs
sat unused on disk.
- Replace both inline renderers with script/render-site-conf.py, shared by
deploy.yml and refresh.yml so they cannot drift on what they substitute.
- Guard rails: the renderer fails if any {{PLACEHOLDER}} lacks an env value
or survives substitution, so a forgotten/misnamed variable is a red build
on the runner instead of a broken vhost on the edge.
- Add the missing WEB_LISTEN to refresh.yml's env (the immediate drift).
- Rename the template's {{DOCROOT}} to {{WEB_ROOT}} so every placeholder
maps to the env var of the same name.
- Remove script/deploy.sh: the third, unused renderer of the same template
(superseded by the Actions workflows) and a standing source of drift.
- Docs (readme, CLAUDE.md) updated to the Actions-only deploy path.
Known follow-up (needs a sudoers change + infra-setup re-run on oolon, so
out of scope here): the rendered vhost is still rsynced straight into the
live conf.d and only then `nginx -t`'d, so a valid-but-wrong config could
still wedge nginx. Stage-validate-swap with rollback would close that.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QsH1rcWQYtRVhvaftiKm22
Cover all data sources (github events/search/repo, gitea with org
discovery, hg revset queries, bugzilla), frontend routes (dash with
contribution graphs, activity timeline with timespan filtering,
project detail with readme/languages, cv), api endpoints including
forge proxy and og image, environment variables, and deployment.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
deploy.sh:
- never rsync into /; stage to /tmp on the remote and install at final
paths via sudo bash heredoc, closing the parent-dir attribute leak
that broke three hosts in the earlier rsync incident
- shell-quote heredoc args via ${var@Q}
- drop -A -X on the remaining (web) rsyncs
- generic worker.secrets loop reads (env-var → pass path) from manifest;
GITEA_TOKEN now flows through automatically
- in-memory bash substitution for templates (secrets never on argv)
- simplify semanage port labelling: --add 2>/dev/null || --modify (the
old grep pre-check matched only the first listed port)
- restorecon back to short flags (Fedora policycoreutils has no long
forms; --recursive errored at deploy time)
- quieter health probe loop: curl diagnostics only on final failure
manifest as source of truth:
- api.config.bind drives BIND_ADDR, firewalld port, semanage label,
health-probe URL
- web.config.{server_name,root,api_upstream} drives nginx render,
rsync targets, restorecon scope
- nginx config renamed to site.conf.tmpl; firewalld svc to
moments-api.xml.tmpl; both rendered at deploy time
- topology flip: api → nikola, worker → frootmig (anjie freed)
new scripts:
- script/teardown.sh: idempotent component teardown, never rsyncs,
shared-state cleanup gated on absence of remaining env files,
--remove-docroot guard against shallow / system paths
- script/db-perms.sh: rewritten — fixes grep/append role mismatch that
appended duplicates on re-run, adds postgres reload, hits primary +
standby in a single invocation
readme: genericized; deployment topology no longer carries real host
or site names.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
nikola and frootmig are flagging power events and drive warnings on
the iLO interface and need drive replacement. Move both moments
components onto anjie.kosherinata.internal until those hosts are
back in service. Update the nginx upstream and the readme topology
table to match; the postgres pg_ident.conf on magrathea now needs
to map anjie's cert CN to both moments_ro and moments_rw (two lines
for the same cert_cn).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The per-site nginx ingress for rob.tn lives on oolon (the host the
external router forwards 443 traffic to), not on nikola. Adjust the
topology so:
- web (static ui + nginx) → oolon.hanzalova.internal
- api binds 0.0.0.0:42424 on nikola.kosherinata.internal so oolon
can reverse-proxy across the WG mesh
- new firewalld service moments-api opens 42424 in the default zone
on nikola
- oolon labels port 42424 http_port_t so httpd_t may name_connect
outbound to it (httpd_can_network_connect was already set)
- nginx ssl_certificate switched to oolon's host cert; upstream
rewritten to nikola.kosherinata.internal:42424
Plaintext between oolon and nikola for now — the WG mesh provides
the encryption layer and the data is already public. Documented
the deferral so a future move to per-hop mTLS is obvious.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires up the prod deployment per architecture-doc conventions:
- api → nikola.kosherinata.internal, loopback bind 127.0.0.1:42424
(less-common port, registered with SELinux as http_port_t).
- worker → frootmig.kosherinata.internal, no listening port.
- web (static ui/dist + nginx server_name rob.tn) → nikola, with
/api/* reverse-proxied to the loopback API.
- db → existing magrathea cluster via mTLS, hostname-baked DATABASE_URL
rendered into /etc/moments/{api,worker}.env at deploy time.
Cert rotation: step-ca renews host certs every 24h; .path units watch
/etc/pki/tls/misc/<host>.pem and trigger systemctl restart of the
relevant service. Both binaries hold cert state in rustls and read
once at startup, so restart is the right reload semantics.
deploy.sh contract matches the architecture doc: positional env arg,
component list (or `all` / `default`), --dry-run support. Renders
config templates from `pass`, rsyncs over ssh+sudo, runs sysusers /
restorecon / semanage / systemctl / nginx -t idempotently.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
