feat(modelwatch): push producer for open-weight model releases
Add newsfeed-modelwatch: a one-shot, systemd-timer-driven producer that watches
the Hugging Face Hub for open-weight releases and pushes filtered candidates to
the ingest endpoint. Answers "how do I automate the HF firehose into my feed" —
the RSS half already works on the pull rail; this is the JSON/API half.
How it works: each run polls the HF models API (a watchlist of orgs +
trendingScore), applies an admission predicate (license allowlist, total-params
cap, safetensors present, not gated, pipeline_tag), and POSTs the survivors to
POST /v1/ingest/candidates under a bearer token, tagged source="huggingface".
Two properties of the existing ingest rail shape the design:
- Ingest is idempotent on (user, external_id). Using the repo id as external_id
makes the producer STATELESS — no dedup table; re-runs/overlapping timers just
re-submit and the server drops repeats.
- HF `tags` are copied onto the candidate, so the user's per-interest weights do
the ranking. The predicate is only an ADMISSION filter (what's worth surfacing
at all) and stays subordinate to explicit weights, per the house rule.
Layout: newsfeed-modelwatch is a client of the API, not an internal component —
it holds no core/data deps. Pure logic (HF types, predicate, mapping to
CandidateSubmission, param formatting) lives in the lib with unit tests; the bin
does the HTTP polling/posting. CandidateSubmission gains Serialize so the
in-workspace producer can build and post one.
Ops:
- asset/systemd/newsfeed-modelwatch.{service,timer}: oneshot + 3-hourly timer.
The ingest token is a secret, kept in /etc/newsfeed/modelwatch.env
(NEWSFEED_TOKEN, mapped to `token` by figment) so a redeploy of the config
never clobbers it.
- asset/config/modelwatch.toml.tmpl: watchlist + predicate, no secret.
- infra-setup.sh installs the units, creates the env placeholder (never
overwritten), enables the timer, and prints the token step.
- deploy.yml builds + ships the binary and the (secret-free) config each deploy.
Verified: unit tests (gated union, license precedence, admit accept/reject,
mapping); dry-run against live HF; full loop against a local API — minted token,
submitted a real release, confirmed it landed in the feed with the huggingface
source linked and HF tags carried through. fmt/clippy/test all green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
This commit is contained in:
16
CLAUDE.md
16
CLAUDE.md
@@ -19,11 +19,19 @@ per-source and per-interest weights the user sets.
|
||||
- `newsfeed-core` — business logic + data-access **ports** (traits). Pure where possible
|
||||
(ranking, auth hashing). No DB or network calls.
|
||||
- `newsfeed-data` — SQLite **adapters** implementing the core ports.
|
||||
- `newsfeed-fetch` — the outbound-feed adapter (RSS fetch, feed discovery, OPML parse);
|
||||
implements the `FeedProbe` port. The one crate that talks HTTP to the wider web for
|
||||
feeds — used by the api (discovery on add) and the worker (polling). Kept separate from
|
||||
`data` (SQLite-only) on purpose.
|
||||
- `newsfeed-api` / `newsfeed-worker` — thin binaries; wire config/logging/signals and the
|
||||
concrete store. No business logic that could live in a library crate.
|
||||
- `newsfeed-modelwatch` — a **push producer** (lib + one-shot bin): watches the Hugging
|
||||
Face Hub for open-weight releases, applies an admission predicate, and POSTs candidates
|
||||
to `/v1/ingest/candidates` under a token. A *client* of the API, not an internal
|
||||
component — it holds no DB/core deps; its pure logic (predicate, mapping) is in the lib.
|
||||
|
||||
New types → entities. New logic → core. New I/O → data. Add a port to
|
||||
`core::ports`, implement it in `data::store`.
|
||||
New types → entities. New logic → core. New feed I/O → `fetch`. New DB I/O → `data`. Add a
|
||||
port to `core::ports`, implement it in the matching adapter crate.
|
||||
|
||||
## Deliberate deviations from `generic.md`
|
||||
|
||||
@@ -55,7 +63,9 @@ Crypto/token logic lives in `core::auth`; the flow orchestration in `core::servi
|
||||
|
||||
## Deploy topology
|
||||
|
||||
api + worker → `slartibartfast.kosherinata.internal`; SPA → `oolon.kosherinata.internal`
|
||||
api + worker (+ the `newsfeed-modelwatch` systemd timer, 3-hourly, whose ingest token lives
|
||||
in `/etc/newsfeed/modelwatch.env` — a secret kept out of the deployable config so redeploys
|
||||
never clobber it) → `slartibartfast.kosherinata.internal`; SPA → `oolon.kosherinata.internal`
|
||||
(nginx serves + proxies `/v1` to the API over the mesh; TLS terminates at oolon). API
|
||||
port 22672 (registered in architecture `port-allocations.md`; deliberately not the crowded
|
||||
`8081`), plain HTTP behind firewalld. See `.gitea/workflows/deploy.yml` (infra truth)
|
||||
|
||||
Reference in New Issue
Block a user