feat(modelwatch): push producer for open-weight model releases
Some checks failed
deploy / build-web (push) Successful in 1m30s
deploy / deploy-web (push) Successful in 4s
deploy / build-api (push) Successful in 6m35s
deploy / deploy-api (push) Failing after 10s

Add newsfeed-modelwatch: a one-shot, systemd-timer-driven producer that watches
the Hugging Face Hub for open-weight releases and pushes filtered candidates to
the ingest endpoint. Answers "how do I automate the HF firehose into my feed" —
the RSS half already works on the pull rail; this is the JSON/API half.

How it works: each run polls the HF models API (a watchlist of orgs +
trendingScore), applies an admission predicate (license allowlist, total-params
cap, safetensors present, not gated, pipeline_tag), and POSTs the survivors to
POST /v1/ingest/candidates under a bearer token, tagged source="huggingface".

Two properties of the existing ingest rail shape the design:
- Ingest is idempotent on (user, external_id). Using the repo id as external_id
  makes the producer STATELESS — no dedup table; re-runs/overlapping timers just
  re-submit and the server drops repeats.
- HF `tags` are copied onto the candidate, so the user's per-interest weights do
  the ranking. The predicate is only an ADMISSION filter (what's worth surfacing
  at all) and stays subordinate to explicit weights, per the house rule.

Layout: newsfeed-modelwatch is a client of the API, not an internal component —
it holds no core/data deps. Pure logic (HF types, predicate, mapping to
CandidateSubmission, param formatting) lives in the lib with unit tests; the bin
does the HTTP polling/posting. CandidateSubmission gains Serialize so the
in-workspace producer can build and post one.

Ops:
- asset/systemd/newsfeed-modelwatch.{service,timer}: oneshot + 3-hourly timer.
  The ingest token is a secret, kept in /etc/newsfeed/modelwatch.env
  (NEWSFEED_TOKEN, mapped to `token` by figment) so a redeploy of the config
  never clobbers it.
- asset/config/modelwatch.toml.tmpl: watchlist + predicate, no secret.
- infra-setup.sh installs the units, creates the env placeholder (never
  overwritten), enables the timer, and prints the token step.
- deploy.yml builds + ships the binary and the (secret-free) config each deploy.

Verified: unit tests (gated union, license precedence, admit accept/reject,
mapping); dry-run against live HF; full loop against a local API — minted token,
submitted a real release, confirmed it landed in the feed with the huggingface
source linked and HF tags carried through. fmt/clippy/test all green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
This commit is contained in:
2026-07-08 14:29:32 +03:00
parent 91fd03a102
commit 3db1f586a8
13 changed files with 837 additions and 13 deletions

View File

@@ -44,9 +44,10 @@ jobs:
- run: cargo test --workspace
- name: build binaries
run: |
cargo build --release -p newsfeed-api -p newsfeed-worker
cargo build --release -p newsfeed-api -p newsfeed-worker -p newsfeed-modelwatch
mkdir -p out
cp target/release/newsfeed-api target/release/newsfeed-worker out/
cp target/release/newsfeed-api target/release/newsfeed-worker \
target/release/newsfeed-modelwatch out/
- uses: actions/upload-artifact@v3
with:
name: binaries
@@ -88,7 +89,7 @@ jobs:
- name: rsync binaries
run: |
chmod +x bin/*
for b in newsfeed-api newsfeed-worker; do
for b in newsfeed-api newsfeed-worker newsfeed-modelwatch; do
rsync -az -e "ssh $SSH_OPTS -i ~/.ssh/id_deploy" \
--rsync-path='sudo rsync' --chown=root:root --chmod=0755 --mkpath \
"bin/$b" gitea_ci@"$API_HOST":/usr/local/bin/"$b"
@@ -102,11 +103,16 @@ jobs:
rsync -az -e "ssh $SSH_OPTS -i ~/.ssh/id_deploy" \
--rsync-path='sudo rsync' --chown=root:newsfeed --chmod=0640 --mkpath \
asset/config/worker.toml.tmpl gitea_ci@"$API_HOST":/etc/newsfeed/worker.toml
# modelwatch config carries no secret (the token lives in modelwatch.env), so it
# is safe to overwrite on every deploy.
rsync -az -e "ssh $SSH_OPTS -i ~/.ssh/id_deploy" \
--rsync-path='sudo rsync' --chown=root:newsfeed --chmod=0640 --mkpath \
asset/config/modelwatch.toml.tmpl gitea_ci@"$API_HOST":/etc/newsfeed/modelwatch.toml
- name: apply system state
run: |
ssh $SSH_OPTS -i ~/.ssh/id_deploy gitea_ci@"$API_HOST" '
sudo restorecon -R /usr/local/bin/newsfeed-api /usr/local/bin/newsfeed-worker /etc/newsfeed /var/lib/newsfeed
sudo restorecon -R /usr/local/bin/newsfeed-api /usr/local/bin/newsfeed-worker /usr/local/bin/newsfeed-modelwatch /etc/newsfeed /var/lib/newsfeed
sudo systemctl daemon-reload
sudo systemctl restart newsfeed-api.service
sudo systemctl restart newsfeed-worker.service