feat(modelwatch): push producer for open-weight model releases
Add newsfeed-modelwatch: a one-shot, systemd-timer-driven producer that watches
the Hugging Face Hub for open-weight releases and pushes filtered candidates to
the ingest endpoint. Answers "how do I automate the HF firehose into my feed" —
the RSS half already works on the pull rail; this is the JSON/API half.
How it works: each run polls the HF models API (a watchlist of orgs +
trendingScore), applies an admission predicate (license allowlist, total-params
cap, safetensors present, not gated, pipeline_tag), and POSTs the survivors to
POST /v1/ingest/candidates under a bearer token, tagged source="huggingface".
Two properties of the existing ingest rail shape the design:
- Ingest is idempotent on (user, external_id). Using the repo id as external_id
makes the producer STATELESS — no dedup table; re-runs/overlapping timers just
re-submit and the server drops repeats.
- HF `tags` are copied onto the candidate, so the user's per-interest weights do
the ranking. The predicate is only an ADMISSION filter (what's worth surfacing
at all) and stays subordinate to explicit weights, per the house rule.
Layout: newsfeed-modelwatch is a client of the API, not an internal component —
it holds no core/data deps. Pure logic (HF types, predicate, mapping to
CandidateSubmission, param formatting) lives in the lib with unit tests; the bin
does the HTTP polling/posting. CandidateSubmission gains Serialize so the
in-workspace producer can build and post one.
Ops:
- asset/systemd/newsfeed-modelwatch.{service,timer}: oneshot + 3-hourly timer.
The ingest token is a secret, kept in /etc/newsfeed/modelwatch.env
(NEWSFEED_TOKEN, mapped to `token` by figment) so a redeploy of the config
never clobbers it.
- asset/config/modelwatch.toml.tmpl: watchlist + predicate, no secret.
- infra-setup.sh installs the units, creates the env placeholder (never
overwritten), enables the timer, and prints the token step.
- deploy.yml builds + ships the binary and the (secret-free) config each deploy.
Verified: unit tests (gated union, license precedence, admit accept/reject,
mapping); dry-run against live HF; full loop against a local API — minted token,
submitted a real release, confirmed it landed in the feed with the huggingface
source linked and HF tags carried through. fmt/clippy/test all green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
This commit is contained in:
@@ -113,6 +113,14 @@ SYSU
|
||||
install -d -m 0750 -o root -g newsfeed /etc/newsfeed
|
||||
install -d -m 0750 -o newsfeed -g newsfeed /var/lib/newsfeed
|
||||
install -d -m 0755 -o root -g root /usr/local/bin
|
||||
|
||||
# modelwatch ingest token holder — a secret, so it lives outside the deployable
|
||||
# config and is created empty here, never clobbered by a redeploy.
|
||||
if [ ! -f /etc/newsfeed/modelwatch.env ]; then
|
||||
printf '# Ingest token for newsfeed-modelwatch. Mint one in the newsfeed UI\n# (Settings -> API tokens), paste below, then: systemctl start newsfeed-modelwatch\nNEWSFEED_TOKEN=\n' > /etc/newsfeed/modelwatch.env
|
||||
chown root:newsfeed /etc/newsfeed/modelwatch.env
|
||||
chmod 0640 /etc/newsfeed/modelwatch.env
|
||||
fi
|
||||
restorecon -R /etc/newsfeed /var/lib/newsfeed
|
||||
|
||||
# SELinux: label the API port so the daemon may bind it
|
||||
@@ -129,14 +137,19 @@ FWD
|
||||
"
|
||||
|
||||
info "[$host] installing systemd units"
|
||||
for unit in newsfeed-api.service newsfeed-worker.service; do
|
||||
for unit in newsfeed-api.service newsfeed-worker.service \
|
||||
newsfeed-modelwatch.service newsfeed-modelwatch.timer; do
|
||||
local content; content="$(cat "$REPO_ROOT/asset/systemd/$unit")"
|
||||
run_remote "$host" "cat > /etc/systemd/system/$unit <<'UNIT'
|
||||
${content}
|
||||
UNIT
|
||||
systemctl daemon-reload
|
||||
systemctl enable $unit"
|
||||
"
|
||||
done
|
||||
run_remote "$host" "
|
||||
systemctl daemon-reload
|
||||
systemctl enable newsfeed-api.service newsfeed-worker.service
|
||||
# modelwatch is a oneshot driven by its timer — enable the timer, not the service.
|
||||
systemctl enable --now newsfeed-modelwatch.timer"
|
||||
}
|
||||
|
||||
# --- WEB host (oolon): webroot, internal cert, nginx vhost --------------------------
|
||||
@@ -284,6 +297,9 @@ main() {
|
||||
provision_api_host
|
||||
provision_web_host
|
||||
dns_reminders
|
||||
info "modelwatch: mint an ingest token in the newsfeed UI (Settings -> API tokens),"
|
||||
info " put it in ${API_HOST}:/etc/newsfeed/modelwatch.env (NEWSFEED_TOKEN=...), then"
|
||||
info " 'systemctl start newsfeed-modelwatch' to seed immediately (the timer runs 3-hourly)."
|
||||
info "done"
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user