feat(modelwatch): push producer for open-weight model releases
Some checks failed
deploy / build-web (push) Successful in 1m30s
deploy / deploy-web (push) Successful in 4s
deploy / build-api (push) Successful in 6m35s
deploy / deploy-api (push) Failing after 10s

Add newsfeed-modelwatch: a one-shot, systemd-timer-driven producer that watches
the Hugging Face Hub for open-weight releases and pushes filtered candidates to
the ingest endpoint. Answers "how do I automate the HF firehose into my feed" —
the RSS half already works on the pull rail; this is the JSON/API half.

How it works: each run polls the HF models API (a watchlist of orgs +
trendingScore), applies an admission predicate (license allowlist, total-params
cap, safetensors present, not gated, pipeline_tag), and POSTs the survivors to
POST /v1/ingest/candidates under a bearer token, tagged source="huggingface".

Two properties of the existing ingest rail shape the design:
- Ingest is idempotent on (user, external_id). Using the repo id as external_id
  makes the producer STATELESS — no dedup table; re-runs/overlapping timers just
  re-submit and the server drops repeats.
- HF `tags` are copied onto the candidate, so the user's per-interest weights do
  the ranking. The predicate is only an ADMISSION filter (what's worth surfacing
  at all) and stays subordinate to explicit weights, per the house rule.

Layout: newsfeed-modelwatch is a client of the API, not an internal component —
it holds no core/data deps. Pure logic (HF types, predicate, mapping to
CandidateSubmission, param formatting) lives in the lib with unit tests; the bin
does the HTTP polling/posting. CandidateSubmission gains Serialize so the
in-workspace producer can build and post one.

Ops:
- asset/systemd/newsfeed-modelwatch.{service,timer}: oneshot + 3-hourly timer.
  The ingest token is a secret, kept in /etc/newsfeed/modelwatch.env
  (NEWSFEED_TOKEN, mapped to `token` by figment) so a redeploy of the config
  never clobbers it.
- asset/config/modelwatch.toml.tmpl: watchlist + predicate, no secret.
- infra-setup.sh installs the units, creates the env placeholder (never
  overwritten), enables the timer, and prints the token step.
- deploy.yml builds + ships the binary and the (secret-free) config each deploy.

Verified: unit tests (gated union, license precedence, admit accept/reject,
mapping); dry-run against live HF; full loop against a local API — minted token,
submitted a real release, confirmed it landed in the feed with the huggingface
source linked and HF tags carried through. fmt/clippy/test all green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
This commit is contained in:
2026-07-08 14:29:32 +03:00
parent 91fd03a102
commit 3db1f586a8
13 changed files with 837 additions and 13 deletions

View File

@@ -0,0 +1,35 @@
# newsfeed-modelwatch configuration (production). Rendered to /etc/newsfeed/modelwatch.toml
# by the deploy workflow.
#
# The ingest token is NOT here — it is a secret, supplied via NEWSFEED_TOKEN in
# /etc/newsfeed/modelwatch.env (created once by infra-setup.sh). That keeps redeploys of
# this file from ever clobbering the token.
ingest_url = "http://127.0.0.1:22672/v1/ingest/candidates"
source_name = "huggingface"
# HF orgs to watch for new open-weight repos.
authors = [
"tencent", "Qwen", "deepseek-ai", "mistralai", "zai-org", "moonshotai",
"openbmb", "allenai", "nvidia", "ibm-granite", "google", "meta-llama", "microsoft",
]
# The trending list is the best built-in noise filter; a notable release surfaces fast.
include_trending = true
trending_limit = 25
per_author_limit = 15
# Ignore repos older than this so the first run doesn't backfill old "newest" repos.
max_age_days = 30
http_timeout_secs = 20
user_agent = "newsfeed-modelwatch (+https://git.lair.cafe/grenade/newsfeed)"
# Admission filter: what's worth putting in front of you at all. Ranking is NOT decided
# here — the HF tags ride onto each candidate so your per-interest weights do the ranking.
[predicate]
licenses = ["apache-2.0", "mit"]
# Total safetensors params. NB: this is total, not MoE-active (that needs config.json math).
max_params = 40000000000
require_safetensors = true
skip_gated = true
pipeline_tags = ["text-generation"]

View File

@@ -0,0 +1,32 @@
[Unit]
Description=newsfeed model-release watcher (Hugging Face Hub -> ingest)
Documentation=https://git.lair.cafe/grenade/newsfeed
After=network-online.target newsfeed-api.service
Wants=network-online.target
# Timer-activated (see newsfeed-modelwatch.timer); no [Install] here.
[Service]
Type=oneshot
User=newsfeed
Group=newsfeed
# The ingest token (NEWSFEED_TOKEN) lives here, kept out of the deployable config so a
# redeploy of modelwatch.toml never clobbers it.
EnvironmentFile=/etc/newsfeed/modelwatch.env
ExecStart=/usr/local/bin/newsfeed-modelwatch --config /etc/newsfeed/modelwatch.toml
# Hardening (architecture generic.md §8). No DB access — this talks HTTP only, so no
# ReadWritePaths are granted.
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictRealtime=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true
SystemCallArchitectures=native
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

View File

@@ -0,0 +1,13 @@
[Unit]
Description=Run newsfeed-modelwatch periodically (open-weight release sweep)
Documentation=https://git.lair.cafe/grenade/newsfeed
[Timer]
# Every 3 hours, jittered, so releases through the day are picked up without hammering HF.
OnCalendar=*-*-* 00/3:00:00
RandomizedDelaySec=600
# Fire a missed run once the host is back up.
Persistent=true
[Install]
WantedBy=timers.target