Add newsfeed-modelwatch: a one-shot, systemd-timer-driven producer that watches
the Hugging Face Hub for open-weight releases and pushes filtered candidates to
the ingest endpoint. Answers "how do I automate the HF firehose into my feed" —
the RSS half already works on the pull rail; this is the JSON/API half.
How it works: each run polls the HF models API (a watchlist of orgs +
trendingScore), applies an admission predicate (license allowlist, total-params
cap, safetensors present, not gated, pipeline_tag), and POSTs the survivors to
POST /v1/ingest/candidates under a bearer token, tagged source="huggingface".
Two properties of the existing ingest rail shape the design:
- Ingest is idempotent on (user, external_id). Using the repo id as external_id
makes the producer STATELESS — no dedup table; re-runs/overlapping timers just
re-submit and the server drops repeats.
- HF `tags` are copied onto the candidate, so the user's per-interest weights do
the ranking. The predicate is only an ADMISSION filter (what's worth surfacing
at all) and stays subordinate to explicit weights, per the house rule.
Layout: newsfeed-modelwatch is a client of the API, not an internal component —
it holds no core/data deps. Pure logic (HF types, predicate, mapping to
CandidateSubmission, param formatting) lives in the lib with unit tests; the bin
does the HTTP polling/posting. CandidateSubmission gains Serialize so the
in-workspace producer can build and post one.
Ops:
- asset/systemd/newsfeed-modelwatch.{service,timer}: oneshot + 3-hourly timer.
The ingest token is a secret, kept in /etc/newsfeed/modelwatch.env
(NEWSFEED_TOKEN, mapped to `token` by figment) so a redeploy of the config
never clobbers it.
- asset/config/modelwatch.toml.tmpl: watchlist + predicate, no secret.
- infra-setup.sh installs the units, creates the env placeholder (never
overwritten), enables the timer, and prints the token step.
- deploy.yml builds + ships the binary and the (secret-free) config each deploy.
Verified: unit tests (gated union, license precedence, admit accept/reject,
mapping); dry-run against live HF; full loop against a local API — minted token,
submitted a real release, confirmed it landed in the feed with the huggingface
source linked and HF tags carried through. fmt/clippy/test all green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
Make adding RSS sources painless. Instead of requiring the exact feed URL,
a user can paste any page — a blog homepage, a YouTube channel, a subreddit,
a Mastodon profile — and the server resolves the concrete feed it advertises.
Plus OPML bulk-import to migrate a reader export or a YouTube subscriptions
list in one shot. No new SourceKind, no migration; this enriches the existing
worker-polled RSS rail.
New crate `newsfeed-fetch` — the one place that does outbound feed HTTP:
- probe(url): normalise (+ Reddit /.rss rewrite) -> fetch -> if it parses as a
feed, done; else scan the HTML <head> for <link rel=alternate type=rss/atom>,
resolve the relative href, and validate. Covers YouTube/Mastodon/Substack/
WordPress/blogs, which all advertise their feed this way.
- parse_opml(): recurses nested outline folders.
- fetch_entries()/entry mapping moved here from the worker so both the API
(discovery) and worker (polling) share a single HTTP+feed-rs path.
- core: add the FeedProbe port (kept I/O-free; adapter lives in newsfeed-fetch).
- api: AppState carries Arc<dyn FeedProbe>; POST /v1/sources resolves a pasted
homepage to its feed; add POST /v1/sources/discover (preview) and
/v1/sources/import (OPML, deduped by URL, per-feed failures collected).
- worker: delegate to newsfeed_fetch::fetch_entries; drop the sourcing/ module.
- web: Sources page gains paste-URL + "Find feed" preview, OPML file import
with a summary, and a "polled Nm ago" hint per source; new client methods
and regenerated ts-rs bindings.
Verified end-to-end locally: homepage URL -> discovered feed.xml + title;
create stores the resolved URL; OPML import added 1/skipped 1 dup; worker
polled and both items landed in the feed. fmt/clippy/test + web build/lint green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
