rob thijssen
3db1f586a8
Add newsfeed-modelwatch: a one-shot, systemd-timer-driven producer that watches
the Hugging Face Hub for open-weight releases and pushes filtered candidates to
the ingest endpoint. Answers "how do I automate the HF firehose into my feed" —
the RSS half already works on the pull rail; this is the JSON/API half.
How it works: each run polls the HF models API (a watchlist of orgs +
trendingScore), applies an admission predicate (license allowlist, total-params
cap, safetensors present, not gated, pipeline_tag), and POSTs the survivors to
POST /v1/ingest/candidates under a bearer token, tagged source="huggingface".
Two properties of the existing ingest rail shape the design:
- Ingest is idempotent on (user, external_id). Using the repo id as external_id
makes the producer STATELESS — no dedup table; re-runs/overlapping timers just
re-submit and the server drops repeats.
- HF `tags` are copied onto the candidate, so the user's per-interest weights do
the ranking. The predicate is only an ADMISSION filter (what's worth surfacing
at all) and stays subordinate to explicit weights, per the house rule.
Layout: newsfeed-modelwatch is a client of the API, not an internal component —
it holds no core/data deps. Pure logic (HF types, predicate, mapping to
CandidateSubmission, param formatting) lives in the lib with unit tests; the bin
does the HTTP polling/posting. CandidateSubmission gains Serialize so the
in-workspace producer can build and post one.
Ops:
- asset/systemd/newsfeed-modelwatch.{service,timer}: oneshot + 3-hourly timer.
The ingest token is a secret, kept in /etc/newsfeed/modelwatch.env
(NEWSFEED_TOKEN, mapped to `token` by figment) so a redeploy of the config
never clobbers it.
- asset/config/modelwatch.toml.tmpl: watchlist + predicate, no secret.
- infra-setup.sh installs the units, creates the env placeholder (never
overwritten), enables the timer, and prints the token step.
- deploy.yml builds + ships the binary and the (secret-free) config each deploy.
Verified: unit tests (gated union, license precedence, admit accept/reject,
mapping); dry-run against live HF; full loop against a local API — minted token,
submitted a real release, confirmed it landed in the feed with the huggingface
source linked and HF tags carried through. fmt/clippy/test all green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
4.6 KiB
CLAUDE.md — newsfeed
Agent-facing notes for working in this repo. Read the workspace-wide conventions in
~/git/architecture/*.md first; this file records only what's specific to newsfeed,
especially the deliberate deviations.
What this is
A self-hosted, user-controlled news feed. The core idea: the user owns the ranking. Never introduce an implicit/opaque signal that overrides the user's explicit weights. Signals (view/click/save/dismiss) may inform future ranking but must remain subordinate to the per-source and per-interest weights the user sets.
Layout & boundaries (strict)
newsfeed-entities— types/DTOs only, no I/O. Wire DTOs derivets_rs::TSwith#[ts(export)];cargo test -p newsfeed-entitiesregeneratesweb/src/api/bindings/(path set by.cargo/config.toml'sTS_RS_EXPORT_DIR). Don't hand-edit bindings.newsfeed-core— business logic + data-access ports (traits). Pure where possible (ranking, auth hashing). No DB or network calls.newsfeed-data— SQLite adapters implementing the core ports.newsfeed-fetch— the outbound-feed adapter (RSS fetch, feed discovery, OPML parse); implements theFeedProbeport. The one crate that talks HTTP to the wider web for feeds — used by the api (discovery on add) and the worker (polling). Kept separate fromdata(SQLite-only) on purpose.newsfeed-api/newsfeed-worker— thin binaries; wire config/logging/signals and the concrete store. No business logic that could live in a library crate.newsfeed-modelwatch— a push producer (lib + one-shot bin): watches the Hugging Face Hub for open-weight releases, applies an admission predicate, and POSTs candidates to/v1/ingest/candidatesunder a token. A client of the API, not an internal component — it holds no DB/core deps; its pure logic (predicate, mapping) is in the lib.
New types → entities. New logic → core. New feed I/O → fetch. New DB I/O → data. Add a
port to core::ports, implement it in the matching adapter crate.
Deliberate deviations from generic.md
- SQLite instead of Postgres (§5). Explicit project choice. Because SQLite is
single-file/single-host, api and worker co-locate and share
/var/lib/newsfeed/newsfeed.db. There is no central DB cluster, no mTLS DB auth, nopg_identmapping. The worker does in-process scheduling — theFOR UPDATE SKIP LOCKEDguidance does not apply. - Runtime sqlx queries, not
query!macros (§5). SQLite's dynamic typing makes compile-time checking low-value and forces a live DB or fiddly offline cache into CI. We usesqlx::query/query_aswithFromRowrow structs (seedata/src/rows.rs) and map to entities explicitly. No.sqlx/cache; CI builds need no database.
Both are documented at their site (data/src/lib.rs header, readme.md) — keep them in
sync if you change the approach.
Datastore conventions
TEXT UUIDs (hyphenated), RFC3339 timestamps, INTEGER booleans, JSON-in-TEXT for small
collections (tags, media). Migrations are immutable once committed (§5): add a new
crates/newsfeed-data/migrations/NNNN_*.sql, never edit a landed one. RETURNING and
ON CONFLICT are used, so the target needs SQLite ≥ 3.35.
Auth model
- Humans: Argon2id password hash + opaque session cookie (
nf_session), hash stored. - Producers: per-user bearer API tokens (
nf_<prefix>_<random>), SHA-256 hashed at rest, shown once. Ingest is attributed to the token's owner. Crypto/token logic lives incore::auth; the flow orchestration incore::service.
Deploy topology
api + worker (+ the newsfeed-modelwatch systemd timer, 3-hourly, whose ingest token lives
in /etc/newsfeed/modelwatch.env — a secret kept out of the deployable config so redeploys
never clobber it) → slartibartfast.kosherinata.internal; SPA → oolon.kosherinata.internal
(nginx serves + proxies /v1 to the API over the mesh; TLS terminates at oolon). API
port 22672 (registered in architecture port-allocations.md; deliberately not the crowded
8081), plain HTTP behind firewalld. See .gitea/workflows/deploy.yml (infra truth)
and script/infra-setup.sh (one-time host prep). Type=exec, not notify, because axum
doesn't sd_notify.
Before you commit
cargo fmt --all·cargo clippy --workspace --all-targets -- -D warnings·cargo test --workspacepnpm --dir web build(tsc typecheck + vite) ·pnpm --dir web lint- If you touched an entity DTO, regenerate bindings and include them in the commit.
- Conventional Commits; scope = crate/area (
feat(api):,fix(core):, …).