newsfeed/readme.md

newsfeed

A self-hosted, user-controlled news feed. You decide what surfaces and how strongly — not an opaque engagement algorithm. It's built to replace the feeds you'd otherwise scroll (Google Discover, YouTube subscriptions, socials) with one you own end-to-end.

Content arrives two ways:

• Algorithmically — the worker polls RSS/Atom sources you configure.
• Agentically — external workloads POST candidates to an ingest endpoint using a per-user API token.

Everything is then ranked by weights you set: a baseline weight per source and a signed weight per interest (positive to lift, negative to bury), decayed by recency. The scoring is a transparent, deterministic function — see crates/newsfeed-core/src/ranking.rs.

Single-user today, multi-user by construction: every row is owned by a user_id and each user is wholly in control of, and isolated within, their own feed.

Architecture

A Rust cargo workspace (per architecture conventions) plus a Vite/React frontend:

crates/
  newsfeed-entities   domain types + DTOs (no I/O); source of the web's TS bindings
  newsfeed-core       business logic: ranking, auth primitives, ingest, data-access ports
  newsfeed-data       SQLite adapters implementing the core ports (sqlx)
  newsfeed-api        axum REST/JSON daemon  (bin)
  newsfeed-worker     RSS polling + rescoring loop  (bin)
web/                  Vite + React + SWC + TS SPA (responsive, mobile-first)
asset/                deployment artifacts (systemd, firewalld, nginx, config)
script/               infra-setup.sh (one-time host provisioning)
.gitea/workflows/     CI-driven deploy

Shared types. The web app's API types are generated from the Rust newsfeed-entities crate via ts-rs into web/src/api/bindings/. Regenerate with pnpm --dir web gen:types (or cargo test -p newsfeed-entities). Don't hand-edit the bindings.

Deliberate deviations from the house conventions

• SQLite, not Postgres (generic.md §5 defaults to Postgres). Chosen for this app. Consequence: the api and worker co-locate on one host sharing a single DB file; the worker uses in-process scheduling, not the Postgres FOR UPDATE SKIP LOCKED pattern.
• Runtime sqlx queries, not the query! macros (generic.md §5). SQLite's dynamic typing makes compile-time query checking low-value and high-friction; runtime queries keep CI database-free. Rationale is documented at the top of crates/newsfeed-data/src/lib.rs.

Build

Prerequisites: a stable Rust toolchain (see rust-toolchain.toml), Node ≥ 20, pnpm.

# backend
cargo build --workspace
cargo test  --workspace
cargo clippy --workspace --all-targets -- -D warnings

# frontend
cd web && pnpm install && pnpm build

Run locally

# 1. API (creates ./data/newsfeed.db, binds 127.0.0.1:8081)
NEWSFEED_DATABASE_PATH=./data/newsfeed.db \
NEWSFEED_BIND=127.0.0.1:8081 \
NEWSFEED_COOKIE_SECURE=false \
  cargo run -p newsfeed-api -- --config /nonexistent

# 2. Worker (same DB file), in another shell
NEWSFEED_DATABASE_PATH=./data/newsfeed.db \
  cargo run -p newsfeed-worker -- --config /nonexistent

# 3. Frontend dev server (proxies /v1 and /health to :8081)
cd web && pnpm dev

Config layers defaults → TOML file (--config) → NEWSFEED_* env. Passing a non-existent --config path just uses defaults + env, which is convenient for dev.

Try the flow

# register + login (cookie jar)
curl -c cj -X POST localhost:8081/v1/auth/register -H content-type:application/json \
  -d '{"username":"me","email":"me@example.com","password":"hunter2hunter2"}'
curl -c cj -X POST localhost:8081/v1/auth/login -H content-type:application/json \
  -d '{"identifier":"me","password":"hunter2hunter2"}'

# weight an interest, mint an ingest token
curl -b cj -X PUT  localhost:8081/v1/interests -H content-type:application/json -d '{"label":"rust","weight":0.9}'
TOKEN=$(curl -b cj -X POST localhost:8081/v1/tokens -H content-type:application/json -d '{"name":"agent"}' | jq -r .secret)

# an agent pushes a candidate; read the ranked feed
curl -X POST localhost:8081/v1/ingest/candidates -H "authorization: Bearer $TOKEN" \
  -H content-type:application/json -d '{"external_id":"a1","title":"Rust 2.0 released"}'
curl -b cj localhost:8081/v1/feed

API surface (/v1)

Method	Path	Auth	Purpose
POST	/auth/register	—	create account
POST	/auth/login	—	open session (sets cookie)
POST	/auth/logout	cookie	end session
GET	/auth/me	cookie	current user
GET	/feed	cookie	ranked, keyset-paginated feed
POST	/feed/signals	cookie	record view/click/save/dismiss
GET/POST	/sources · DELETE /sources/{id}	cookie	manage sources
GET/PUT	/interests · DELETE /interests/{id}	cookie	manage weightings
GET/POST	/tokens · DELETE /tokens/{id}	cookie	manage ingest tokens
POST	/ingest/candidates	bearer token	submit a candidate (idempotent per external_id)
GET	/health	—	liveness/readiness

Deploy

CI-driven via Gitea Actions (.gitea/workflows/deploy.yml), following deployment-gitea-actions.md. Topology:

• api + worker → slartibartfast.kosherinata.internal (share /var/lib/newsfeed/newsfeed.db)
• SPA → oolon.kosherinata.internal — nginx serves /var/www/newsfeed and reverse-proxies /v1 + /health to the API. TLS terminates here; the API speaks plain HTTP behind firewalld on the mesh.

One-time per host: ./script/infra-setup.sh (creates the gitea_ci deploy user, scoped sudoers, the newsfeed service account, directories, the newsfeed.internal TLS cert + renewal, and the nginx vhost). Thereafter every push to main builds static musl binaries + the SPA bundle and rsyncs them to the targets.

Mesh users reach https://newsfeed.internal; public access is at https://rob.fyi. infra-setup.sh provisions both vhosts on oolon — the internal one (internal-CA cert) and the public one (Let's Encrypt via certbot + Cloudflare DNS-01, gated on the cert existing). The public cert needs the Cloudflare API token at /root/.certbot-internal on oolon; you still add the Cloudflare A record (rob.fyi → kosherinata WAN, unproxied) and the OPNsense :443 forward separately.

For the workspace-wide architectural conventions this project inherits, see the architecture repo.