rob thijssen
5ce52fff4d
A self-hosted feed where the user owns the ranking: per-source and signed per-interest weights, decayed by recency, via a transparent deterministic scorer. Content is sourced algorithmically (worker RSS/Atom polling) and agentically (per-user API tokens POSTing candidates to the ingest endpoint). Single-user today, multi-user by construction (every row keyed on user_id). Rust cargo workspace + Vite/React/SWC/TS SPA: - newsfeed-entities: DTOs (ts-rs bindings -> web/src/api/bindings) - newsfeed-core: ranking, auth primitives, ingest, data-access ports - newsfeed-data: SQLite adapters (sqlx, runtime queries) - newsfeed-api: axum REST/JSON daemon - newsfeed-worker: RSS polling + rescoring loop - web: responsive, mobile-first SPA (React Query, generated types) Deploy (Gitea Actions, build static musl + SPA, rsync as gitea_ci): api+worker -> slartibartfast, SPA -> oolon (nginx serves + proxies /v1). Deliberate deviations from house conventions (documented in CLAUDE.md/readme): - SQLite instead of Postgres; api+worker co-locate sharing one DB file. - Runtime sqlx queries instead of query! macros (SQLite dynamic typing; keeps CI database-free). Verified end-to-end: auth, token ingest, interest-weighted ranking, signals, pagination (curl + browser); cargo fmt/clippy -D/test and pnpm build/lint pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
141 lines
6.2 KiB
Markdown
141 lines
6.2 KiB
Markdown
# newsfeed
|
|
|
|
A self-hosted, user-controlled news feed. You decide what surfaces and how strongly —
|
|
not an opaque engagement algorithm. It's built to replace the feeds you'd otherwise
|
|
scroll (Google Discover, YouTube subscriptions, socials) with one you own end-to-end.
|
|
|
|
Content arrives two ways:
|
|
|
|
- **Algorithmically** — the worker polls RSS/Atom sources you configure.
|
|
- **Agentically** — external workloads POST candidates to an ingest endpoint using a
|
|
per-user API token.
|
|
|
|
Everything is then ranked by **weights you set**: a baseline weight per source and a
|
|
signed weight per interest (positive to lift, negative to bury), decayed by recency. The
|
|
scoring is a transparent, deterministic function — see `crates/newsfeed-core/src/ranking.rs`.
|
|
|
|
Single-user today, multi-user by construction: every row is owned by a `user_id` and each
|
|
user is wholly in control of, and isolated within, their own feed.
|
|
|
|
## Architecture
|
|
|
|
A Rust cargo workspace (per [architecture conventions](https://git.lair.cafe/grenade/architecture))
|
|
plus a Vite/React frontend:
|
|
|
|
```
|
|
crates/
|
|
newsfeed-entities domain types + DTOs (no I/O); source of the web's TS bindings
|
|
newsfeed-core business logic: ranking, auth primitives, ingest, data-access ports
|
|
newsfeed-data SQLite adapters implementing the core ports (sqlx)
|
|
newsfeed-api axum REST/JSON daemon (bin)
|
|
newsfeed-worker RSS polling + rescoring loop (bin)
|
|
web/ Vite + React + SWC + TS SPA (responsive, mobile-first)
|
|
asset/ deployment artifacts (systemd, firewalld, nginx, config)
|
|
script/ infra-setup.sh (one-time host provisioning)
|
|
.gitea/workflows/ CI-driven deploy
|
|
```
|
|
|
|
**Shared types.** The web app's API types are generated from the Rust `newsfeed-entities`
|
|
crate via `ts-rs` into `web/src/api/bindings/`. Regenerate with `pnpm --dir web gen:types`
|
|
(or `cargo test -p newsfeed-entities`). Don't hand-edit the bindings.
|
|
|
|
### Deliberate deviations from the house conventions
|
|
|
|
- **SQLite, not Postgres** (generic.md §5 defaults to Postgres). Chosen for this app.
|
|
Consequence: the **api and worker co-locate** on one host sharing a single DB file; the
|
|
worker uses in-process scheduling, not the Postgres `FOR UPDATE SKIP LOCKED` pattern.
|
|
- **Runtime sqlx queries, not the `query!` macros** (generic.md §5). SQLite's dynamic
|
|
typing makes compile-time query checking low-value and high-friction; runtime queries
|
|
keep CI database-free. Rationale is documented at the top of `crates/newsfeed-data/src/lib.rs`.
|
|
|
|
## Build
|
|
|
|
Prerequisites: a stable Rust toolchain (see `rust-toolchain.toml`), Node ≥ 20, pnpm.
|
|
|
|
```sh
|
|
# backend
|
|
cargo build --workspace
|
|
cargo test --workspace
|
|
cargo clippy --workspace --all-targets -- -D warnings
|
|
|
|
# frontend
|
|
cd web && pnpm install && pnpm build
|
|
```
|
|
|
|
## Run locally
|
|
|
|
```sh
|
|
# 1. API (creates ./data/newsfeed.db, binds 127.0.0.1:8081)
|
|
NEWSFEED_DATABASE_PATH=./data/newsfeed.db \
|
|
NEWSFEED_BIND=127.0.0.1:8081 \
|
|
NEWSFEED_COOKIE_SECURE=false \
|
|
cargo run -p newsfeed-api -- --config /nonexistent
|
|
|
|
# 2. Worker (same DB file), in another shell
|
|
NEWSFEED_DATABASE_PATH=./data/newsfeed.db \
|
|
cargo run -p newsfeed-worker -- --config /nonexistent
|
|
|
|
# 3. Frontend dev server (proxies /v1 and /health to :8081)
|
|
cd web && pnpm dev
|
|
```
|
|
|
|
Config layers **defaults → TOML file (`--config`) → `NEWSFEED_*` env**. Passing a
|
|
non-existent `--config` path just uses defaults + env, which is convenient for dev.
|
|
|
|
### Try the flow
|
|
|
|
```sh
|
|
# register + login (cookie jar)
|
|
curl -c cj -X POST localhost:8081/v1/auth/register -H content-type:application/json \
|
|
-d '{"username":"me","email":"me@example.com","password":"hunter2hunter2"}'
|
|
curl -c cj -X POST localhost:8081/v1/auth/login -H content-type:application/json \
|
|
-d '{"identifier":"me","password":"hunter2hunter2"}'
|
|
|
|
# weight an interest, mint an ingest token
|
|
curl -b cj -X PUT localhost:8081/v1/interests -H content-type:application/json -d '{"label":"rust","weight":0.9}'
|
|
TOKEN=$(curl -b cj -X POST localhost:8081/v1/tokens -H content-type:application/json -d '{"name":"agent"}' | jq -r .secret)
|
|
|
|
# an agent pushes a candidate; read the ranked feed
|
|
curl -X POST localhost:8081/v1/ingest/candidates -H "authorization: Bearer $TOKEN" \
|
|
-H content-type:application/json -d '{"external_id":"a1","title":"Rust 2.0 released"}'
|
|
curl -b cj localhost:8081/v1/feed
|
|
```
|
|
|
|
## API surface (`/v1`)
|
|
|
|
| Method | Path | Auth | Purpose |
|
|
| --- | --- | --- | --- |
|
|
| POST | `/auth/register` | — | create account |
|
|
| POST | `/auth/login` | — | open session (sets cookie) |
|
|
| POST | `/auth/logout` | cookie | end session |
|
|
| GET | `/auth/me` | cookie | current user |
|
|
| GET | `/feed` | cookie | ranked, keyset-paginated feed |
|
|
| POST | `/feed/signals` | cookie | record view/click/save/dismiss |
|
|
| GET/POST | `/sources` · DELETE `/sources/{id}` | cookie | manage sources |
|
|
| GET/PUT | `/interests` · DELETE `/interests/{id}` | cookie | manage weightings |
|
|
| GET/POST | `/tokens` · DELETE `/tokens/{id}` | cookie | manage ingest tokens |
|
|
| POST | `/ingest/candidates` | bearer token | submit a candidate (idempotent per `external_id`) |
|
|
| GET | `/health` | — | liveness/readiness |
|
|
|
|
## Deploy
|
|
|
|
CI-driven via Gitea Actions (`.gitea/workflows/deploy.yml`), following
|
|
[deployment-gitea-actions.md](https://git.lair.cafe/grenade/architecture). Topology:
|
|
|
|
- **api + worker → `slartibartfast.kosherinata.internal`** (share `/var/lib/newsfeed/newsfeed.db`)
|
|
- **SPA → `oolon.kosherinata.internal`** — nginx serves `/var/www/newsfeed` and
|
|
reverse-proxies `/v1` + `/health` to the API. TLS terminates here; the API speaks plain
|
|
HTTP behind firewalld on the mesh.
|
|
|
|
One-time per host: `./script/infra-setup.sh` (creates the `gitea_ci` deploy user, scoped
|
|
sudoers, the `newsfeed` service account, directories, the `newsfeed.internal` TLS cert +
|
|
renewal, and the nginx vhost). Thereafter every push to `main` builds static musl
|
|
binaries + the SPA bundle and rsyncs them to the targets.
|
|
|
|
Mesh users reach `https://newsfeed.internal`. Public access is at `https://rob.fyi`
|
|
(`asset/nginx/newsfeed.public.conf`) — provision its Let's Encrypt cert, then symlink the
|
|
vhost into `sites-enabled`.
|
|
|
|
For the workspace-wide architectural conventions this project inherits, see the
|
|
[architecture repo](https://git.lair.cafe/grenade/architecture).
|