feat(sources): feed discovery + OPML import for the RSS rail · 91fd03a102 - newsfeed

feat(sources): feed discovery + OPML import for the RSS rail

All checks were successful

deploy / build-web (push) Successful in 1m31s

Details

deploy / deploy-web (push) Successful in 4s

Details

deploy / build-api (push) Successful in 6m24s

Details

deploy / deploy-api (push) Successful in 10s

Details

Make adding RSS sources painless. Instead of requiring the exact feed URL,
a user can paste any page — a blog homepage, a YouTube channel, a subreddit,
a Mastodon profile — and the server resolves the concrete feed it advertises.
Plus OPML bulk-import to migrate a reader export or a YouTube subscriptions
list in one shot. No new SourceKind, no migration; this enriches the existing
worker-polled RSS rail.

New crate `newsfeed-fetch` — the one place that does outbound feed HTTP:
- probe(url): normalise (+ Reddit /.rss rewrite) -> fetch -> if it parses as a
  feed, done; else scan the HTML <head> for <link rel=alternate type=rss/atom>,
  resolve the relative href, and validate. Covers YouTube/Mastodon/Substack/
  WordPress/blogs, which all advertise their feed this way.
- parse_opml(): recurses nested outline folders.
- fetch_entries()/entry mapping moved here from the worker so both the API
  (discovery) and worker (polling) share a single HTTP+feed-rs path.

- core: add the FeedProbe port (kept I/O-free; adapter lives in newsfeed-fetch).
- api: AppState carries Arc<dyn FeedProbe>; POST /v1/sources resolves a pasted
  homepage to its feed; add POST /v1/sources/discover (preview) and
  /v1/sources/import (OPML, deduped by URL, per-feed failures collected).
- worker: delegate to newsfeed_fetch::fetch_entries; drop the sourcing/ module.
- web: Sources page gains paste-URL + "Find feed" preview, OPML file import
  with a summary, and a "polled Nm ago" hint per source; new client methods
  and regenerated ts-rs bindings.

Verified end-to-end locally: homepage URL -> discovered feed.xml + title;
create stores the resolved URL; OPML import added 1/skipped 1 dup; worker
polled and both items landed in the feed. fmt/clippy/test + web build/lint green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP

This commit is contained in:

rob thijssen

2026-07-08 14:09:21 +03:00

parent ee63939151

commit 91fd03a102

21 changed files with 849 additions and 145 deletions

									
										2

crates/newsfeed-worker/Cargo.toml
									
												View File
												
				@@ -15,10 +15,10 @@ path = "src/main.rs"

				newsfeed-entities.workspace = true

				newsfeed-core.workspace = true

				newsfeed-data.workspace = true

				newsfeed-fetch.workspace = true

				tokio.workspace = true

				reqwest.workspace = true

				feed-rs.workspace = true

				serde.workspace = true

				chrono.workspace = true

2 crates/newsfeed-worker/Cargo.toml Unescape Escape View File

2

crates/newsfeed-worker/Cargo.toml

View File