grenade/newsfeed
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sources): feed discovery + OPML import for the RSS rail
All checks were successful
deploy / build-web (push) Successful in 1m31s
Details
deploy / deploy-web (push) Successful in 4s
Details
deploy / build-api (push) Successful in 6m24s
Details
deploy / deploy-api (push) Successful in 10s
Details

Browse Source 
Make adding RSS sources painless. Instead of requiring the exact feed URL,
a user can paste any page — a blog homepage, a YouTube channel, a subreddit,
a Mastodon profile — and the server resolves the concrete feed it advertises.
Plus OPML bulk-import to migrate a reader export or a YouTube subscriptions
list in one shot. No new SourceKind, no migration; this enriches the existing
worker-polled RSS rail.

New crate `newsfeed-fetch` — the one place that does outbound feed HTTP:
- probe(url): normalise (+ Reddit /.rss rewrite) -> fetch -> if it parses as a
  feed, done; else scan the HTML <head> for <link rel=alternate type=rss/atom>,
  resolve the relative href, and validate. Covers YouTube/Mastodon/Substack/
  WordPress/blogs, which all advertise their feed this way.
- parse_opml(): recurses nested outline folders.
- fetch_entries()/entry mapping moved here from the worker so both the API
  (discovery) and worker (polling) share a single HTTP+feed-rs path.

- core: add the FeedProbe port (kept I/O-free; adapter lives in newsfeed-fetch).
- api: AppState carries Arc<dyn FeedProbe>; POST /v1/sources resolves a pasted
  homepage to its feed; add POST /v1/sources/discover (preview) and
  /v1/sources/import (OPML, deduped by URL, per-feed failures collected).
- worker: delegate to newsfeed_fetch::fetch_entries; drop the sourcing/ module.
- web: Sources page gains paste-URL + "Find feed" preview, OPML file import
  with a summary, and a "polled Nm ago" hint per source; new client methods
  and regenerated ts-rs bindings.

Verified end-to-end locally: homepage URL -> discovered feed.xml + title;
create stores the resolved URL; OPML import added 1/skipped 1 dup; worker
polled and both items landed in the feed. fmt/clippy/test + web build/lint green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
This commit is contained in:
rob thijssen
2026-07-08 14:09:21 +03:00
parent ee63939151
commit 91fd03a102
21 changed files with 849 additions and 145 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
Cargo.toml
View File

@@ -14,6 +14,7 @@ authors = ["Rob Thijssen <rob@example>"]
newsfeed-entities = { path = "crates/newsfeed-entities", version = "=0.1.0" }
newsfeed-core = { path = "crates/newsfeed-core", version = "=0.1.0" }
newsfeed-data = { path = "crates/newsfeed-data", version = "=0.1.0" }
newsfeed-fetch = { path = "crates/newsfeed-fetch", version = "=0.1.0" }
# async runtime + web
tokio = { version = "1", features = ["full"] }
@@ -54,9 +55,12 @@ rand = "0.8"
sha2 = "0.10"
base64 = "0.22"
# worker: feed sourcing
# feed sourcing (worker poll + api discovery, via newsfeed-fetch)
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "gzip", "json"] }
feed-rs = "2"
url = "2"
regex = "1"
opml = "1"
# cli niceties
clap = { version = "4", features = ["derive", "env"] }