grenade/newsfeed
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sources): feed discovery + OPML import for the RSS rail
All checks were successful
deploy / build-web (push) Successful in 1m31s
Details
deploy / deploy-web (push) Successful in 4s
Details
deploy / build-api (push) Successful in 6m24s
Details
deploy / deploy-api (push) Successful in 10s
Details

Browse Source 
Make adding RSS sources painless. Instead of requiring the exact feed URL,
a user can paste any page — a blog homepage, a YouTube channel, a subreddit,
a Mastodon profile — and the server resolves the concrete feed it advertises.
Plus OPML bulk-import to migrate a reader export or a YouTube subscriptions
list in one shot. No new SourceKind, no migration; this enriches the existing
worker-polled RSS rail.

New crate `newsfeed-fetch` — the one place that does outbound feed HTTP:
- probe(url): normalise (+ Reddit /.rss rewrite) -> fetch -> if it parses as a
  feed, done; else scan the HTML <head> for <link rel=alternate type=rss/atom>,
  resolve the relative href, and validate. Covers YouTube/Mastodon/Substack/
  WordPress/blogs, which all advertise their feed this way.
- parse_opml(): recurses nested outline folders.
- fetch_entries()/entry mapping moved here from the worker so both the API
  (discovery) and worker (polling) share a single HTTP+feed-rs path.

- core: add the FeedProbe port (kept I/O-free; adapter lives in newsfeed-fetch).
- api: AppState carries Arc<dyn FeedProbe>; POST /v1/sources resolves a pasted
  homepage to its feed; add POST /v1/sources/discover (preview) and
  /v1/sources/import (OPML, deduped by URL, per-feed failures collected).
- worker: delegate to newsfeed_fetch::fetch_entries; drop the sourcing/ module.
- web: Sources page gains paste-URL + "Find feed" preview, OPML file import
  with a summary, and a "polled Nm ago" hint per source; new client methods
  and regenerated ts-rs bindings.

Verified end-to-end locally: homepage URL -> discovered feed.xml + title;
create stores the resolved URL; OPML import added 1/skipped 1 dup; worker
polled and both items landed in the feed. fmt/clippy/test + web build/lint green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
This commit is contained in:
rob thijssen
2026-07-08 14:09:21 +03:00
parent ee63939151
commit 91fd03a102
21 changed files with 849 additions and 145 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
crates/newsfeed-fetch/Cargo.toml Normal file
View File

@@ -0,0 +1,25 @@
[package]
name = "newsfeed-fetch"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
authors.workspace = true
description = "Outbound feed I/O for newsfeed: RSS/Atom fetch, feed discovery, OPML parse."
[dependencies]
newsfeed-entities.workspace = true
newsfeed-core.workspace = true
reqwest.workspace = true
feed-rs.workspace = true
url.workspace = true
regex.workspace = true
opml.workspace = true
chrono.workspace = true
anyhow.workspace = true
async-trait.workspace = true
[dev-dependencies]
tokio.workspace = true