grenade/newsfeed
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sources): feed discovery + OPML import for the RSS rail
All checks were successful
deploy / build-web (push) Successful in 1m31s
Details
deploy / deploy-web (push) Successful in 4s
Details
deploy / build-api (push) Successful in 6m24s
Details
deploy / deploy-api (push) Successful in 10s
Details

Browse Source 
Make adding RSS sources painless. Instead of requiring the exact feed URL,
a user can paste any page — a blog homepage, a YouTube channel, a subreddit,
a Mastodon profile — and the server resolves the concrete feed it advertises.
Plus OPML bulk-import to migrate a reader export or a YouTube subscriptions
list in one shot. No new SourceKind, no migration; this enriches the existing
worker-polled RSS rail.

New crate `newsfeed-fetch` — the one place that does outbound feed HTTP:
- probe(url): normalise (+ Reddit /.rss rewrite) -> fetch -> if it parses as a
  feed, done; else scan the HTML <head> for <link rel=alternate type=rss/atom>,
  resolve the relative href, and validate. Covers YouTube/Mastodon/Substack/
  WordPress/blogs, which all advertise their feed this way.
- parse_opml(): recurses nested outline folders.
- fetch_entries()/entry mapping moved here from the worker so both the API
  (discovery) and worker (polling) share a single HTTP+feed-rs path.

- core: add the FeedProbe port (kept I/O-free; adapter lives in newsfeed-fetch).
- api: AppState carries Arc<dyn FeedProbe>; POST /v1/sources resolves a pasted
  homepage to its feed; add POST /v1/sources/discover (preview) and
  /v1/sources/import (OPML, deduped by URL, per-feed failures collected).
- worker: delegate to newsfeed_fetch::fetch_entries; drop the sourcing/ module.
- web: Sources page gains paste-URL + "Find feed" preview, OPML file import
  with a summary, and a "polled Nm ago" hint per source; new client methods
  and regenerated ts-rs bindings.

Verified end-to-end locally: homepage URL -> discovered feed.xml + title;
create stores the resolved URL; OPML import added 1/skipped 1 dup; worker
polled and both items landed in the feed. fmt/clippy/test + web build/lint green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
This commit is contained in:
rob thijssen
2026-07-08 14:09:21 +03:00
parent ee63939151
commit 91fd03a102
21 changed files with 849 additions and 145 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
crates/newsfeed-core/src/ports.rs
View File

@@ -8,7 +8,7 @@ use uuid::Uuid;
use newsfeed_entities::auth::{ApiTokenInfo, Session};
use newsfeed_entities::interest::{Interest, UpsertInterest};
use newsfeed_entities::item::ContentItem;
use newsfeed_entities::source::{NewSource, Source};
use newsfeed_entities::source::{DiscoveredFeed, NewSource, Source};
use newsfeed_entities::user::User;
use crate::error::CoreResult;
@@ -96,6 +96,16 @@ pub trait SourceStore: Send + Sync {
async fn mark_polled(&self, source_id: Uuid, at: DateTime<Utc>) -> CoreResult<()>;
}
/// Resolves an arbitrary page URL to a concrete feed. Implemented by an outbound-HTTP
/// adapter (`newsfeed-fetch`), which is I/O and therefore lives outside this crate; the
/// port keeps the API handlers backend-agnostic and unit-testable.
#[async_trait]
pub trait FeedProbe: Send + Sync {
/// Resolve `input_url` (a homepage, channel, subreddit, or direct feed URL) to the
/// feed it advertises, verifying the target actually parses as RSS/Atom.
async fn probe(&self, input_url: &str) -> CoreResult<DiscoveredFeed>;
}
/// User interests / weightings.
#[async_trait]
pub trait InterestStore: Send + Sync {