fix(modelwatch,fetch): param-count fallback + browser-shaped feed headers
Two fixes surfaced by the live feed. modelwatch: some repos report a bogus tiny `safetensors.total` (e.g. deepreinforce-ai/Ornith-1.0-35B came through as "1M params"), which both misrendered and — worse — let an oversized model slip past the size cap. Add `params_from_name` (parse the first <num>B|M token from the repo id, which by convention is the total size) and `effective_params` (trust safetensors.total only when it's >= 100M, else fall back to the name). Use it for the size filter and the summary. New unit tests cover the name parser, the fallback, and that a 70B reporting a 1M total is still rejected as too large. (Forward-looking: the idempotent ingest won't re-render items already in the feed.) fetch: feed requests now send browser-shaped `Accept` / `Accept-Language` headers. reqwest sends no Accept by default, which some anti-bot filters 403. NB this is hygiene, not a Reddit fix — Reddit fingerprints the TLS/HTTP client (rustls) and hard-403s it where curl only gets rate-limited (429); headers don't change the fingerprint. The control path (github releases.atom, etc.) is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
This commit is contained in:
@@ -31,6 +31,18 @@ const DEFAULT_UA: &str = concat!(
|
||||
" (feed discovery)"
|
||||
);
|
||||
|
||||
/// Browser-like `Accept` advertising a feed preference. Sending no `Accept` (reqwest's
|
||||
/// default) trips some anti-bot filters — notably Reddit, which hard-403s a header-less
|
||||
/// client while only rate-limiting a browser-shaped one.
|
||||
const FEED_ACCEPT: &str =
|
||||
"application/rss+xml, application/atom+xml, application/xml;q=0.9, text/html;q=0.8, */*;q=0.7";
|
||||
|
||||
/// Attach the browser-shaped headers every feed request should carry.
|
||||
fn feed_headers(rb: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
|
||||
rb.header(reqwest::header::ACCEPT, FEED_ACCEPT)
|
||||
.header(reqwest::header::ACCEPT_LANGUAGE, "en-US,en;q=0.9")
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Worker path: fetch a known feed URL and parse it into backend-agnostic entries.
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -43,8 +55,7 @@ pub async fn fetch_entries(client: &Client, url: &str) -> anyhow::Result<Vec<Fee
|
||||
}
|
||||
|
||||
async fn get_bytes(client: &Client, url: &str) -> anyhow::Result<Vec<u8>> {
|
||||
let bytes = client
|
||||
.get(url)
|
||||
let bytes = feed_headers(client.get(url))
|
||||
.send()
|
||||
.await
|
||||
.with_context(|| format!("fetching {url}"))?
|
||||
@@ -170,9 +181,7 @@ impl HttpFeedProber {
|
||||
|
||||
/// Fetch a URL, returning the final URL (after redirects) and the body bytes.
|
||||
async fn fetch(&self, url: &str) -> anyhow::Result<(Url, Vec<u8>)> {
|
||||
let resp = self
|
||||
.client
|
||||
.get(url)
|
||||
let resp = feed_headers(self.client.get(url))
|
||||
.send()
|
||||
.await
|
||||
.with_context(|| format!("fetching {url}"))?
|
||||
|
||||
Reference in New Issue
Block a user