fix(modelwatch,fetch): param-count fallback + browser-shaped feed headers
All checks were successful
deploy / build-web (push) Successful in 1m38s
deploy / deploy-web (push) Successful in 3s
deploy / build-api (push) Successful in 6m51s
deploy / deploy-api (push) Successful in 12s

Two fixes surfaced by the live feed.

modelwatch: some repos report a bogus tiny `safetensors.total` (e.g.
deepreinforce-ai/Ornith-1.0-35B came through as "1M params"), which both
misrendered and — worse — let an oversized model slip past the size cap. Add
`params_from_name` (parse the first <num>B|M token from the repo id, which by
convention is the total size) and `effective_params` (trust safetensors.total
only when it's >= 100M, else fall back to the name). Use it for the size filter
and the summary. New unit tests cover the name parser, the fallback, and that a
70B reporting a 1M total is still rejected as too large. (Forward-looking: the
idempotent ingest won't re-render items already in the feed.)

fetch: feed requests now send browser-shaped `Accept` / `Accept-Language`
headers. reqwest sends no Accept by default, which some anti-bot filters 403.
NB this is hygiene, not a Reddit fix — Reddit fingerprints the TLS/HTTP client
(rustls) and hard-403s it where curl only gets rate-limited (429); headers don't
change the fingerprint. The control path (github releases.atom, etc.) is
unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
This commit is contained in:
2026-07-08 15:25:36 +03:00
parent 6555ed2fed
commit 243b17e9b3
4 changed files with 104 additions and 11 deletions

View File

@@ -31,6 +31,18 @@ const DEFAULT_UA: &str = concat!(
" (feed discovery)"
);
/// Browser-like `Accept` advertising a feed preference. Sending no `Accept` (reqwest's
/// default) trips some anti-bot filters — notably Reddit, which hard-403s a header-less
/// client while only rate-limiting a browser-shaped one.
const FEED_ACCEPT: &str =
"application/rss+xml, application/atom+xml, application/xml;q=0.9, text/html;q=0.8, */*;q=0.7";
/// Attach the browser-shaped headers every feed request should carry.
fn feed_headers(rb: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
rb.header(reqwest::header::ACCEPT, FEED_ACCEPT)
.header(reqwest::header::ACCEPT_LANGUAGE, "en-US,en;q=0.9")
}
// ---------------------------------------------------------------------------
// Worker path: fetch a known feed URL and parse it into backend-agnostic entries.
// ---------------------------------------------------------------------------
@@ -43,8 +55,7 @@ pub async fn fetch_entries(client: &Client, url: &str) -> anyhow::Result<Vec<Fee
}
async fn get_bytes(client: &Client, url: &str) -> anyhow::Result<Vec<u8>> {
let bytes = client
.get(url)
let bytes = feed_headers(client.get(url))
.send()
.await
.with_context(|| format!("fetching {url}"))?
@@ -170,9 +181,7 @@ impl HttpFeedProber {
/// Fetch a URL, returning the final URL (after redirects) and the body bytes.
async fn fetch(&self, url: &str) -> anyhow::Result<(Url, Vec<u8>)> {
let resp = self
.client
.get(url)
let resp = feed_headers(self.client.get(url))
.send()
.await
.with_context(|| format!("fetching {url}"))?