fix(data): a hidden 30s body timeout capped every proxied blob

`object_store` puts a 30-second `reqwest` timeout on the whole request,
body included. On a proxied blob that is not a timeout at all: the bucket
is read at the pace the client drains it, so it is a maximum servable
file size.

The reason it never looked like one is that object_store hides its own
consequences. It catches every response-body error and silently retries
with a resumed Range, so a large read does not fail at 30s -- it
reconnects every 30s and keeps going, until RetryConfig's 180s
retry_timeout is spent. Then the body simply ends, with a 200 already on
the wire and nothing logged anywhere. The ceiling is 180s x bandwidth: at
the 8MB/s a GPU host gets to caveman, no file over ~1.4GB can ever be
served, and no client retry converges because the next attempt is no
faster. Measured against the live bucket, 500MiB at 1MB/s: cut at 184.1s,
193,340,351 of 524,288,000 bytes. With the timeout disabled, 499.6s and
complete -- the same wall time curl takes reading the same object
straight from MinIO.

Disable it, and guard the stream the way a stream should be guarded:
`storage.connect_timeout` and an idle `storage.read_timeout` that bounds
silence rather than progress. A test asserts the total cap stays off.

Issue #1 named `server.request_timeout` as the cause. It was not -- that
key was parsed and then read by nothing, which is its own problem and is
why the hunt started 300 seconds away from a 180-second bug. Remove it.

Also log when a blob response ends before its Content-Length. Once the
headers are out the bytes leave through hyper, so a cut mid-body was
invisible on this side: the service logged a clean 200 while the client
saw a short read and retried forever.

Refs #1

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
This commit is contained in:
2026-09-02 13:03:34 +03:00
co-authored by Claude Opus 5
parent 64c1e0e3ad
commit cfa53e14bd
9 changed files with 217 additions and 16 deletions
+56 -4
View File
@@ -9,11 +9,14 @@ use std::time::Instant;
use axum::body::Body as AxumBody;
use axum::extract::State;
use axum::response::{IntoResponse, Response};
use futures::StreamExt;
use bytes::Bytes;
use futures::stream::BoxStream;
use futures::{Stream, StreamExt};
use http::{HeaderMap, HeaderValue, StatusCode, header};
use rustingface_core::{Body, ResolvedFile, Source};
use rustingface_entities::error::{Error, Result};
use rustingface_entities::repo::RepoType;
use tracing::warn;
use crate::error::ApiError;
use crate::path::{ResolvePath, parse_resolve};
@@ -126,9 +129,12 @@ pub async fn get(
};
telemetry::served(source, len);
telemetry::request("resolve_get", status.as_u16());
let body = AxumBody::from_stream(
stream.map(|chunk| chunk.map_err(|err| std::io::Error::other(err.to_string()))),
);
let body = AxumBody::from_stream(observed(
stream,
len,
source,
format!("{repo_id}/{filename}"),
));
Ok((status, headers, body).into_response())
}
Body::None => Err(ApiError(Error::Internal(
@@ -137,6 +143,52 @@ pub async fn get(
}
}
/// Wrap a body stream so a response that ends before its `Content-Length` says
/// so on this side of the wire.
///
/// Once the headers are out the bytes leave through hyper, and a handler that
/// has already returned cannot report anything. Without this, a transfer cut
/// mid-body — a storage read that died, a client that vanished — is logged as a
/// successful 200 while the client sees a short read and retries forever. That
/// asymmetry is what makes such a failure read as "the client is flaky".
fn observed(
stream: BoxStream<'static, Result<Bytes>>,
expected: u64,
source: Source,
what: String,
) -> impl Stream<Item = std::result::Result<Bytes, std::io::Error>> {
futures::stream::unfold(
(stream, 0u64, what),
move |(mut stream, sent, what)| async move {
match stream.next().await {
Some(Ok(chunk)) => {
let sent = sent + chunk.len() as u64;
Some((Ok(chunk), (stream, sent, what)))
}
Some(Err(err)) => {
warn!(
path = %what, sent, expected, %err,
"blob response failed mid-body; the client sees a short read, not an error"
);
telemetry::truncated(source, sent);
let err = std::io::Error::other(err.to_string());
Some((Err(err), (stream, sent, what)))
}
None => {
if sent < expected {
warn!(
path = %what, sent, expected,
"blob response ended early with no error; the client sees a short read"
);
telemetry::truncated(source, sent);
}
None
}
}
},
)
}
/// Build the header set both `HEAD` and `GET` must carry.
///
/// `Content-Encoding` is deliberately absent and must stay that way: the
+14
View File
@@ -34,6 +34,20 @@ pub fn served(source: Source, bytes: u64) {
counter!("rustingface_bytes_served_total", "source" => label).increment(bytes);
}
/// Record a blob response that ended before its `Content-Length`.
///
/// The bytes leave through hyper rather than through a handler that can return
/// a status, so a cut mid-body is otherwise invisible on this side: the client
/// retries forever and the service logs a clean 200.
pub fn truncated(source: Source, bytes: u64) {
let label = match source {
Source::Bucket => "bucket",
Source::Upstream => "upstream",
};
counter!("rustingface_blob_truncations_total", "source" => label).increment(1);
counter!("rustingface_bytes_truncated_total", "source" => label).increment(bytes);
}
/// Record the outcome of an upstream fetch.
pub fn upstream_fetch(result: &'static str, seconds: f64) {
counter!("rustingface_upstream_fetches_total", "result" => result).increment(1);