fix(data): a hidden 30s body timeout capped every proxied blob
`object_store` puts a 30-second `reqwest` timeout on the whole request, body included. On a proxied blob that is not a timeout at all: the bucket is read at the pace the client drains it, so it is a maximum servable file size. The reason it never looked like one is that object_store hides its own consequences. It catches every response-body error and silently retries with a resumed Range, so a large read does not fail at 30s -- it reconnects every 30s and keeps going, until RetryConfig's 180s retry_timeout is spent. Then the body simply ends, with a 200 already on the wire and nothing logged anywhere. The ceiling is 180s x bandwidth: at the 8MB/s a GPU host gets to caveman, no file over ~1.4GB can ever be served, and no client retry converges because the next attempt is no faster. Measured against the live bucket, 500MiB at 1MB/s: cut at 184.1s, 193,340,351 of 524,288,000 bytes. With the timeout disabled, 499.6s and complete -- the same wall time curl takes reading the same object straight from MinIO. Disable it, and guard the stream the way a stream should be guarded: `storage.connect_timeout` and an idle `storage.read_timeout` that bounds silence rather than progress. A test asserts the total cap stays off. Issue #1 named `server.request_timeout` as the cause. It was not -- that key was parsed and then read by nothing, which is its own problem and is why the hunt started 300 seconds away from a 180-second bug. Remove it. Also log when a blob response ends before its Content-Length. Once the headers are out the bytes leave through hyper, so a cut mid-body was invisible on this side: the service logged a clean 200 while the client saw a short read and retried forever. Refs #1 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
This commit is contained in:
@@ -9,11 +9,14 @@ use std::time::Instant;
|
||||
use axum::body::Body as AxumBody;
|
||||
use axum::extract::State;
|
||||
use axum::response::{IntoResponse, Response};
|
||||
use futures::StreamExt;
|
||||
use bytes::Bytes;
|
||||
use futures::stream::BoxStream;
|
||||
use futures::{Stream, StreamExt};
|
||||
use http::{HeaderMap, HeaderValue, StatusCode, header};
|
||||
use rustingface_core::{Body, ResolvedFile, Source};
|
||||
use rustingface_entities::error::{Error, Result};
|
||||
use rustingface_entities::repo::RepoType;
|
||||
use tracing::warn;
|
||||
|
||||
use crate::error::ApiError;
|
||||
use crate::path::{ResolvePath, parse_resolve};
|
||||
@@ -126,9 +129,12 @@ pub async fn get(
|
||||
};
|
||||
telemetry::served(source, len);
|
||||
telemetry::request("resolve_get", status.as_u16());
|
||||
let body = AxumBody::from_stream(
|
||||
stream.map(|chunk| chunk.map_err(|err| std::io::Error::other(err.to_string()))),
|
||||
);
|
||||
let body = AxumBody::from_stream(observed(
|
||||
stream,
|
||||
len,
|
||||
source,
|
||||
format!("{repo_id}/{filename}"),
|
||||
));
|
||||
Ok((status, headers, body).into_response())
|
||||
}
|
||||
Body::None => Err(ApiError(Error::Internal(
|
||||
@@ -137,6 +143,52 @@ pub async fn get(
|
||||
}
|
||||
}
|
||||
|
||||
/// Wrap a body stream so a response that ends before its `Content-Length` says
|
||||
/// so on this side of the wire.
|
||||
///
|
||||
/// Once the headers are out the bytes leave through hyper, and a handler that
|
||||
/// has already returned cannot report anything. Without this, a transfer cut
|
||||
/// mid-body — a storage read that died, a client that vanished — is logged as a
|
||||
/// successful 200 while the client sees a short read and retries forever. That
|
||||
/// asymmetry is what makes such a failure read as "the client is flaky".
|
||||
fn observed(
|
||||
stream: BoxStream<'static, Result<Bytes>>,
|
||||
expected: u64,
|
||||
source: Source,
|
||||
what: String,
|
||||
) -> impl Stream<Item = std::result::Result<Bytes, std::io::Error>> {
|
||||
futures::stream::unfold(
|
||||
(stream, 0u64, what),
|
||||
move |(mut stream, sent, what)| async move {
|
||||
match stream.next().await {
|
||||
Some(Ok(chunk)) => {
|
||||
let sent = sent + chunk.len() as u64;
|
||||
Some((Ok(chunk), (stream, sent, what)))
|
||||
}
|
||||
Some(Err(err)) => {
|
||||
warn!(
|
||||
path = %what, sent, expected, %err,
|
||||
"blob response failed mid-body; the client sees a short read, not an error"
|
||||
);
|
||||
telemetry::truncated(source, sent);
|
||||
let err = std::io::Error::other(err.to_string());
|
||||
Some((Err(err), (stream, sent, what)))
|
||||
}
|
||||
None => {
|
||||
if sent < expected {
|
||||
warn!(
|
||||
path = %what, sent, expected,
|
||||
"blob response ended early with no error; the client sees a short read"
|
||||
);
|
||||
telemetry::truncated(source, sent);
|
||||
}
|
||||
None
|
||||
}
|
||||
}
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
/// Build the header set both `HEAD` and `GET` must carry.
|
||||
///
|
||||
/// `Content-Encoding` is deliberately absent and must stay that way: the
|
||||
|
||||
@@ -34,6 +34,20 @@ pub fn served(source: Source, bytes: u64) {
|
||||
counter!("rustingface_bytes_served_total", "source" => label).increment(bytes);
|
||||
}
|
||||
|
||||
/// Record a blob response that ended before its `Content-Length`.
|
||||
///
|
||||
/// The bytes leave through hyper rather than through a handler that can return
|
||||
/// a status, so a cut mid-body is otherwise invisible on this side: the client
|
||||
/// retries forever and the service logs a clean 200.
|
||||
pub fn truncated(source: Source, bytes: u64) {
|
||||
let label = match source {
|
||||
Source::Bucket => "bucket",
|
||||
Source::Upstream => "upstream",
|
||||
};
|
||||
counter!("rustingface_blob_truncations_total", "source" => label).increment(1);
|
||||
counter!("rustingface_bytes_truncated_total", "source" => label).increment(bytes);
|
||||
}
|
||||
|
||||
/// Record the outcome of an upstream fetch.
|
||||
pub fn upstream_fetch(result: &'static str, seconds: f64) {
|
||||
counter!("rustingface_upstream_fetches_total", "result" => result).increment(1);
|
||||
|
||||
Reference in New Issue
Block a user