A total-duration request_timeout makes large-blob fetches fail deterministically in proxy mode #1

New Issue

grenade · 2026-09-02T08:53:51Z

grenade commented

2026-09-02 08:53:51 +00:00

Summary

With blob_delivery = "proxy", a blob transfer is bounded by
request_timeout (300s here). Since a streaming download's duration is
size / bandwidth, this turns a total-duration cap into a maximum
downloadable file size — and above that size, no number of retries can
ever succeed.

Fetching Qwen/Qwen3.8-Flash-Next (360 GB, 131 shards of ~3.3 GB) from
rf.internal stalls permanently at 52 of 144 blobs.

Evidence

Client is hf download on a fleet host, HF_ENDPOINT=https://rf.internal.

Measured bandwidth from that host, on a blob rustingface has already
stored (so no upstream pull-through in the path):

rf.internal      32 MB ranged GET    7,966,383 B/s   ttfb 0.29s
huggingface.co   32 MB ranged GET   14,108,269 B/s   ttfb 0.72s
beast -> bob     /healthz                            ttfb 0.003s

The mesh link is fine; the ~8 MB/s is the S3 backend
(caveman.kosherinata.internal:9000, a different site) streaming
through the service.

The partial files match request_timeout x per-connection bandwidth
almost exactly. With two concurrent workers sharing ~8 MB/s, each gets
~4–4.8 MB/s, and 300s at that rate is 1.2–1.4 GB. The three
.incomplete files were:

1,426,063,360
1,163,919,360
1,436,549,120

That is the cap cutting the transfer mid-stream, not a stall — and it
implies even --max-workers 1 fails: 300s at the full 8 MB/s is
~2.4 GB, still short of a 3.3 GB shard.

Across 20 attempts: 21 httpx.ReadTimeout, 2
httpx.RemoteProtocolError, and the .incomplete files stopped
growing entirely (mtimes went stale while attempts continued), so the
client is not resuming them across attempts either. The retry loop
cannot converge.

Why I think the cap is the wrong control

client_stall_timeout = 60s already handles the case a total-duration
timeout is usually reaching for — a peer that has gone away, or a
transfer making no progress. It is an idle/rate guard, which is the
right shape for a stream.

request_timeout on top of it adds a second, different rule: this
transfer must finish within 300s regardless of how healthy it is. For a
model registry, that makes success a function of file size × bandwidth —
so the same client, same file, succeeds on a fast link and fails forever
on a slow one, with no diagnostic difference between them.

Also: the failure is invisible server-side

While the client was accumulating 21 ReadTimeouts, journalctl -u rustingface for the same window logged no error, warning, 502 or 504 —
only a successful stored blob and recorded manifest entry. Whatever
cuts the transfer does not say so. Operationally this reads as "the
client is flaky", which is where I started.

Suggestions, in the order I would take them

Exempt blob streaming from request_timeout, leaving
client_stall_timeout as the guard. A transfer that is still moving
bytes is not a hung request.
Default blob_delivery to redirect for blobs over some size,
or generally. The 302-to-presigned-URL path takes the service out of
the byte path entirely, which is what a 360 GB repo wants, and it is
already implemented — it is just not the default. (Needs clients to
reach the S3 endpoint directly, which is a deployment question, not a
code one.)
Log when a transfer is terminated by a timeout, with the blob
path, bytes sent and which timeout fired. Right now the server is
silent about its own cut.

Workaround for anyone hitting this

Ranged GETs sized to stay under the cap work fine — the 32 MB probes
above are from the same host, same file, same session. So a client that
chunks below request_timeout x bandwidth completes; hf download does
not chunk, so it cannot.

## Summary With `blob_delivery = "proxy"`, a blob transfer is bounded by `request_timeout` (300s here). Since a streaming download's duration is `size / bandwidth`, this turns a **total-duration cap into a maximum downloadable file size** — and above that size, no number of retries can ever succeed. Fetching `Qwen/Qwen3.8-Flash-Next` (360 GB, 131 shards of ~3.3 GB) from `rf.internal` stalls permanently at 52 of 144 blobs. ## Evidence Client is `hf download` on a fleet host, `HF_ENDPOINT=https://rf.internal`. Measured bandwidth from that host, on a blob rustingface has **already stored** (so no upstream pull-through in the path): ``` rf.internal 32 MB ranged GET 7,966,383 B/s ttfb 0.29s huggingface.co 32 MB ranged GET 14,108,269 B/s ttfb 0.72s beast -> bob /healthz ttfb 0.003s ``` The mesh link is fine; the ~8 MB/s is the S3 backend (`caveman.kosherinata.internal:9000`, a different site) streaming through the service. **The partial files match `request_timeout x per-connection bandwidth` almost exactly.** With two concurrent workers sharing ~8 MB/s, each gets ~4–4.8 MB/s, and 300s at that rate is 1.2–1.4 GB. The three `.incomplete` files were: ``` 1,426,063,360 1,163,919,360 1,436,549,120 ``` That is the cap cutting the transfer mid-stream, not a stall — and it implies **even `--max-workers 1` fails**: 300s at the full 8 MB/s is ~2.4 GB, still short of a 3.3 GB shard. Across 20 attempts: 21 `httpx.ReadTimeout`, 2 `httpx.RemoteProtocolError`, and the `.incomplete` files stopped growing entirely (mtimes went stale while attempts continued), so the client is not resuming them across attempts either. The retry loop cannot converge. ## Why I think the cap is the wrong control `client_stall_timeout = 60s` already handles the case a total-duration timeout is usually reaching for — a peer that has gone away, or a transfer making no progress. It is an idle/rate guard, which is the right shape for a stream. `request_timeout` on top of it adds a second, different rule: *this transfer must finish within 300s regardless of how healthy it is*. For a model registry, that makes success a function of file size × bandwidth — so the same client, same file, succeeds on a fast link and fails forever on a slow one, with no diagnostic difference between them. ## Also: the failure is invisible server-side While the client was accumulating 21 `ReadTimeout`s, `journalctl -u rustingface` for the same window logged no error, warning, 502 or 504 — only a successful `stored blob and recorded manifest entry`. Whatever cuts the transfer does not say so. Operationally this reads as "the client is flaky", which is where I started. ## Suggestions, in the order I would take them 1. **Exempt blob streaming from `request_timeout`**, leaving `client_stall_timeout` as the guard. A transfer that is still moving bytes is not a hung request. 2. **Default `blob_delivery` to `redirect` for blobs over some size**, or generally. The 302-to-presigned-URL path takes the service out of the byte path entirely, which is what a 360 GB repo wants, and it is already implemented — it is just not the default. (Needs clients to reach the S3 endpoint directly, which is a deployment question, not a code one.) 3. **Log when a transfer is terminated by a timeout**, with the blob path, bytes sent and which timeout fired. Right now the server is silent about its own cut. ## Workaround for anyone hitting this Ranged GETs sized to stay under the cap work fine — the 32 MB probes above are from the same host, same file, same session. So a client that chunks below `request_timeout x bandwidth` completes; `hf download` does not chunk, so it cannot.

grenade referenced this issue from a commit

2026-09-02 10:08:47 +00:00

fix(data): a hidden 30s body timeout capped every proxied blob

grenade referenced this issue from a commit

2026-09-02 10:08:47 +00:00

fix(core): a ranged fetch of a cold file left the client in silence

grenade referenced this issue from a commit

2026-09-02 10:08:47 +00:00

Merge branch 'fix/no-total-duration-cap-on-blob-transfers'

grenade closed this issue

2026-09-02 10:08:47 +00:00

grenade referenced this issue