With blob_delivery = "proxy", a blob transfer is bounded by request_timeout (300s here). Since a streaming download's duration is size / bandwidth, this turns a total-duration cap into a maximum
downloadable file size — and above that size, no number of retries can
ever succeed.
Fetching Qwen/Qwen3.8-Flash-Next (360 GB, 131 shards of ~3.3 GB) from rf.internal stalls permanently at 52 of 144 blobs.
Evidence
Client is hf download on a fleet host, HF_ENDPOINT=https://rf.internal.
Measured bandwidth from that host, on a blob rustingface has already
stored (so no upstream pull-through in the path):
rf.internal 32 MB ranged GET 7,966,383 B/s ttfb 0.29s
huggingface.co 32 MB ranged GET 14,108,269 B/s ttfb 0.72s
beast -> bob /healthz ttfb 0.003s
The mesh link is fine; the ~8 MB/s is the S3 backend
(caveman.kosherinata.internal:9000, a different site) streaming
through the service.
The partial files match request_timeout x per-connection bandwidth
almost exactly. With two concurrent workers sharing ~8 MB/s, each gets
~4–4.8 MB/s, and 300s at that rate is 1.2–1.4 GB. The three .incomplete files were:
1,426,063,360
1,163,919,360
1,436,549,120
That is the cap cutting the transfer mid-stream, not a stall — and it
implies even --max-workers 1 fails: 300s at the full 8 MB/s is
~2.4 GB, still short of a 3.3 GB shard.
Across 20 attempts: 21 httpx.ReadTimeout, 2 httpx.RemoteProtocolError, and the .incomplete files stopped
growing entirely (mtimes went stale while attempts continued), so the
client is not resuming them across attempts either. The retry loop
cannot converge.
Why I think the cap is the wrong control
client_stall_timeout = 60s already handles the case a total-duration
timeout is usually reaching for — a peer that has gone away, or a
transfer making no progress. It is an idle/rate guard, which is the
right shape for a stream.
request_timeout on top of it adds a second, different rule: this
transfer must finish within 300s regardless of how healthy it is. For a
model registry, that makes success a function of file size × bandwidth —
so the same client, same file, succeeds on a fast link and fails forever
on a slow one, with no diagnostic difference between them.
Also: the failure is invisible server-side
While the client was accumulating 21 ReadTimeouts, journalctl -u rustingface for the same window logged no error, warning, 502 or 504 —
only a successful stored blob and recorded manifest entry. Whatever
cuts the transfer does not say so. Operationally this reads as "the
client is flaky", which is where I started.
Suggestions, in the order I would take them
Exempt blob streaming from request_timeout, leaving client_stall_timeout as the guard. A transfer that is still moving
bytes is not a hung request.
Default blob_delivery to redirect for blobs over some size,
or generally. The 302-to-presigned-URL path takes the service out of
the byte path entirely, which is what a 360 GB repo wants, and it is
already implemented — it is just not the default. (Needs clients to
reach the S3 endpoint directly, which is a deployment question, not a
code one.)
Log when a transfer is terminated by a timeout, with the blob
path, bytes sent and which timeout fired. Right now the server is
silent about its own cut.
Workaround for anyone hitting this
Ranged GETs sized to stay under the cap work fine — the 32 MB probes
above are from the same host, same file, same session. So a client that
chunks below request_timeout x bandwidth completes; hf download does
not chunk, so it cannot.
## Summary
With `blob_delivery = "proxy"`, a blob transfer is bounded by
`request_timeout` (300s here). Since a streaming download's duration is
`size / bandwidth`, this turns a **total-duration cap into a maximum
downloadable file size** — and above that size, no number of retries can
ever succeed.
Fetching `Qwen/Qwen3.8-Flash-Next` (360 GB, 131 shards of ~3.3 GB) from
`rf.internal` stalls permanently at 52 of 144 blobs.
## Evidence
Client is `hf download` on a fleet host, `HF_ENDPOINT=https://rf.internal`.
Measured bandwidth from that host, on a blob rustingface has **already
stored** (so no upstream pull-through in the path):
```
rf.internal 32 MB ranged GET 7,966,383 B/s ttfb 0.29s
huggingface.co 32 MB ranged GET 14,108,269 B/s ttfb 0.72s
beast -> bob /healthz ttfb 0.003s
```
The mesh link is fine; the ~8 MB/s is the S3 backend
(`caveman.kosherinata.internal:9000`, a different site) streaming
through the service.
**The partial files match `request_timeout x per-connection bandwidth`
almost exactly.** With two concurrent workers sharing ~8 MB/s, each gets
~4–4.8 MB/s, and 300s at that rate is 1.2–1.4 GB. The three
`.incomplete` files were:
```
1,426,063,360
1,163,919,360
1,436,549,120
```
That is the cap cutting the transfer mid-stream, not a stall — and it
implies **even `--max-workers 1` fails**: 300s at the full 8 MB/s is
~2.4 GB, still short of a 3.3 GB shard.
Across 20 attempts: 21 `httpx.ReadTimeout`, 2
`httpx.RemoteProtocolError`, and the `.incomplete` files stopped
growing entirely (mtimes went stale while attempts continued), so the
client is not resuming them across attempts either. The retry loop
cannot converge.
## Why I think the cap is the wrong control
`client_stall_timeout = 60s` already handles the case a total-duration
timeout is usually reaching for — a peer that has gone away, or a
transfer making no progress. It is an idle/rate guard, which is the
right shape for a stream.
`request_timeout` on top of it adds a second, different rule: *this
transfer must finish within 300s regardless of how healthy it is*. For a
model registry, that makes success a function of file size × bandwidth —
so the same client, same file, succeeds on a fast link and fails forever
on a slow one, with no diagnostic difference between them.
## Also: the failure is invisible server-side
While the client was accumulating 21 `ReadTimeout`s, `journalctl -u
rustingface` for the same window logged no error, warning, 502 or 504 —
only a successful `stored blob and recorded manifest entry`. Whatever
cuts the transfer does not say so. Operationally this reads as "the
client is flaky", which is where I started.
## Suggestions, in the order I would take them
1. **Exempt blob streaming from `request_timeout`**, leaving
`client_stall_timeout` as the guard. A transfer that is still moving
bytes is not a hung request.
2. **Default `blob_delivery` to `redirect` for blobs over some size**,
or generally. The 302-to-presigned-URL path takes the service out of
the byte path entirely, which is what a 360 GB repo wants, and it is
already implemented — it is just not the default. (Needs clients to
reach the S3 endpoint directly, which is a deployment question, not a
code one.)
3. **Log when a transfer is terminated by a timeout**, with the blob
path, bytes sent and which timeout fired. Right now the server is
silent about its own cut.
## Workaround for anyone hitting this
Ranged GETs sized to stay under the cap work fine — the 32 MB probes
above are from the same host, same file, same session. So a client that
chunks below `request_timeout x bandwidth` completes; `hf download` does
not chunk, so it cannot.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
With
blob_delivery = "proxy", a blob transfer is bounded byrequest_timeout(300s here). Since a streaming download's duration issize / bandwidth, this turns a total-duration cap into a maximumdownloadable file size — and above that size, no number of retries can
ever succeed.
Fetching
Qwen/Qwen3.8-Flash-Next(360 GB, 131 shards of ~3.3 GB) fromrf.internalstalls permanently at 52 of 144 blobs.Evidence
Client is
hf downloadon a fleet host,HF_ENDPOINT=https://rf.internal.Measured bandwidth from that host, on a blob rustingface has already
stored (so no upstream pull-through in the path):
The mesh link is fine; the ~8 MB/s is the S3 backend
(
caveman.kosherinata.internal:9000, a different site) streamingthrough the service.
The partial files match
request_timeout x per-connection bandwidthalmost exactly. With two concurrent workers sharing ~8 MB/s, each gets
~4–4.8 MB/s, and 300s at that rate is 1.2–1.4 GB. The three
.incompletefiles were:That is the cap cutting the transfer mid-stream, not a stall — and it
implies even
--max-workers 1fails: 300s at the full 8 MB/s is~2.4 GB, still short of a 3.3 GB shard.
Across 20 attempts: 21
httpx.ReadTimeout, 2httpx.RemoteProtocolError, and the.incompletefiles stoppedgrowing entirely (mtimes went stale while attempts continued), so the
client is not resuming them across attempts either. The retry loop
cannot converge.
Why I think the cap is the wrong control
client_stall_timeout = 60salready handles the case a total-durationtimeout is usually reaching for — a peer that has gone away, or a
transfer making no progress. It is an idle/rate guard, which is the
right shape for a stream.
request_timeouton top of it adds a second, different rule: thistransfer must finish within 300s regardless of how healthy it is. For a
model registry, that makes success a function of file size × bandwidth —
so the same client, same file, succeeds on a fast link and fails forever
on a slow one, with no diagnostic difference between them.
Also: the failure is invisible server-side
While the client was accumulating 21
ReadTimeouts,journalctl -u rustingfacefor the same window logged no error, warning, 502 or 504 —only a successful
stored blob and recorded manifest entry. Whatevercuts the transfer does not say so. Operationally this reads as "the
client is flaky", which is where I started.
Suggestions, in the order I would take them
request_timeout, leavingclient_stall_timeoutas the guard. A transfer that is still movingbytes is not a hung request.
blob_deliverytoredirectfor blobs over some size,or generally. The 302-to-presigned-URL path takes the service out of
the byte path entirely, which is what a 360 GB repo wants, and it is
already implemented — it is just not the default. (Needs clients to
reach the S3 endpoint directly, which is a deployment question, not a
code one.)
path, bytes sent and which timeout fired. Right now the server is
silent about its own cut.
Workaround for anyone hitting this
Ranged GETs sized to stay under the cap work fine — the 32 MB probes
above are from the same host, same file, same session. So a client that
chunks below
request_timeout x bandwidthcompletes;hf downloaddoesnot chunk, so it cannot.