A request that joins an in-flight fetch waits in silence for the whole transfer #2

New Issue

grenade · 2026-09-02T11:02:11Z

grenade commented

2026-09-02 11:02:11 +00:00

Summary

When a request arrives for a blob another request is already fetching, it
joins the flight as a follower and calls Flight::wait
(crates/rustingface-core/src/flight.rs:100), which blocks — with no
timeout and no bytes on the wire — until the leader's entire transfer
completes. Only then is the blob read back from the bucket and served.

For a 3.3GB shard at the ~8MB/s a GPU host gets to caveman, that is 400+
seconds before the response headers are even written. huggingface_hub's
default read timeout is 10 seconds (HF_HUB_DOWNLOAD_TIMEOUT), so the
client gives up roughly forty times over before rustingface says anything.

From the client this is indistinguishable from a hung server. From the
server nothing is wrong at all: the leader's transfer succeeds and logs
stored blob and recorded manifest entry.

Why this is worth fixing now

This is the same failure shape as the ranged-fetch half of #1 — the client
holding an open connection receiving nothing while the server considers
itself healthy — and it is the part #1's fix did not address. #1 fixed
the leader path (a ranged request now streams its slice from the tee as
the bytes pass). The follower path still blocks.

It also gets more likely as a result of fixing #1, not less. Lifting the
180s x bandwidth ceiling means transfers now legitimately run for many
minutes, which widens the window in which a second request can land on one.

How it is reached

hf download --max-workers N is enough. Any of:

two workers asked for the same file;
a client retrying a file whose first attempt was cut, while the detached
transfer from that first attempt is still running (this is the common
one — the transfer deliberately survives client disconnect, so a prompt
retry lands squarely on it);
two hosts pulling the same model at once, which is exactly the
fleet-rollout case single-flight exists to serve.

Constraints on any fix

Single-flight must hold. Spec §7, and tests/sovereignty.rs
asserts it ("a fleet-wide rollout must not become eight concurrent
pulls"). Letting followers fetch upstream independently trades this bug
for the one the design was built to prevent.
A follower genuinely cannot be streamed the bytes that already
passed. They are gone from the leader's stream and not yet readable in
the bucket (the multipart upload is incomplete). flight.rs is right
that a follower attached mid-body would get a stream missing its start.
So the fix is not "stream to the follower". It is "stop pretending to
be about to answer": bound the wait, and when it is exceeded, return a
retryable response rather than continue holding a connection that the
client has already given up on.

Suggested direction

Bound the follower's wait well under a client read timeout (a few
seconds), and on expiry answer 503 with Retry-After instead of
blocking. The leader keeps transferring, the client backs off and retries,
and the retry is served from the bucket once the blob lands. That
converges, keeps single-flight intact, and is honest about what is
happening.

Worth verifying against the pinned client rather than assuming: the
conformance suite should grow a case asserting that a second concurrent
request for a cold blob gets a prompt retryable answer, and that an
unmodified huggingface_hub recovers from it.

## Summary When a request arrives for a blob another request is already fetching, it joins the flight as a follower and calls `Flight::wait` (`crates/rustingface-core/src/flight.rs:100`), which blocks — with **no timeout and no bytes on the wire** — until the leader's entire transfer completes. Only then is the blob read back from the bucket and served. For a 3.3GB shard at the ~8MB/s a GPU host gets to caveman, that is 400+ seconds before the response headers are even written. `huggingface_hub`'s default read timeout is 10 seconds (`HF_HUB_DOWNLOAD_TIMEOUT`), so the client gives up roughly forty times over before rustingface says anything. From the client this is indistinguishable from a hung server. From the server nothing is wrong at all: the leader's transfer succeeds and logs `stored blob and recorded manifest entry`. ## Why this is worth fixing now This is the same failure shape as the ranged-fetch half of #1 — the client holding an open connection receiving nothing while the server considers itself healthy — and it is the part #1's fix did **not** address. #1 fixed the *leader* path (a ranged request now streams its slice from the tee as the bytes pass). The follower path still blocks. It also gets *more* likely as a result of fixing #1, not less. Lifting the `180s x bandwidth` ceiling means transfers now legitimately run for many minutes, which widens the window in which a second request can land on one. ## How it is reached `hf download --max-workers N` is enough. Any of: - two workers asked for the same file; - a client retrying a file whose first attempt was cut, while the detached transfer from that first attempt is still running (this is the common one — the transfer deliberately survives client disconnect, so a prompt retry lands squarely on it); - two hosts pulling the same model at once, which is exactly the fleet-rollout case single-flight exists to serve. ## Constraints on any fix - **Single-flight must hold.** Spec §7, and `tests/sovereignty.rs` asserts it ("a fleet-wide rollout must not become eight concurrent pulls"). Letting followers fetch upstream independently trades this bug for the one the design was built to prevent. - **A follower genuinely cannot be streamed the bytes that already passed.** They are gone from the leader's stream and not yet readable in the bucket (the multipart upload is incomplete). `flight.rs` is right that a follower attached mid-body would get a stream missing its start. - So the fix is not "stream to the follower". It is "**stop pretending to be about to answer**": bound the wait, and when it is exceeded, return a retryable response rather than continue holding a connection that the client has already given up on. ## Suggested direction Bound the follower's wait well under a client read timeout (a few seconds), and on expiry answer `503` with `Retry-After` instead of blocking. The leader keeps transferring, the client backs off and retries, and the retry is served from the bucket once the blob lands. That converges, keeps single-flight intact, and is honest about what is happening. Worth verifying against the pinned client rather than assuming: the conformance suite should grow a case asserting that a second concurrent request for a cold blob gets a prompt retryable answer, and that an unmodified `huggingface_hub` recovers from it.

grenade commented

2026-09-02 11:04:07 +00:00

Measured the pinned client before building on the suggestion above, and it
does not hold up. Correcting it.

huggingface_hub 0.36 routes file requests through
_request_wrapper -> http_backoff, whose defaults are:

max_retries    = 5
base_wait_time = 1
max_wait_time  = 8
retry_on_status_codes = (500, 502, 503, 504)
retry_on_exceptions   = (Timeout, ConnectionError, ChunkedEncodingError)

Two things follow:

503 is retried — good, the direction is not useless.
The retry budget is tiny, and Retry-After is ignored. The loop
sleeps its own exponential backoff (time.sleep(sleep_time)) and never
reads the header. Five retries at 1, 2, 4, 8, 8 is ~23s of sleeping
plus five round trips.

So for a 3.3GB shard at 8MB/s — a ~420s transfer — a follower answering
503 promptly gives the client roughly 50 seconds of trying before it
raises. It does not converge. The same arithmetic sinks "wait and let the
client time out", since requests.Timeout is retried out of the same
budget.

What this means. A prompt 503 is still a strict improvement over
hanging: it converges for everything the leader can finish inside the
client's budget (configs, tokenizers, and shards up to roughly 200MB at
8MB/s), and for anything larger it fails in seconds with a legible reason
instead of hanging invisibly. But it is a mitigation, not a fix, and it
should not be described as one.

Converging for large blobs requires serving the follower bytes, and
every way of doing that runs into something:

Follower fetches its own range from upstream — converges perfectly and
keeps a single stored copy, but a fleet rollout becomes N concurrent
upstream pulls, which is the exact harm single-flight exists to prevent
(spec §7, and the sovereignty test asserting it).
Replay buffer on the leader's tee — only helps a follower whose
requested offset is within the buffer of the leader's current position.
A client resuming from where it was cut is usually far behind that.
Spool the transfer to local disk so followers can read the completed
prefix while the leader appends — actually converges, but makes a local
file part of the serving path, which cuts against the bucket being the
whole of the system state.

That last one is a real design question rather than a bug fix, so it wants
deciding on its own merits rather than being smuggled in here.

Proposed split: take the prompt-503 mitigation now, and keep this issue
open for the convergence question with the three options above on the
table.

Measured the pinned client before building on the suggestion above, and it does not hold up. Correcting it. `huggingface_hub` 0.36 routes file requests through `_request_wrapper` -> `http_backoff`, whose defaults are: ``` max_retries = 5 base_wait_time = 1 max_wait_time = 8 retry_on_status_codes = (500, 502, 503, 504) retry_on_exceptions = (Timeout, ConnectionError, ChunkedEncodingError) ``` Two things follow: 1. **503 is retried** — good, the direction is not useless. 2. **The retry budget is tiny, and `Retry-After` is ignored.** The loop sleeps its own exponential backoff (`time.sleep(sleep_time)`) and never reads the header. Five retries at 1, 2, 4, 8, 8 is ~23s of sleeping plus five round trips. So for a 3.3GB shard at 8MB/s — a ~420s transfer — a follower answering 503 promptly gives the client roughly 50 seconds of trying before it raises. It does not converge. The same arithmetic sinks "wait and let the client time out", since `requests.Timeout` is retried out of the same budget. **What this means.** A prompt 503 is still a strict improvement over hanging: it converges for everything the leader can finish inside the client's budget (configs, tokenizers, and shards up to roughly 200MB at 8MB/s), and for anything larger it fails in seconds with a legible reason instead of hanging invisibly. But it is a mitigation, not a fix, and it should not be described as one. **Converging for large blobs requires serving the follower bytes**, and every way of doing that runs into something: - *Follower fetches its own range from upstream* — converges perfectly and keeps a single stored copy, but a fleet rollout becomes N concurrent upstream pulls, which is the exact harm single-flight exists to prevent (spec §7, and the sovereignty test asserting it). - *Replay buffer on the leader's tee* — only helps a follower whose requested offset is within the buffer of the leader's current position. A client resuming from where it was cut is usually far behind that. - *Spool the transfer to local disk so followers can read the completed prefix while the leader appends* — actually converges, but makes a local file part of the serving path, which cuts against the bucket being the whole of the system state. That last one is a real design question rather than a bug fix, so it wants deciding on its own merits rather than being smuggled in here. Proposed split: take the prompt-503 mitigation now, and keep this issue open for the convergence question with the three options above on the table.

grenade referenced this issue from a commit

2026-09-02 11:34:22 +00:00

fix(core): followers stream from the transfer instead of waiting it out

grenade closed this issue

2026-09-02 11:44:08 +00:00

grenade referenced this issue from a commit

2026-09-02 11:44:08 +00:00

Merge branch 'fix/followers-stream-from-the-transfer'

Sign in to join this conversation.