Since #1 a ranged GET on a file the bucket does not hold streams its slice
from the tee instead of waiting for the whole transfer. For a range that
runs to the end of the file -- what a resumed `hf download` sends -- that
is exactly right, and it stays.
For a range that stops short of the end it is not enough. The slice
arrives quickly and then the *last chunk* is withheld until the manifest
write lands, which waits on every remaining byte, because the digest is
verified only once the whole file has passed. Measured against the live
service, asking for the first 1MiB of a 3.3GB shard:
http=206 ttfb=1.040851s total=120.001638s bytes=1048169
1,048,169 of 1,048,576 bytes in about a second, then an idle connection
for the rest of the transfer. The client's read timeout ends it long
before, so in practice the request fails anyway -- slowly and with no
explanation.
The holdback is not the thing to change: releasing that tail early would
signal "durable and recorded" over bytes that are neither, which is
precisely what it exists to prevent. So answer 503 with Retry-After
instead, and let the fetch run on detached; the retry is served from the
bucket. `hf_transfer` splits every download into bounded ranged chunks, so
this shape is not hypothetical wherever it is enabled.
The existing ranged test now asserts the two-step behaviour and keeps its
guarantee that the bucket holds the whole file and never a fragment. A new
test pins that a resume to EOF is still served from the tee, so the fix for
#1 cannot be undone by this one.
Closes#4
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
rob thijssen
9b35dfdc09
