fix(core): followers stream from the transfer instead of waiting it out
ci / web (push) Successful in 1m37s
ci / check (push) Successful in 6m34s

Spec §7 says concurrent requests for the same uncached blob must produce
exactly one upstream fetch, and that "subsequent callers subscribe to the
same broadcast". Subscribing was the part that did not exist. A follower
called `Flight::wait` and blocked -- no timeout, nothing on the wire --
until the leader's entire transfer finished. For a 3.3GB shard at 8MB/s
that is 400+ seconds before the response headers are written, against a
client that allows 10. Indistinguishable from a hung server, while the
leader logs that it stored the blob.

It could not be fixed by handing over the live stream: the bytes that
already passed are gone from it, and the multipart upload is not readable
until it completes. So add the missing buffer. The leader writes every
chunk to `server.spool_dir` as it passes and publishes how much is safe to
read; a follower opens that file and streams it from byte zero, following
it as it grows, with the last chunk withheld until the flight reports the
blob stored -- the same rule the tee follows, for the same reason.

The spool is not state, and the bucket remains the whole of it. Losing the
spool loses nothing: a blob is not recorded until it is durable, so a crash
mid-transfer leaves an unreferenced blob and the next request refetches. It
is the role the in-memory buffer already plays for small files, on disk so
it can hold a large one. Every transfer is spooled rather than only large
ones, because size-dependent behaviour is what this service has already
been bitten by once.

With `spool_dir` unset there is nothing to read and a follower is answered
503 rather than left hanging, which is the honest fallback.

The new test asserts the thing that actually distinguishes streaming from
waiting -- that the follower's first bytes arrive while the leader is still
transferring -- because chunked delivery alone proves nothing: the old path
returned chunks too, just all of them after the transfer had ended. It
fails with spooling disabled. The eight-concurrent-clients test still sees
exactly one upstream fetch, now through the spool.

Closes #2

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
This commit is contained in:
2026-09-02 14:34:13 +03:00
co-authored by Claude Opus 5
parent 9b35dfdc09
commit 14f8342cf8
9 changed files with 534 additions and 5 deletions
+18
View File
@@ -227,6 +227,8 @@ listen = "127.0.0.1:8080"
external_url = "https://models.internal.example" # used to build redirect URLs
client_stall_timeout = "60s" # idle guard on the tee; see below
shutdown_grace = "15s" # bounded drain on SIGTERM; see below
spool_dir = "/tmp/rustingface-spool" # in-flight bytes, readable by followers
flight_wait_timeout = "5s" # how long a follower waits for a spool
[storage]
endpoint = "https://s3.internal.example:9000"
@@ -410,6 +412,22 @@ subsequent callers subscribe to the same broadcast rather than issuing their own
request. Without this, a fleet-wide rollout of a new model produces N concurrent
40GB pulls.
Subscribing is what the spool is for. A follower cannot be handed the leader's
live stream — the bytes that already passed are gone from it, and the multipart
upload is not readable until it completes — so without one it can only wait for
the entire transfer, with nothing on the wire, which a client's read timeout
ends long before. The leader therefore writes every chunk to `server.spool_dir`
as it passes and publishes how much is safe to read; a follower opens that file
and streams it from the start, following it as it grows. One upstream fetch,
every caller served, nobody in silence.
The spool is not state. Losing it loses nothing, because a blob is not recorded
until it is durable in the bucket, so a crash mid-transfer leaves an
unreferenced blob and the next request refetches. It is the role the in-memory
buffer already plays for small files, on disk so it can hold a large one. With
`spool_dir` unset there is nothing for a follower to read and it is answered
`503` instead.
### Streaming tee
The upstream response body is consumed once and written to two sinks