Spec §7 says concurrent requests for the same uncached blob must produce
exactly one upstream fetch, and that "subsequent callers subscribe to the
same broadcast". Subscribing was the part that did not exist. A follower
called `Flight::wait` and blocked -- no timeout, nothing on the wire --
until the leader's entire transfer finished. For a 3.3GB shard at 8MB/s
that is 400+ seconds before the response headers are written, against a
client that allows 10. Indistinguishable from a hung server, while the
leader logs that it stored the blob.
It could not be fixed by handing over the live stream: the bytes that
already passed are gone from it, and the multipart upload is not readable
until it completes. So add the missing buffer. The leader writes every
chunk to `server.spool_dir` as it passes and publishes how much is safe to
read; a follower opens that file and streams it from byte zero, following
it as it grows, with the last chunk withheld until the flight reports the
blob stored -- the same rule the tee follows, for the same reason.
The spool is not state, and the bucket remains the whole of it. Losing the
spool loses nothing: a blob is not recorded until it is durable, so a crash
mid-transfer leaves an unreferenced blob and the next request refetches. It
is the role the in-memory buffer already plays for small files, on disk so
it can hold a large one. Every transfer is spooled rather than only large
ones, because size-dependent behaviour is what this service has already
been bitten by once.
With `spool_dir` unset there is nothing to read and a follower is answered
503 rather than left hanging, which is the honest fallback.
The new test asserts the thing that actually distinguishes streaming from
waiting -- that the follower's first bytes arrive while the leader is still
transferring -- because chunked delivery alone proves nothing: the old path
returned chunks too, just all of them after the transfer had ended. It
fails with spooling disabled. The eight-concurrent-clients test still sees
exactly one upstream fetch, now through the spool.
Closes#2
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
rob thijssen
14f8342cf8
