fix(core): followers stream from the transfer instead of waiting it out
Spec §7 says concurrent requests for the same uncached blob must produce exactly one upstream fetch, and that "subsequent callers subscribe to the same broadcast". Subscribing was the part that did not exist. A follower called `Flight::wait` and blocked -- no timeout, nothing on the wire -- until the leader's entire transfer finished. For a 3.3GB shard at 8MB/s that is 400+ seconds before the response headers are written, against a client that allows 10. Indistinguishable from a hung server, while the leader logs that it stored the blob. It could not be fixed by handing over the live stream: the bytes that already passed are gone from it, and the multipart upload is not readable until it completes. So add the missing buffer. The leader writes every chunk to `server.spool_dir` as it passes and publishes how much is safe to read; a follower opens that file and streams it from byte zero, following it as it grows, with the last chunk withheld until the flight reports the blob stored -- the same rule the tee follows, for the same reason. The spool is not state, and the bucket remains the whole of it. Losing the spool loses nothing: a blob is not recorded until it is durable, so a crash mid-transfer leaves an unreferenced blob and the next request refetches. It is the role the in-memory buffer already plays for small files, on disk so it can hold a large one. Every transfer is spooled rather than only large ones, because size-dependent behaviour is what this service has already been bitten by once. With `spool_dir` unset there is nothing to read and a follower is answered 503 rather than left hanging, which is the honest fallback. The new test asserts the thing that actually distinguishes streaming from waiting -- that the follower's first bytes arrive while the leader is still transferring -- because chunked delivery alone proves nothing: the old path returned chunks too, just all of them after the transfer had ended. It fails with spooling disabled. The eight-concurrent-clients test still sees exactly one upstream fetch, now through the spool. Closes #2 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
This commit is contained in:
@@ -253,6 +253,101 @@ async fn a_sealed_instance_refuses_loudly_for_anything_it_never_stored() {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_follower_streams_from_the_transfer_instead_of_waiting_it_out() {
|
||||
// Spec §7 says concurrent callers "subscribe to the same broadcast". They
|
||||
// could not: the leader's live stream cannot be replayed to someone who
|
||||
// arrived mid-body, and the multipart upload is unreadable until it
|
||||
// completes, so a follower simply waited -- with nothing on the wire --
|
||||
// for the whole transfer. On a multi-gigabyte blob that is minutes against
|
||||
// a client that gives up in ten seconds.
|
||||
//
|
||||
// The spool is the missing buffer: the leader writes each chunk to local
|
||||
// disk as it passes, and a follower reads that from byte zero, following
|
||||
// it as it grows.
|
||||
let hub = FakeHub::start().await;
|
||||
let weights = publish_model(&hub).await;
|
||||
// Slow enough that the leader is still going well after the follower has
|
||||
// started reading; the whole point is what happens during that window.
|
||||
hub.throttle(2 * 1024, Duration::from_millis(20)).await;
|
||||
|
||||
let bucket = bucket_dir("follower-streams");
|
||||
let rf = instance(&bucket, Some(&hub), |_| {});
|
||||
hub.reset_counts();
|
||||
|
||||
let leader = {
|
||||
let app = rf.app.clone();
|
||||
tokio::spawn(async move {
|
||||
call(
|
||||
&app,
|
||||
"GET",
|
||||
"/Qwen/Qwen3-32B/resolve/main/model.safetensors",
|
||||
&[],
|
||||
)
|
||||
.await
|
||||
})
|
||||
};
|
||||
|
||||
// Join once the leader's fetch is genuinely under way.
|
||||
for _ in 0..400 {
|
||||
if hub.file_gets() >= 1 {
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(5)).await;
|
||||
}
|
||||
|
||||
let response = rf
|
||||
.app
|
||||
.clone()
|
||||
.oneshot(
|
||||
Request::builder()
|
||||
.uri("/Qwen/Qwen3-32B/resolve/main/model.safetensors")
|
||||
.body(Body::empty())
|
||||
.unwrap(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
response.status(),
|
||||
StatusCode::OK,
|
||||
"a follower is served, not turned away"
|
||||
);
|
||||
|
||||
use futures::StreamExt;
|
||||
let mut body = response.into_body().into_data_stream();
|
||||
let first = body
|
||||
.next()
|
||||
.await
|
||||
.expect("a chunk while the leader is still transferring")
|
||||
.unwrap();
|
||||
|
||||
// The assertion that matters. Chunked delivery proves nothing -- the old
|
||||
// wait-it-out path also returned the file in chunks, just all of them
|
||||
// after the transfer had finished. What distinguishes streaming from
|
||||
// waiting is *when* the first byte lands: here, while the leader is still
|
||||
// reading from upstream.
|
||||
assert!(
|
||||
!leader.is_finished(),
|
||||
"the follower's first bytes arrived while the leader was still transferring"
|
||||
);
|
||||
assert!(!first.is_empty());
|
||||
|
||||
let mut served = first.to_vec();
|
||||
while let Some(chunk) = body.next().await {
|
||||
served.extend_from_slice(&chunk.unwrap());
|
||||
}
|
||||
assert_eq!(served, weights, "and it receives the whole, correct file");
|
||||
|
||||
let (status, _, leader_body) = leader.await.unwrap();
|
||||
assert_eq!(status, StatusCode::OK);
|
||||
assert_eq!(leader_body, weights);
|
||||
assert_eq!(
|
||||
hub.file_gets(),
|
||||
1,
|
||||
"and it cost no second upstream fetch: single-flight still holds"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn eight_concurrent_clients_produce_exactly_one_upstream_fetch() {
|
||||
let hub = FakeHub::start().await;
|
||||
|
||||
Reference in New Issue
Block a user