Seen on 2026-09-29, on the first pull of Tongyi-MAI/Z-Image-Turbo (commit f332072a) through rf.internal. No client had ever fetched that repo through the registry. Two clients were involved:
neuron (Rust hf-hub 0.4.3), loading the model directly;
Python huggingface_hub 1.xsnapshot_download, pre-fetching the same repo about two minutes later.
Both problems occur only while rustingface is filling a blob it does not hold yet. Once a blob is stored, the same requests succeed (a retry of model_index.json a few minutes later returned 200).
1. 503 on a bounded range during an in-flight fetch breaks neuron's cold load
neuron's first request for the repo was the hf-hub size probe on model_index.json, a bounded range for the first byte. rustingface refused it deliberately:
18:55:15.355138Z WARN a bounded range on a file the bucket does not hold cannot be completed until the
whole file is verified; answering 503 rather than stalling the client on its last chunk
repo_id=Tongyi-MAI/Z-Image-Turbo path=model_index.json asked=0..1 size=467
18:55:15.355224Z ERROR request failed err="fetch in progress: …/model_index.json is being fetched; a range
that stops short of the end can only be served once it is stored. Retry shortly."
18:55:15.355296Z ERROR response failed classification="Status code: 503 Service Unavailable" latency="684 ms"
18:55:15.434235Z INFO stored blob and recorded manifest entry path=model_index.json size=467
The blob was stored 79 ms after the 503 went out. neuron's hf-hub does not retry a 503, so the whole model load failed:
load_model failed model=Tongyi-MAI/Z-Image-Turbo error=fetch model_index.json from huggingface:Tongyi-MAI/Z-Image-Turbo:
request error: HTTP status server error (503 Service Unavailable) for url (https://rf.internal/Tongyi-MAI/Z-Image-Turbo/resolve/main/model_index.json)
Impact: any model rf.internal has not cached yet cannot be cold-loaded by a neuron. That includes a cortex-driven cold load, which only surfaces as a failed request. It works only after some other client has warmed the registry. Every hf-hubget() starts with this bounded-range probe, so it isn't specific to this repo.
Possible directions (not verified against the code):
For a bounded range, wait for the in-flight fetch with a short cap. The probe here would have waited 79 ms.
Or serve a range that lies wholly within the prefix already streamed from upstream, instead of requiring the whole file to be verified first.
At minimum, send Retry-After with the 503. neuron would still need a retry on its side, which I'll handle separately in helexa.
2. A resumed range request gets headers but an empty body
The Python pre-fetch had its connections dropped mid-shard and resumed with a Range for the remainder. Twice, a resumed request arriving just as the blob was being stored got a response announcing the remaining length and then zero bytes:
18:57:18.161384Z INFO stored blob and recorded manifest entry path=text_encoder/model-00003-of-00003.safetensors size=99630640
18:57:18.189596Z WARN blob response ended early with no error; the client sees a short read
path=…/text_encoder/model-00003-of-00003.safetensors sent=0 expected=5258800
18:57:33.688382Z INFO stored blob and recorded manifest entry path=vae/diffusion_pytorch_model.safetensors size=167666902
18:57:33.726743Z WARN blob response ended early with no error; the client sees a short read
path=…/vae/diffusion_pytorch_model.safetensors sent=0 expected=10380502
In both cases the short read comes 28–38 ms after the "stored blob" line for the same path. expected is the tail the client was resuming (5.26 MB of a 99.6 MB shard), not the full size. The client saw peer closed connection without sending complete message body (received 0 bytes, expected 5258800), retried, and succeeded once the blob was served from the bucket.
That looks like a race: a range response is set up against the in-flight fetch, the fetch completes and hands over to the stored copy, and the response body stream ends without yielding. Python's huggingface_hub absorbs it by resuming again. A client without resume logic would fail the file.
Scale and reproduction
2 short reads and 1 503 over a ~29 GB first pull (5 components, 3 text-encoder shards).
Throughput otherwise about 2.2 GB/min.
To reproduce: pick a repo rf.internal has never served and hf-hub/huggingface_hub-fetch it from a clean client cache. The 503 is the first request. The short reads need a resumed range landing near the end of a blob's fill, so they are timing-dependent.
Client versions: hf-hub 0.4.3 (neuron); huggingface_hub 1.x with HF_HUB_DISABLE_XET=1.
Seen on 2026-09-29, on the first pull of `Tongyi-MAI/Z-Image-Turbo` (commit `f332072a`) through rf.internal. No client had ever fetched that repo through the registry. Two clients were involved:
- **neuron** (Rust `hf-hub` 0.4.3), loading the model directly;
- **Python `huggingface_hub` 1.x** `snapshot_download`, pre-fetching the same repo about two minutes later.
Both problems occur only while rustingface is **filling a blob it does not hold yet**. Once a blob is stored, the same requests succeed (a retry of `model_index.json` a few minutes later returned 200).
## 1. 503 on a bounded range during an in-flight fetch breaks neuron's cold load
neuron's first request for the repo was the hf-hub size probe on `model_index.json`, a bounded range for the first byte. rustingface refused it deliberately:
```
18:55:15.355138Z WARN a bounded range on a file the bucket does not hold cannot be completed until the
whole file is verified; answering 503 rather than stalling the client on its last chunk
repo_id=Tongyi-MAI/Z-Image-Turbo path=model_index.json asked=0..1 size=467
18:55:15.355224Z ERROR request failed err="fetch in progress: …/model_index.json is being fetched; a range
that stops short of the end can only be served once it is stored. Retry shortly."
18:55:15.355296Z ERROR response failed classification="Status code: 503 Service Unavailable" latency="684 ms"
18:55:15.434235Z INFO stored blob and recorded manifest entry path=model_index.json size=467
```
The blob was stored **79 ms after** the 503 went out. neuron's `hf-hub` does not retry a 503, so the whole model load failed:
```
load_model failed model=Tongyi-MAI/Z-Image-Turbo error=fetch model_index.json from huggingface:Tongyi-MAI/Z-Image-Turbo:
request error: HTTP status server error (503 Service Unavailable) for url (https://rf.internal/Tongyi-MAI/Z-Image-Turbo/resolve/main/model_index.json)
```
**Impact:** any model rf.internal has not cached yet cannot be cold-loaded by a neuron. That includes a cortex-driven cold load, which only surfaces as a failed request. It works only after some other client has warmed the registry. Every `hf-hub` `get()` starts with this bounded-range probe, so it isn't specific to this repo.
**Possible directions** (not verified against the code):
- For a bounded range, wait for the in-flight fetch with a short cap. The probe here would have waited 79 ms.
- Or serve a range that lies wholly within the prefix already streamed from upstream, instead of requiring the whole file to be verified first.
- At minimum, send `Retry-After` with the 503. neuron would still need a retry on its side, which I'll handle separately in helexa.
## 2. A resumed range request gets headers but an empty body
The Python pre-fetch had its connections dropped mid-shard and resumed with a `Range` for the remainder. Twice, a resumed request arriving just as the blob was being stored got a response announcing the remaining length and then **zero bytes**:
```
18:57:18.161384Z INFO stored blob and recorded manifest entry path=text_encoder/model-00003-of-00003.safetensors size=99630640
18:57:18.189596Z WARN blob response ended early with no error; the client sees a short read
path=…/text_encoder/model-00003-of-00003.safetensors sent=0 expected=5258800
18:57:33.688382Z INFO stored blob and recorded manifest entry path=vae/diffusion_pytorch_model.safetensors size=167666902
18:57:33.726743Z WARN blob response ended early with no error; the client sees a short read
path=…/vae/diffusion_pytorch_model.safetensors sent=0 expected=10380502
```
In both cases the short read comes **28–38 ms after** the "stored blob" line for the same path. `expected` is the tail the client was resuming (5.26 MB of a 99.6 MB shard), not the full size. The client saw `peer closed connection without sending complete message body (received 0 bytes, expected 5258800)`, retried, and succeeded once the blob was served from the bucket.
That looks like a race: a range response is set up against the in-flight fetch, the fetch completes and hands over to the stored copy, and the response body stream ends without yielding. Python's `huggingface_hub` absorbs it by resuming again. A client without resume logic would fail the file.
## Scale and reproduction
- 2 short reads and 1 503 over a ~29 GB first pull (5 components, 3 text-encoder shards).
- Throughput otherwise about 2.2 GB/min.
- **To reproduce:** pick a repo rf.internal has never served and `hf-hub`/`huggingface_hub`-fetch it from a clean client cache. The 503 is the first request. The short reads need a resumed range landing near the end of a blob's fill, so they are timing-dependent.
Client versions: `hf-hub` 0.4.3 (neuron); `huggingface_hub` 1.x with `HF_HUB_DISABLE_XET=1`.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Seen on 2026-09-29, on the first pull of
Tongyi-MAI/Z-Image-Turbo(commitf332072a) through rf.internal. No client had ever fetched that repo through the registry. Two clients were involved:hf-hub0.4.3), loading the model directly;huggingface_hub1.xsnapshot_download, pre-fetching the same repo about two minutes later.Both problems occur only while rustingface is filling a blob it does not hold yet. Once a blob is stored, the same requests succeed (a retry of
model_index.jsona few minutes later returned 200).1. 503 on a bounded range during an in-flight fetch breaks neuron's cold load
neuron's first request for the repo was the hf-hub size probe on
model_index.json, a bounded range for the first byte. rustingface refused it deliberately:The blob was stored 79 ms after the 503 went out. neuron's
hf-hubdoes not retry a 503, so the whole model load failed:Impact: any model rf.internal has not cached yet cannot be cold-loaded by a neuron. That includes a cortex-driven cold load, which only surfaces as a failed request. It works only after some other client has warmed the registry. Every
hf-hubget()starts with this bounded-range probe, so it isn't specific to this repo.Possible directions (not verified against the code):
Retry-Afterwith the 503. neuron would still need a retry on its side, which I'll handle separately in helexa.2. A resumed range request gets headers but an empty body
The Python pre-fetch had its connections dropped mid-shard and resumed with a
Rangefor the remainder. Twice, a resumed request arriving just as the blob was being stored got a response announcing the remaining length and then zero bytes:In both cases the short read comes 28–38 ms after the "stored blob" line for the same path.
expectedis the tail the client was resuming (5.26 MB of a 99.6 MB shard), not the full size. The client sawpeer closed connection without sending complete message body (received 0 bytes, expected 5258800), retried, and succeeded once the blob was served from the bucket.That looks like a race: a range response is set up against the in-flight fetch, the fetch completes and hands over to the stored copy, and the response body stream ends without yielding. Python's
huggingface_hubabsorbs it by resuming again. A client without resume logic would fail the file.Scale and reproduction
hf-hub/huggingface_hub-fetch it from a clean client cache. The 503 is the first request. The short reads need a resumed range landing near the end of a blob's fill, so they are timing-dependent.Client versions:
hf-hub0.4.3 (neuron);huggingface_hub1.x withHF_HUB_DISABLE_XET=1.