Spec §7 says concurrent requests for the same uncached blob must produce
exactly one upstream fetch, and that "subsequent callers subscribe to the
same broadcast". Subscribing was the part that did not exist. A follower
called `Flight::wait` and blocked -- no timeout, nothing on the wire --
until the leader's entire transfer finished. For a 3.3GB shard at 8MB/s
that is 400+ seconds before the response headers are written, against a
client that allows 10. Indistinguishable from a hung server, while the
leader logs that it stored the blob.
It could not be fixed by handing over the live stream: the bytes that
already passed are gone from it, and the multipart upload is not readable
until it completes. So add the missing buffer. The leader writes every
chunk to `server.spool_dir` as it passes and publishes how much is safe to
read; a follower opens that file and streams it from byte zero, following
it as it grows, with the last chunk withheld until the flight reports the
blob stored -- the same rule the tee follows, for the same reason.
The spool is not state, and the bucket remains the whole of it. Losing the
spool loses nothing: a blob is not recorded until it is durable, so a crash
mid-transfer leaves an unreferenced blob and the next request refetches. It
is the role the in-memory buffer already plays for small files, on disk so
it can hold a large one. Every transfer is spooled rather than only large
ones, because size-dependent behaviour is what this service has already
been bitten by once.
With `spool_dir` unset there is nothing to read and a follower is answered
503 rather than left hanging, which is the honest fallback.
The new test asserts the thing that actually distinguishes streaming from
waiting -- that the follower's first bytes arrive while the leader is still
transferring -- because chunked delivery alone proves nothing: the old path
returned chunks too, just all of them after the transfer had ended. It
fails with spooling disabled. The eight-concurrent-clients test still sees
exactly one upstream fetch, now through the spool.
Closes#2
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
Since #1 a ranged GET on a file the bucket does not hold streams its slice
from the tee instead of waiting for the whole transfer. For a range that
runs to the end of the file -- what a resumed `hf download` sends -- that
is exactly right, and it stays.
For a range that stops short of the end it is not enough. The slice
arrives quickly and then the *last chunk* is withheld until the manifest
write lands, which waits on every remaining byte, because the digest is
verified only once the whole file has passed. Measured against the live
service, asking for the first 1MiB of a 3.3GB shard:
http=206 ttfb=1.040851s total=120.001638s bytes=1048169
1,048,169 of 1,048,576 bytes in about a second, then an idle connection
for the rest of the transfer. The client's read timeout ends it long
before, so in practice the request fails anyway -- slowly and with no
explanation.
The holdback is not the thing to change: releasing that tail early would
signal "durable and recorded" over bytes that are neither, which is
precisely what it exists to prevent. So answer 503 with Retry-After
instead, and let the fetch run on detached; the retry is served from the
bucket. `hf_transfer` splits every download into bounded ranged chunks, so
this shape is not hypothetical wherever it is enabled.
The existing ranged test now asserts the two-step behaviour and keeps its
guarantee that the bucket holds the whole file and never a fragment. A new
test pins that a resume to EOF is still served from the tee, so the fix for
#1 cannot be undone by this one.
Closes#4
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
A ranged GET on a file the bucket does not hold used to fetch and store
the whole thing before sending a single byte. Asking for the first
megabyte of a 3.3GB shard therefore held the connection open, silent, for
the length of the entire transfer: 93 seconds measured against the live
service at 28MB/s, and 400-800s at the 8MB/s a GPU host gets. No client
waits that long for a first byte.
What makes it unrecoverable rather than merely slow is that
`huggingface_hub` resumes with a Range header. So the first failure
leaves an `.incomplete` file, and *every retry from then on* takes this
path and is answered with silence. The partial file never grows, its
mtime goes stale, and the retry loop cannot converge -- while the server
logs nothing but `stored blob and recorded manifest entry`, because from
its side the fetch succeeded.
The whole file must still be fetched and stored, since a fragment must
never reach the bucket. What must not happen is the client waiting on
that in silence. So narrow the tee instead of blocking on it: the client
is sent its slice as those bytes pass, one chunk behind as before, with
the last piece still withheld until the manifest write lands. For the
open-ended `bytes=N-` a resume actually asks for, that is a continuous
stream to the end of the file.
`blocking` and `LeadResult::Completed` go with it; nothing else used
them. The test asserting a ranged cold request stores the whole file and
never a fragment is unchanged and still passes.
Refs #1
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
`object_store` puts a 30-second `reqwest` timeout on the whole request,
body included. On a proxied blob that is not a timeout at all: the bucket
is read at the pace the client drains it, so it is a maximum servable
file size.
The reason it never looked like one is that object_store hides its own
consequences. It catches every response-body error and silently retries
with a resumed Range, so a large read does not fail at 30s -- it
reconnects every 30s and keeps going, until RetryConfig's 180s
retry_timeout is spent. Then the body simply ends, with a 200 already on
the wire and nothing logged anywhere. The ceiling is 180s x bandwidth: at
the 8MB/s a GPU host gets to caveman, no file over ~1.4GB can ever be
served, and no client retry converges because the next attempt is no
faster. Measured against the live bucket, 500MiB at 1MB/s: cut at 184.1s,
193,340,351 of 524,288,000 bytes. With the timeout disabled, 499.6s and
complete -- the same wall time curl takes reading the same object
straight from MinIO.
Disable it, and guard the stream the way a stream should be guarded:
`storage.connect_timeout` and an idle `storage.read_timeout` that bounds
silence rather than progress. A test asserts the total cap stays off.
Issue #1 named `server.request_timeout` as the cause. It was not -- that
key was parsed and then read by nothing, which is its own problem and is
why the hunt started 300 seconds away from a 180-second bug. Remove it.
Also log when a blob response ends before its Content-Length. Once the
headers are out the bytes leave through hyper, so a cut mid-body was
invisible on this side: the service logged a clean 200 while the client
saw a short read and retried forever.
Refs #1
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
The lock added alongside compare-and-set removed its map entry inside
the critical section. A queued waiter still held that mutex, so the next
arrival found no entry, created a fresh one, and the two then ran
concurrently -- defeating the lock exactly when there was contention,
which is the only time it does anything.
I previously reported this fix as working on the strength of ten clean
release-mode runs. That was wrong: in debug, where the timing differs,
the guarding test fails 3 times in 12. Retiring the entry only after
dropping our own reference, under the same shard lock `entry` takes,
gives 0 failures in 25 -- and restoring the old behaviour reproduces 3
in 12, so the test really does catch it.
The consequence was the one the test names: a file served to a client
but never recorded, and so invisible the moment the instance is sealed.
Real against the local filesystem store, which has no conditional put
and relies on the verify-after-write retry; MinIO's compare-and-set
covers the same race, which is why the deployed path never showed it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
Run 12 failed both build jobs, for two unrelated reasons of mine.
cargo fmt: the previous commit ran clippy and the test suite but not
the formatter, and inventory.rs was written by hand. Formatted.
pnpm: the runner enforces a minimum release age on lockfile entries --
a supply-chain safeguard against a package published minutes ago -- and
rejected electron-to-chromium@1.5.417 and ignore@7.0.7, both published
inside its 24h cutoff. My local install had no such policy, so the
lockfile resolved to them and the failure only surfaced on the runner.
Adds web/.npmrc pinning the same 1440-minute policy locally, so a
lockfile resolved here is one CI will accept, and re-resolves against
it (1.5.416 and 7.0.6). Setting it in the repo rather than fixing the
two versions by hand is what stops this recurring on the next
dependency change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
Three routes, per architecture/generic.md §4 (Vite + React + SWC + TS,
static, served by nginx): / renders the repository readme, /models is a
paged and filterable listing of what the bucket holds, and
/{namespace}/{name} shows one repository -- its pinned refs, the files
actually stored, and its model card.
The API it reads is /v1/, deliberately not /api/. That surface is the
Hub's, recorded and replayed verbatim, and adding routes of our own to
it risks a client mistaking one for the real thing. /v1/ answers a
question the Hub has no equivalent for -- what is in this bucket --
which is inventory, not the model search the spec rules out.
The listing shows only what has actually been fetched, never upstream's
siblings. A repository pulled one file at a time shows one file, which
is the honest answer to "what can I get from here offline".
Model cards are third-party content, so their HTML is parsed and then
sanitised against GitHub's allowlist. The plugin order is load-bearing
and commented as such. Verified against a card crafted with <script>,
onerror, a javascript: href, an <iframe>, an SVG-embedded script and an
inline handler: none execute and ordinary markdown still renders. Card
images are not loaded at all -- fetching them would leak the viewer's
address to a third party and make an offline registry's pages depend on
the internet.
Routing: rustingface's URL space is the Hub's, so /Qwen/Qwen3-0.6B is
both a page and the prefix of a file. The vhost splits them the way the
Hub does -- /resolve/ anywhere in the path, plus /api/ and /v1/, go to
the service; everything else is the app. A repo legitimately named
v1/repos is handed back to the resolve path by the router, and there is
a test for it. The /resolve/ test is on the repo type rather than the
path substring, because a repo may contain a directory called resolve
and /api/models/a/b/tree/main/resolve/f must stay a tree request.
Deployment: the frontend ships to hanzalova:/var/www/rustingface, so
that host now gets its own scoped gitea_ci drop-in -- narrower than the
service host's: a webroot rsync, a relabel, nginx -t and a reload. The
health check probes rf.internal from the proxy rather than from the
runner, because a runner is a plain Fedora container with no internal
root CA (verified: fedora:43 gets 000, the proxy gets 200).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
Guarantee 1 is "durably stored in the configured bucket before the
response completes". The tee sent every chunk straight through, so the
client saw the last byte before the multipart upload was completed and
well before the manifest referenced it. A client that read a file and
immediately failed over to a sealed instance could be told the file had
never been seen -- and if either storage phase failed, it had already
received a clean 200 over bytes nothing recorded.
The tee now runs one chunk behind and releases the final chunk only
after the upload completes and the manifest write lands. Content-Length
is already on the wire, so a client cannot treat the transfer as
complete without those bytes; a storage or manifest failure now ends the
body short instead, which is what the client detects and retries on.
Costs one chunk of extra latency at the end of a transfer and nothing
else: the withheld chunk is bounded by the upstream chunk size, and a
disconnected client still has its transfer finished without it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
A snapshot_download fans out across a repo's files, so several transfers
finish at once and each rewrites the same manifest document. The
read-merge-write had no conditional put behind it: two writers that read
the same document both wrote their own version, and the second silently
dropped the first's entry. The verify-after-write retry caught most of
that, but the write happens in the detached transfer task -- after the
client already has its bytes -- so a lost entry surfaced only later, as a
sealed instance claiming never to have seen a file it had served. That
breaks the fetch-once retention guarantee.
Adds a compare-and-set pair to the Store port (get_versioned /
put_if_version, conditional on the etag last read) and uses it for
manifest updates, so a racing write is refused and retried rather than
merged away. The local filesystem store cannot do conditional updates,
so it reports supports_compare_and_set() = false and the caller keeps
the verify-after-write path for it.
Also serialises manifest writes per key within the process. CAS alone is
correct but turns eight parallel workers into eight writers contending
for one document, burning the retry budget on self-inflicted collisions;
the lock makes the common case uncontended and leaves the retry loop for
genuinely concurrent writers -- another instance, or the CLI against the
same bucket.
Regression test fetches twelve files of one repo concurrently and
asserts every one survives into the manifest and replays when sealed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
Scaffolds the workspace per architecture/generic.md §1 and implements
phases 0-3 of doc/spec.md §12.
Crates:
entities manifest/ref/repo schemas, bucket key layout, config, the
X-Error-Code taxonomy. No I/O.
core resolver, freeze pinning, single-flight, the streaming tee,
policy, gc/verify/refresh. Defines the Store and Upstream
ports.
data object_store (S3 + local) and reqwest Hub adapters.
api the axum surface: resolve, model/dataset info, tree, refs,
whoami, metrics, bearer auth, range handling.
bin one binary: serve plus fetch/pin/refresh/list/show/rm/gc/
verify/doctor.
Deployment targets bob.hanzalova.internal:20482 (port derived per
architecture/port-allocations.md §3), storing to the MinIO on
caveman.kosherinata.internal, fronted by hanzalova at rf.internal.
Ships the sysusers drop-in, hardened unit, firewalld service, nginx
vhost, config template, infra-setup.sh and the Gitea Actions
ci/deploy/conformance workflows.
Testing: 112 unit and integration tests, including the sovereignty
suite (cold fetch, sealed replay, single-flight, client disconnect,
range resume, freeze stability, gc-after-rm, digest mismatch), plus a
conformance suite driving a pinned huggingface_hub against a real Hub.
Deviations from the spec, all deliberate:
- one binary with subcommands (spec §9) rather than generic.md's
separate -api and -cli binaries; the library split is unchanged.
- a dedicated sysusers account and hardened unit (generic.md §8)
rather than the spec's illustrative DynamicUser unit.
- manifests carry an optional repo_tree recorded verbatim, resolving
spec §13's "record whole, filter on read" question for the tree
endpoint as well as model-info.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
