`object_store` puts a 30-second `reqwest` timeout on the whole request,
body included. On a proxied blob that is not a timeout at all: the bucket
is read at the pace the client drains it, so it is a maximum servable
file size.
The reason it never looked like one is that object_store hides its own
consequences. It catches every response-body error and silently retries
with a resumed Range, so a large read does not fail at 30s -- it
reconnects every 30s and keeps going, until RetryConfig's 180s
retry_timeout is spent. Then the body simply ends, with a 200 already on
the wire and nothing logged anywhere. The ceiling is 180s x bandwidth: at
the 8MB/s a GPU host gets to caveman, no file over ~1.4GB can ever be
served, and no client retry converges because the next attempt is no
faster. Measured against the live bucket, 500MiB at 1MB/s: cut at 184.1s,
193,340,351 of 524,288,000 bytes. With the timeout disabled, 499.6s and
complete -- the same wall time curl takes reading the same object
straight from MinIO.
Disable it, and guard the stream the way a stream should be guarded:
`storage.connect_timeout` and an idle `storage.read_timeout` that bounds
silence rather than progress. A test asserts the total cap stays off.
Issue #1 named `server.request_timeout` as the cause. It was not -- that
key was parsed and then read by nothing, which is its own problem and is
why the hunt started 300 seconds away from a 180-second bug. Remove it.
Also log when a blob response ends before its Content-Length. Once the
headers are out the bytes leave through hyper, so a cut mid-body was
invisible on this side: the service logged a clean 200 while the client
saw a short read and retried forever.
Refs #1
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
A snapshot_download fans out across a repo's files, so several transfers
finish at once and each rewrites the same manifest document. The
read-merge-write had no conditional put behind it: two writers that read
the same document both wrote their own version, and the second silently
dropped the first's entry. The verify-after-write retry caught most of
that, but the write happens in the detached transfer task -- after the
client already has its bytes -- so a lost entry surfaced only later, as a
sealed instance claiming never to have seen a file it had served. That
breaks the fetch-once retention guarantee.
Adds a compare-and-set pair to the Store port (get_versioned /
put_if_version, conditional on the etag last read) and uses it for
manifest updates, so a racing write is refused and retried rather than
merged away. The local filesystem store cannot do conditional updates,
so it reports supports_compare_and_set() = false and the caller keeps
the verify-after-write path for it.
Also serialises manifest writes per key within the process. CAS alone is
correct but turns eight parallel workers into eight writers contending
for one document, burning the retry budget on self-inflicted collisions;
the lock makes the common case uncontended and leaves the retry loop for
genuinely concurrent writers -- another instance, or the CLI against the
same bucket.
Regression test fetches twelve files of one repo concurrently and
asserts every one survives into the manifest and replays when sealed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
The Hub answers a resolve request with a redirect: 307 to its own
resolve-cache for an ordinary file, 302 to a CDN for an LFS-backed one.
X-Repo-Commit, X-Linked-Etag and X-Linked-Size ride on that redirect
response; the CDN it points at carries none of them and sets an ETag of
its own.
reqwest was following the redirect itself, so head_of() only ever saw
the final hop. Every recorded etag was the CDN's, X-Linked-Etag and
X-Linked-Size were lost, and no file was ever recognised as LFS -- which
also meant expected_sha256() never fired, so nothing was verified
against upstream's oid and every transfer took the staging path instead
of writing straight to its content-addressed key.
Follow redirects by hand instead, absorbing replayable metadata only
while still on the Hub's origin and taking the content length from
whichever hop actually serves the bytes. The operator's upstream token
is dropped when a redirect crosses origins, since a presigned CDN URL
needs no authorization of ours and forwarding one leaks a credential.
Also derives the lfs flag from X-Linked-Size rather than the presence of
X-Linked-Etag: the Hub sends a linked etag for every file (a git blob
SHA-1 for ordinary ones), so the etag alone marked everything as LFS.
Found by running the conformance suite against the MinIO on caveman.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
Scaffolds the workspace per architecture/generic.md §1 and implements
phases 0-3 of doc/spec.md §12.
Crates:
entities manifest/ref/repo schemas, bucket key layout, config, the
X-Error-Code taxonomy. No I/O.
core resolver, freeze pinning, single-flight, the streaming tee,
policy, gc/verify/refresh. Defines the Store and Upstream
ports.
data object_store (S3 + local) and reqwest Hub adapters.
api the axum surface: resolve, model/dataset info, tree, refs,
whoami, metrics, bearer auth, range handling.
bin one binary: serve plus fetch/pin/refresh/list/show/rm/gc/
verify/doctor.
Deployment targets bob.hanzalova.internal:20482 (port derived per
architecture/port-allocations.md §3), storing to the MinIO on
caveman.kosherinata.internal, fronted by hanzalova at rf.internal.
Ships the sysusers drop-in, hardened unit, firewalld service, nginx
vhost, config template, infra-setup.sh and the Gitea Actions
ci/deploy/conformance workflows.
Testing: 112 unit and integration tests, including the sovereignty
suite (cold fetch, sealed replay, single-flight, client disconnect,
range resume, freeze stability, gc-after-rm, digest mismatch), plus a
conformance suite driving a pinned huggingface_hub against a real Hub.
Deviations from the spec, all deliberate:
- one binary with subcommands (spec §9) rather than generic.md's
separate -api and -cli binaries; the library split is unchanged.
- a dedicated sysusers account and hardened unit (generic.md §8)
rather than the spec's illustrative DynamicUser unit.
- manifests carry an optional repo_tree recorded verbatim, resolving
spec §13's "record whole, filter on read" question for the tree
endpoint as well as model-info.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
