Commit Graph
5 Commits
Author SHA1 Message Date
grenadeandClaude Opus 5 9b35dfdc09 fix(core): a cold range that stops short of the end no longer stalls
ci / web (push) Successful in 1m36s
ci / check (push) Successful in 6m24s
Since #1 a ranged GET on a file the bucket does not hold streams its slice
from the tee instead of waiting for the whole transfer. For a range that
runs to the end of the file -- what a resumed `hf download` sends -- that
is exactly right, and it stays.

For a range that stops short of the end it is not enough. The slice
arrives quickly and then the *last chunk* is withheld until the manifest
write lands, which waits on every remaining byte, because the digest is
verified only once the whole file has passed. Measured against the live
service, asking for the first 1MiB of a 3.3GB shard:

    http=206 ttfb=1.040851s total=120.001638s bytes=1048169

1,048,169 of 1,048,576 bytes in about a second, then an idle connection
for the rest of the transfer. The client's read timeout ends it long
before, so in practice the request fails anyway -- slowly and with no
explanation.

The holdback is not the thing to change: releasing that tail early would
signal "durable and recorded" over bytes that are neither, which is
precisely what it exists to prevent. So answer 503 with Retry-After
instead, and let the fetch run on detached; the retry is served from the
bucket. `hf_transfer` splits every download into bounded ranged chunks, so
this shape is not hypothetical wherever it is enabled.

The existing ranged test now asserts the two-step behaviour and keeps its
guarantee that the bucket holds the whole file and never a fragment. A new
test pins that a resume to EOF is still served from the tee, so the fix for
#1 cannot be undone by this one.

Closes #4

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
2026-09-02 14:24:37 +03:00
grenadeandClaude Opus 5 b41964fae0 fix(serve): bound the drain so a restart cannot escalate to SIGABRT
ci / web (push) Successful in 1m24s
ci / check (push) Successful in 6m29s
`systemctl restart` during an in-flight blob transfer hung the drain until
`TimeoutStopSec` expired, at which point systemd escalated to SIGABRT and
the process core-dumped. Observed on bob during deploy 28:

    rustingface.service: Main process exited, code=dumped, status=6/ABRT
    rustingface.service: Failed with result 'timeout'.

axum's graceful shutdown waits for in-flight requests, and a blob response
*is* an in-flight request lasting size/bandwidth -- seven minutes for a
3.3GB shard at 8MB/s, against a 120s stop timeout. That wait cannot
converge, so the escalation was not an edge case but the normal outcome.

It also got more likely the moment #1 was fixed: a proxied blob previously
could not stream for longer than ~180s, so the size ceiling was doing the
job of a stop timeout by accident.

Cap the drain with `server.shutdown_grace` (15s) and exit cleanly when it
expires. Cutting a download deliberately is the better failure: the client
retries, which is a normal event, and nothing in the bucket depends on the
response, because a manifest entry is only written once its blob is
durable. TimeoutStopSec drops to 60s and is now only a backstop.

Verified against the live bucket with a transfer in flight: SIGTERM ->
clean exit 0 after exactly 15s, with a warning naming the reason. Before,
the same shape produced a two-minute stall and a core dump.

Closes #3

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
2026-09-02 14:16:50 +03:00
grenadeandClaude Opus 5 cfa53e14bd fix(data): a hidden 30s body timeout capped every proxied blob
`object_store` puts a 30-second `reqwest` timeout on the whole request,
body included. On a proxied blob that is not a timeout at all: the bucket
is read at the pace the client drains it, so it is a maximum servable
file size.

The reason it never looked like one is that object_store hides its own
consequences. It catches every response-body error and silently retries
with a resumed Range, so a large read does not fail at 30s -- it
reconnects every 30s and keeps going, until RetryConfig's 180s
retry_timeout is spent. Then the body simply ends, with a 200 already on
the wire and nothing logged anywhere. The ceiling is 180s x bandwidth: at
the 8MB/s a GPU host gets to caveman, no file over ~1.4GB can ever be
served, and no client retry converges because the next attempt is no
faster. Measured against the live bucket, 500MiB at 1MB/s: cut at 184.1s,
193,340,351 of 524,288,000 bytes. With the timeout disabled, 499.6s and
complete -- the same wall time curl takes reading the same object
straight from MinIO.

Disable it, and guard the stream the way a stream should be guarded:
`storage.connect_timeout` and an idle `storage.read_timeout` that bounds
silence rather than progress. A test asserts the total cap stays off.

Issue #1 named `server.request_timeout` as the cause. It was not -- that
key was parsed and then read by nothing, which is its own problem and is
why the hunt started 300 seconds away from a 180-second bug. Remove it.

Also log when a blob response ends before its Content-Length. Once the
headers are out the bytes leave through hyper, so a cut mid-body was
invisible on this side: the service logged a clean 200 while the client
saw a short read and retried forever.

Refs #1

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
2026-09-02 13:03:34 +03:00
grenadeandClaude Opus 5 3790f2589c fix(core): the per-key manifest lock did not actually exclude
deploy / build (push) Waiting to run
deploy / build-web (push) Failing after 55s
deploy / deploy (push) Has been cancelled
deploy / deploy-web (push) Has been cancelled
The lock added alongside compare-and-set removed its map entry inside
the critical section. A queued waiter still held that mutex, so the next
arrival found no entry, created a fresh one, and the two then ran
concurrently -- defeating the lock exactly when there was contention,
which is the only time it does anything.

I previously reported this fix as working on the strength of ten clean
release-mode runs. That was wrong: in debug, where the timing differs,
the guarding test fails 3 times in 12. Retiring the entry only after
dropping our own reference, under the same shard lock `entry` takes,
gives 0 failures in 25 -- and restoring the old behaviour reproduces 3
in 12, so the test really does catch it.

The consequence was the one the test names: a file served to a client
but never recorded, and so invisible the moment the instance is sealed.
Real against the local filesystem store, which has no conditional put
and relies on the verify-after-write retry; MinIO's compare-and-set
covers the same race, which is why the deployed path never showed it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
2026-08-31 13:03:55 +03:00
grenadeandClaude Opus 5 cb1ca8b6af feat(rustingface): implement the registry, admin CLI and deployment
deploy / build (push) Waiting to run
deploy / deploy (push) Has been cancelled
Scaffolds the workspace per architecture/generic.md §1 and implements
phases 0-3 of doc/spec.md §12.

Crates:
  entities  manifest/ref/repo schemas, bucket key layout, config, the
            X-Error-Code taxonomy. No I/O.
  core      resolver, freeze pinning, single-flight, the streaming tee,
            policy, gc/verify/refresh. Defines the Store and Upstream
            ports.
  data      object_store (S3 + local) and reqwest Hub adapters.
  api       the axum surface: resolve, model/dataset info, tree, refs,
            whoami, metrics, bearer auth, range handling.
  bin       one binary: serve plus fetch/pin/refresh/list/show/rm/gc/
            verify/doctor.

Deployment targets bob.hanzalova.internal:20482 (port derived per
architecture/port-allocations.md §3), storing to the MinIO on
caveman.kosherinata.internal, fronted by hanzalova at rf.internal.
Ships the sysusers drop-in, hardened unit, firewalld service, nginx
vhost, config template, infra-setup.sh and the Gitea Actions
ci/deploy/conformance workflows.

Testing: 112 unit and integration tests, including the sovereignty
suite (cold fetch, sealed replay, single-flight, client disconnect,
range resume, freeze stability, gc-after-rm, digest mismatch), plus a
conformance suite driving a pinned huggingface_hub against a real Hub.

Deviations from the spec, all deliberate:
  - one binary with subcommands (spec §9) rather than generic.md's
    separate -api and -cli binaries; the library split is unchanged.
  - a dedicated sysusers account and hardened unit (generic.md §8)
    rather than the spec's illustrative DynamicUser unit.
  - manifests carry an optional repo_tree recorded verbatim, resolving
    spec §13's "record whole, filter on read" question for the tree
    endpoint as well as model-info.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
2026-08-31 10:46:25 +03:00