main
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  grenadeandClaude Opus 5
|
14f8342cf8
|
fix(core): followers stream from the transfer instead of waiting it out
Spec §7 says concurrent requests for the same uncached blob must produce exactly one upstream fetch, and that "subsequent callers subscribe to the same broadcast". Subscribing was the part that did not exist. A follower called `Flight::wait` and blocked -- no timeout, nothing on the wire -- until the leader's entire transfer finished. For a 3.3GB shard at 8MB/s that is 400+ seconds before the response headers are written, against a client that allows 10. Indistinguishable from a hung server, while the leader logs that it stored the blob. It could not be fixed by handing over the live stream: the bytes that already passed are gone from it, and the multipart upload is not readable until it completes. So add the missing buffer. The leader writes every chunk to `server.spool_dir` as it passes and publishes how much is safe to read; a follower opens that file and streams it from byte zero, following it as it grows, with the last chunk withheld until the flight reports the blob stored -- the same rule the tee follows, for the same reason. The spool is not state, and the bucket remains the whole of it. Losing the spool loses nothing: a blob is not recorded until it is durable, so a crash mid-transfer leaves an unreferenced blob and the next request refetches. It is the role the in-memory buffer already plays for small files, on disk so it can hold a large one. Every transfer is spooled rather than only large ones, because size-dependent behaviour is what this service has already been bitten by once. With `spool_dir` unset there is nothing to read and a follower is answered 503 rather than left hanging, which is the honest fallback. The new test asserts the thing that actually distinguishes streaming from waiting -- that the follower's first bytes arrive while the leader is still transferring -- because chunked delivery alone proves nothing: the old path returned chunks too, just all of them after the transfer had ended. It fails with spooling disabled. The eight-concurrent-clients test still sees exactly one upstream fetch, now through the spool. Closes #2 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L |
||
|
grenadeandClaude Opus 5
|
b41964fae0
|
fix(serve): bound the drain so a restart cannot escalate to SIGABRT
`systemctl restart` during an in-flight blob transfer hung the drain until
`TimeoutStopSec` expired, at which point systemd escalated to SIGABRT and
the process core-dumped. Observed on bob during deploy 28:
rustingface.service: Main process exited, code=dumped, status=6/ABRT
rustingface.service: Failed with result 'timeout'.
axum's graceful shutdown waits for in-flight requests, and a blob response
*is* an in-flight request lasting size/bandwidth -- seven minutes for a
3.3GB shard at 8MB/s, against a 120s stop timeout. That wait cannot
converge, so the escalation was not an edge case but the normal outcome.
It also got more likely the moment #1 was fixed: a proxied blob previously
could not stream for longer than ~180s, so the size ceiling was doing the
job of a stop timeout by accident.
Cap the drain with `server.shutdown_grace` (15s) and exit cleanly when it
expires. Cutting a download deliberately is the better failure: the client
retries, which is a normal event, and nothing in the bucket depends on the
response, because a manifest entry is only written once its blob is
durable. TimeoutStopSec drops to 60s and is now only a backstop.
Verified against the live bucket with a transfer in flight: SIGTERM ->
clean exit 0 after exactly 15s, with a warning naming the reason. Before,
the same shape produced a two-minute stall and a core dump.
Closes #3
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
|
||
|
grenadeandClaude Opus 5
|
cfa53e14bd
|
fix(data): a hidden 30s body timeout capped every proxied blob
`object_store` puts a 30-second `reqwest` timeout on the whole request, body included. On a proxied blob that is not a timeout at all: the bucket is read at the pace the client drains it, so it is a maximum servable file size. The reason it never looked like one is that object_store hides its own consequences. It catches every response-body error and silently retries with a resumed Range, so a large read does not fail at 30s -- it reconnects every 30s and keeps going, until RetryConfig's 180s retry_timeout is spent. Then the body simply ends, with a 200 already on the wire and nothing logged anywhere. The ceiling is 180s x bandwidth: at the 8MB/s a GPU host gets to caveman, no file over ~1.4GB can ever be served, and no client retry converges because the next attempt is no faster. Measured against the live bucket, 500MiB at 1MB/s: cut at 184.1s, 193,340,351 of 524,288,000 bytes. With the timeout disabled, 499.6s and complete -- the same wall time curl takes reading the same object straight from MinIO. Disable it, and guard the stream the way a stream should be guarded: `storage.connect_timeout` and an idle `storage.read_timeout` that bounds silence rather than progress. A test asserts the total cap stays off. Issue #1 named `server.request_timeout` as the cause. It was not -- that key was parsed and then read by nothing, which is its own problem and is why the hunt started 300 seconds away from a 180-second bug. Remove it. Also log when a blob response ends before its Content-Length. Once the headers are out the bytes leave through hyper, so a cut mid-body was invisible on this side: the service logged a clean 200 while the client saw a short read and retried forever. Refs #1 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L |
||
|
grenade
|
4b359133b9
|
feat(deploy): mirror on demand for authenticated clients
Sets allow_new_repos true in the deployed configuration, so an authenticated request for a repository the bucket does not hold fetches it from upstream and stores it on the way past. This is what makes the registry useful as a drop-in HF_ENDPOINT for a consumer: helexa's neurons resolve eight catalogued models, none of which were in the bucket, and pre-seeding each one by hand before it can be served defeats the purpose of putting a mirror in front of the Hub at all. With the flag off the service answered exactly as designed -- "forbidden by policy: Qwen/Qwen3-1.7B is not already stored and policy.allow_new_repos is false" -- which is correct behaviour and the wrong deployment for this use. The trade is real and now stated where it will be read rather than buried: the bucket becomes bounded by auth rather than by policy, so every issued client token is a licence to store arbitrary upstream repositories. policy.allowlist is the middle setting that keeps on-demand mirroring while confining it to expected namespaces, and it stays empty here only because the consumer's catalogue is expected to grow. The agent notes previously asserted that the deployment relied on the flag being false; that claim would have been quietly wrong from this commit onward, so it is rewritten rather than left to mislead. |
||
|
grenadeandClaude Opus 5
|
3790f2589c
|
fix(core): the per-key manifest lock did not actually exclude
The lock added alongside compare-and-set removed its map entry inside the critical section. A queued waiter still held that mutex, so the next arrival found no entry, created a fresh one, and the two then ran concurrently -- defeating the lock exactly when there was contention, which is the only time it does anything. I previously reported this fix as working on the strength of ten clean release-mode runs. That was wrong: in debug, where the timing differs, the guarding test fails 3 times in 12. Retiring the entry only after dropping our own reference, under the same shard lock `entry` takes, gives 0 failures in 25 -- and restoring the old behaviour reproduces 3 in 12, so the test really does catch it. The consequence was the one the test names: a file served to a client but never recorded, and so invisible the moment the instance is sealed. Real against the local filesystem store, which has no conditional put and relies on the verify-after-write retry; MinIO's compare-and-set covers the same race, which is why the deployed path never showed it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64 |
||
|
grenadeandClaude Opus 5
|
d3be7937d5
|
fix(deploy): stop the placeholder guard tripping over its own comment
The render step guards against shipping a config with an unrendered
{{PLACEHOLDER}} by grepping for a bare "{{". The template's own header
comment explains the substitution syntax and contains that string, so
the guard matched line 3 of every successfully rendered config and
failed the deploy with "unrendered placeholder in the config" -- while
every actual placeholder had been substituted correctly.
Matches the placeholder shape ({{UPPER_SNAKE}}) instead, so prose about
the syntax is not a false positive, and rewores the template comment to
keep it clear of the guard either way.
Also whitelists the post-deploy `doctor` invocation in the scoped
sudoers drop-in. It runs as the service account rather than root -- the
point is to prove the credentials the service will actually use can
reach and write to the bucket -- and `sudo -u rustingface` was not
covered by any existing rule, so the step would have been refused.
Verified with `sudo -l -U gitea_ci`: the rule is listed, and `sudo -u
rustingface /bin/bash` remains denied.
Found by auditing every deploy step against the installed whitelist
after run 7 failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
|
||
|
grenadeandClaude Opus 5
|
cb1ca8b6af
|
feat(rustingface): implement the registry, admin CLI and deployment
Scaffolds the workspace per architecture/generic.md §1 and implements
phases 0-3 of doc/spec.md §12.
Crates:
entities manifest/ref/repo schemas, bucket key layout, config, the
X-Error-Code taxonomy. No I/O.
core resolver, freeze pinning, single-flight, the streaming tee,
policy, gc/verify/refresh. Defines the Store and Upstream
ports.
data object_store (S3 + local) and reqwest Hub adapters.
api the axum surface: resolve, model/dataset info, tree, refs,
whoami, metrics, bearer auth, range handling.
bin one binary: serve plus fetch/pin/refresh/list/show/rm/gc/
verify/doctor.
Deployment targets bob.hanzalova.internal:20482 (port derived per
architecture/port-allocations.md §3), storing to the MinIO on
caveman.kosherinata.internal, fronted by hanzalova at rf.internal.
Ships the sysusers drop-in, hardened unit, firewalld service, nginx
vhost, config template, infra-setup.sh and the Gitea Actions
ci/deploy/conformance workflows.
Testing: 112 unit and integration tests, including the sovereignty
suite (cold fetch, sealed replay, single-flight, client disconnect,
range resume, freeze stability, gc-after-rm, digest mismatch), plus a
conformance suite driving a pinned huggingface_hub against a real Hub.
Deviations from the spec, all deliberate:
- one binary with subcommands (spec §9) rather than generic.md's
separate -api and -cli binaries; the library split is unchanged.
- a dedicated sysusers account and hardened unit (generic.md §8)
rather than the spec's illustrative DynamicUser unit.
- manifests carry an optional repo_tree recorded verbatim, resolving
spec §13's "record whole, filter on read" question for the tree
endpoint as well as model-info.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
|