b41964fae03cf2c003d5cfba632e285741b97483
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  grenadeandClaude Opus 5
|
b41964fae0
|
fix(serve): bound the drain so a restart cannot escalate to SIGABRT
`systemctl restart` during an in-flight blob transfer hung the drain until
`TimeoutStopSec` expired, at which point systemd escalated to SIGABRT and
the process core-dumped. Observed on bob during deploy 28:
rustingface.service: Main process exited, code=dumped, status=6/ABRT
rustingface.service: Failed with result 'timeout'.
axum's graceful shutdown waits for in-flight requests, and a blob response
*is* an in-flight request lasting size/bandwidth -- seven minutes for a
3.3GB shard at 8MB/s, against a 120s stop timeout. That wait cannot
converge, so the escalation was not an edge case but the normal outcome.
It also got more likely the moment #1 was fixed: a proxied blob previously
could not stream for longer than ~180s, so the size ceiling was doing the
job of a stop timeout by accident.
Cap the drain with `server.shutdown_grace` (15s) and exit cleanly when it
expires. Cutting a download deliberately is the better failure: the client
retries, which is a normal event, and nothing in the bucket depends on the
response, because a manifest entry is only written once its blob is
durable. TimeoutStopSec drops to 60s and is now only a backstop.
Verified against the live bucket with a transfer in flight: SIGTERM ->
clean exit 0 after exactly 15s, with a warning naming the reason. Before,
the same shape produced a two-minute stall and a core dump.
Closes #3
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
|
||
|
grenadeandClaude Opus 5
|
5f91032e6f
|
fix(core): a ranged fetch of a cold file left the client in silence
A ranged GET on a file the bucket does not hold used to fetch and store the whole thing before sending a single byte. Asking for the first megabyte of a 3.3GB shard therefore held the connection open, silent, for the length of the entire transfer: 93 seconds measured against the live service at 28MB/s, and 400-800s at the 8MB/s a GPU host gets. No client waits that long for a first byte. What makes it unrecoverable rather than merely slow is that `huggingface_hub` resumes with a Range header. So the first failure leaves an `.incomplete` file, and *every retry from then on* takes this path and is answered with silence. The partial file never grows, its mtime goes stale, and the retry loop cannot converge -- while the server logs nothing but `stored blob and recorded manifest entry`, because from its side the fetch succeeded. The whole file must still be fetched and stored, since a fragment must never reach the bucket. What must not happen is the client waiting on that in silence. So narrow the tee instead of blocking on it: the client is sent its slice as those bytes pass, one chunk behind as before, with the last piece still withheld until the manifest write lands. For the open-ended `bytes=N-` a resume actually asks for, that is a continuous stream to the end of the file. `blocking` and `LeadResult::Completed` go with it; nothing else used them. The test asserting a ranged cold request stores the whole file and never a fragment is unchanged and still passes. Refs #1 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L |
||
|
grenadeandClaude Opus 5
|
cfa53e14bd
|
fix(data): a hidden 30s body timeout capped every proxied blob
`object_store` puts a 30-second `reqwest` timeout on the whole request, body included. On a proxied blob that is not a timeout at all: the bucket is read at the pace the client drains it, so it is a maximum servable file size. The reason it never looked like one is that object_store hides its own consequences. It catches every response-body error and silently retries with a resumed Range, so a large read does not fail at 30s -- it reconnects every 30s and keeps going, until RetryConfig's 180s retry_timeout is spent. Then the body simply ends, with a 200 already on the wire and nothing logged anywhere. The ceiling is 180s x bandwidth: at the 8MB/s a GPU host gets to caveman, no file over ~1.4GB can ever be served, and no client retry converges because the next attempt is no faster. Measured against the live bucket, 500MiB at 1MB/s: cut at 184.1s, 193,340,351 of 524,288,000 bytes. With the timeout disabled, 499.6s and complete -- the same wall time curl takes reading the same object straight from MinIO. Disable it, and guard the stream the way a stream should be guarded: `storage.connect_timeout` and an idle `storage.read_timeout` that bounds silence rather than progress. A test asserts the total cap stays off. Issue #1 named `server.request_timeout` as the cause. It was not -- that key was parsed and then read by nothing, which is its own problem and is why the hunt started 300 seconds away from a 180-second bug. Remove it. Also log when a blob response ends before its Content-Length. Once the headers are out the bytes leave through hyper, so a cut mid-body was invisible on this side: the service logged a clean 200 while the client saw a short read and retried forever. Refs #1 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L |
||
|
grenadeandClaude Opus 5
|
3790f2589c
|
fix(core): the per-key manifest lock did not actually exclude
The lock added alongside compare-and-set removed its map entry inside the critical section. A queued waiter still held that mutex, so the next arrival found no entry, created a fresh one, and the two then ran concurrently -- defeating the lock exactly when there was contention, which is the only time it does anything. I previously reported this fix as working on the strength of ten clean release-mode runs. That was wrong: in debug, where the timing differs, the guarding test fails 3 times in 12. Retiring the entry only after dropping our own reference, under the same shard lock `entry` takes, gives 0 failures in 25 -- and restoring the old behaviour reproduces 3 in 12, so the test really does catch it. The consequence was the one the test names: a file served to a client but never recorded, and so invisible the moment the instance is sealed. Real against the local filesystem store, which has no conditional put and relies on the verify-after-write retry; MinIO's compare-and-set covers the same race, which is why the deployed path never showed it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64 |
||
|
grenadeandClaude Opus 5
|
5a94d90b79
|
fix(ci): restore the format gate and match the runner's package policy
Run 12 failed both build jobs, for two unrelated reasons of mine. cargo fmt: the previous commit ran clippy and the test suite but not the formatter, and inventory.rs was written by hand. Formatted. pnpm: the runner enforces a minimum release age on lockfile entries -- a supply-chain safeguard against a package published minutes ago -- and rejected electron-to-chromium@1.5.417 and ignore@7.0.7, both published inside its 24h cutoff. My local install had no such policy, so the lockfile resolved to them and the failure only surfaced on the runner. Adds web/.npmrc pinning the same 1440-minute policy locally, so a lockfile resolved here is one CI will accept, and re-resolves against it (1.5.416 and 7.0.6). Setting it in the repo rather than fixing the two versions by hand is what stops this recurring on the next dependency change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64 |
||
|
grenadeandClaude Opus 5
|
acd47a31c8
|
feat(web): add the operator UI and the /v1 inventory API it reads
Three routes, per architecture/generic.md §4 (Vite + React + SWC + TS,
static, served by nginx): / renders the repository readme, /models is a
paged and filterable listing of what the bucket holds, and
/{namespace}/{name} shows one repository -- its pinned refs, the files
actually stored, and its model card.
The API it reads is /v1/, deliberately not /api/. That surface is the
Hub's, recorded and replayed verbatim, and adding routes of our own to
it risks a client mistaking one for the real thing. /v1/ answers a
question the Hub has no equivalent for -- what is in this bucket --
which is inventory, not the model search the spec rules out.
The listing shows only what has actually been fetched, never upstream's
siblings. A repository pulled one file at a time shows one file, which
is the honest answer to "what can I get from here offline".
Model cards are third-party content, so their HTML is parsed and then
sanitised against GitHub's allowlist. The plugin order is load-bearing
and commented as such. Verified against a card crafted with <script>,
onerror, a javascript: href, an <iframe>, an SVG-embedded script and an
inline handler: none execute and ordinary markdown still renders. Card
images are not loaded at all -- fetching them would leak the viewer's
address to a third party and make an offline registry's pages depend on
the internet.
Routing: rustingface's URL space is the Hub's, so /Qwen/Qwen3-0.6B is
both a page and the prefix of a file. The vhost splits them the way the
Hub does -- /resolve/ anywhere in the path, plus /api/ and /v1/, go to
the service; everything else is the app. A repo legitimately named
v1/repos is handed back to the resolve path by the router, and there is
a test for it. The /resolve/ test is on the repo type rather than the
path substring, because a repo may contain a directory called resolve
and /api/models/a/b/tree/main/resolve/f must stay a tree request.
Deployment: the frontend ships to hanzalova:/var/www/rustingface, so
that host now gets its own scoped gitea_ci drop-in -- narrower than the
service host's: a webroot rsync, a relabel, nginx -t and a reload. The
health check probes rf.internal from the proxy rather than from the
runner, because a runner is a plain Fedora container with no internal
root CA (verified: fedora:43 gets 000, the proxy gets 200).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
|
||
|
grenadeandClaude Opus 5
|
010a8bbc0e
|
fix(bin): make observability.log_format actually take effect
The bootstrap subscriber installed before the config was loaded won the race for tracing's global subscriber, which can only be set once. The configured subscriber's try_init() was therefore always a silent no-op and observability.log_format was inert -- the deployed service, with log_format = "json", was writing ANSI-coloured text into the journal. The escape codes land in journald as literal bytes. Loads the configuration first and installs the subscriber once, since the configuration is what decides the format. Nothing is lost: a config load failure leaves main as an anyhow error with its context on stderr, which is where it was going anyway. Also disables ANSI unless stderr is a terminal. tracing_subscriber colours by default regardless of the writer, so escape codes were going into the journal and would go into any redirected log file. Verified: json emits one parseable object per line with no escapes, text emits plain text, and JOURNAL_STREAM still forces json for a service started by systemd whatever the file says. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64 |
||
|
grenadeandClaude Opus 5
|
4f72e4bfa6
|
feat(bin): flag serving gated weights without client auth
An upstream token is what makes gated repositories fetchable, and a gated repository's licence is between the operator whose token accepted the terms and its publisher. Serving those weights to anyone who can reach the port is redistribution, which most such licences forbid -- and the design rests on the stored copy being for the operator's own use (spec §7). Nothing said so when upstream.token_file was set and auth.mode was "none", which is the shipped default. Warns at startup and reports an advisory from `doctor`. Checks that the token file has content rather than merely existing, because the deploy renders it whether or not the secret behind it was set. Advisories are a new, non-failing tier in `doctor`. It runs in the deploy, so promoting this to an error would block a deployment on a question that is the operator's to answer: whether mesh-only reach already counts as own use is a licensing judgement, not something this binary should decide. auth.mode stays explicit rather than being inferred from the presence of auth.token_file. Inference fails open -- a rendered-but-empty credential would silently disable authentication and the service would come up looking healthy -- whereas an explicit mode turns the same slip into a refusal to start. The redundancy is what makes losing either half detectable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64 |
||
|
grenadeandClaude Opus 5
|
2d2dd7152a
|
fix(bin): don't refuse to start over a credential the config never reads
The deployed config template names every credential path unconditionally so an operator can see where each one goes. `serve` then read auth.token_file whether or not bearer auth was on, so a deployment with auth.mode = "none" -- the shipped default -- failed at startup with "/etc/rustingface/client-tokens does not exist". The documentation was a landmine. Reads a credential only when the running configuration actually consults it: the client token file only under bearer auth, and the object-store keys only when there is an S3 endpoint rather than a local path. A credential that *is* needed and missing still fails loudly and by name; that is the behaviour worth keeping. Adds tests/startup.rs, which drives the real binary. The first case spawns `serve` rather than a subcommand that never touches the file -- running `list` there would have passed either way. Verified the test catches the bug: reverting the fix reproduces the exact production error. Observed on deploy run 8, which got the whole pipeline through sysusers, rsync, firewalld and restart before the service exited on this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64 |
||
|
grenadeandClaude Opus 5
|
aafe132779
|
fix(core): hold the last chunk until the blob and manifest are durable
Guarantee 1 is "durably stored in the configured bucket before the response completes". The tee sent every chunk straight through, so the client saw the last byte before the multipart upload was completed and well before the manifest referenced it. A client that read a file and immediately failed over to a sealed instance could be told the file had never been seen -- and if either storage phase failed, it had already received a clean 200 over bytes nothing recorded. The tee now runs one chunk behind and releases the final chunk only after the upload completes and the manifest write lands. Content-Length is already on the wire, so a client cannot treat the transfer as complete without those bytes; a storage or manifest failure now ends the body short instead, which is what the client detects and retries on. Costs one chunk of extra latency at the end of a transfer and nothing else: the withheld chunk is bounded by the upstream chunk size, and a disconnected client still has its transfer finished without it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64 |
||
|
grenadeandClaude Opus 5
|
d461d466c2
|
fix(core): stop concurrent manifest writes from losing a served file
A snapshot_download fans out across a repo's files, so several transfers finish at once and each rewrites the same manifest document. The read-merge-write had no conditional put behind it: two writers that read the same document both wrote their own version, and the second silently dropped the first's entry. The verify-after-write retry caught most of that, but the write happens in the detached transfer task -- after the client already has its bytes -- so a lost entry surfaced only later, as a sealed instance claiming never to have seen a file it had served. That breaks the fetch-once retention guarantee. Adds a compare-and-set pair to the Store port (get_versioned / put_if_version, conditional on the etag last read) and uses it for manifest updates, so a racing write is refused and retried rather than merged away. The local filesystem store cannot do conditional updates, so it reports supports_compare_and_set() = false and the caller keeps the verify-after-write path for it. Also serialises manifest writes per key within the process. CAS alone is correct but turns eight parallel workers into eight writers contending for one document, burning the retry budget on self-inflicted collisions; the lock makes the common case uncontended and leaves the retry loop for genuinely concurrent writers -- another instance, or the CLI against the same bucket. Regression test fetches twelve files of one repo concurrently and asserts every one survives into the manifest and replays when sealed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64 |
||
|
grenadeandClaude Opus 5
|
1fe721e651
|
fix(data): record the Hub's headers, not the CDN's, across a redirect
The Hub answers a resolve request with a redirect: 307 to its own resolve-cache for an ordinary file, 302 to a CDN for an LFS-backed one. X-Repo-Commit, X-Linked-Etag and X-Linked-Size ride on that redirect response; the CDN it points at carries none of them and sets an ETag of its own. reqwest was following the redirect itself, so head_of() only ever saw the final hop. Every recorded etag was the CDN's, X-Linked-Etag and X-Linked-Size were lost, and no file was ever recognised as LFS -- which also meant expected_sha256() never fired, so nothing was verified against upstream's oid and every transfer took the staging path instead of writing straight to its content-addressed key. Follow redirects by hand instead, absorbing replayable metadata only while still on the Hub's origin and taking the content length from whichever hop actually serves the bytes. The operator's upstream token is dropped when a redirect crosses origins, since a presigned CDN URL needs no authorization of ours and forwarding one leaks a credential. Also derives the lfs flag from X-Linked-Size rather than the presence of X-Linked-Etag: the Hub sends a linked etag for every file (a git blob SHA-1 for ordinary ones), so the etag alone marked everything as LFS. Found by running the conformance suite against the MinIO on caveman. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64 |
||
|
grenadeandClaude Opus 5
|
cb1ca8b6af
|
feat(rustingface): implement the registry, admin CLI and deployment
Scaffolds the workspace per architecture/generic.md §1 and implements
phases 0-3 of doc/spec.md §12.
Crates:
entities manifest/ref/repo schemas, bucket key layout, config, the
X-Error-Code taxonomy. No I/O.
core resolver, freeze pinning, single-flight, the streaming tee,
policy, gc/verify/refresh. Defines the Store and Upstream
ports.
data object_store (S3 + local) and reqwest Hub adapters.
api the axum surface: resolve, model/dataset info, tree, refs,
whoami, metrics, bearer auth, range handling.
bin one binary: serve plus fetch/pin/refresh/list/show/rm/gc/
verify/doctor.
Deployment targets bob.hanzalova.internal:20482 (port derived per
architecture/port-allocations.md §3), storing to the MinIO on
caveman.kosherinata.internal, fronted by hanzalova at rf.internal.
Ships the sysusers drop-in, hardened unit, firewalld service, nginx
vhost, config template, infra-setup.sh and the Gitea Actions
ci/deploy/conformance workflows.
Testing: 112 unit and integration tests, including the sovereignty
suite (cold fetch, sealed replay, single-flight, client disconnect,
range resume, freeze stability, gc-after-rm, digest mismatch), plus a
conformance suite driving a pinned huggingface_hub against a real Hub.
Deviations from the spec, all deliberate:
- one binary with subcommands (spec §9) rather than generic.md's
separate -api and -cli binaries; the library split is unchanged.
- a dedicated sysusers account and hardened unit (generic.md §8)
rather than the spec's illustrative DynamicUser unit.
- manifests carry an optional repo_tree recorded verbatim, resolving
spec §13's "record whole, filter on read" question for the tree
endpoint as well as model-info.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
|