The bootstrap subscriber installed before the config was loaded won the
race for tracing's global subscriber, which can only be set once. The
configured subscriber's try_init() was therefore always a silent no-op
and observability.log_format was inert -- the deployed service, with
log_format = "json", was writing ANSI-coloured text into the journal.
The escape codes land in journald as literal bytes.
Loads the configuration first and installs the subscriber once, since
the configuration is what decides the format. Nothing is lost: a config
load failure leaves main as an anyhow error with its context on stderr,
which is where it was going anyway.
Also disables ANSI unless stderr is a terminal. tracing_subscriber
colours by default regardless of the writer, so escape codes were going
into the journal and would go into any redirected log file.
Verified: json emits one parseable object per line with no escapes,
text emits plain text, and JOURNAL_STREAM still forces json for a
service started by systemd whatever the file says.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
An upstream token is what makes gated repositories fetchable, and a
gated repository's licence is between the operator whose token accepted
the terms and its publisher. Serving those weights to anyone who can
reach the port is redistribution, which most such licences forbid --
and the design rests on the stored copy being for the operator's own
use (spec §7). Nothing said so when upstream.token_file was set and
auth.mode was "none", which is the shipped default.
Warns at startup and reports an advisory from `doctor`. Checks that the
token file has content rather than merely existing, because the deploy
renders it whether or not the secret behind it was set.
Advisories are a new, non-failing tier in `doctor`. It runs in the
deploy, so promoting this to an error would block a deployment on a
question that is the operator's to answer: whether mesh-only reach
already counts as own use is a licensing judgement, not something this
binary should decide.
auth.mode stays explicit rather than being inferred from the presence
of auth.token_file. Inference fails open -- a rendered-but-empty
credential would silently disable authentication and the service would
come up looking healthy -- whereas an explicit mode turns the same slip
into a refusal to start. The redundancy is what makes losing either
half detectable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
The deployed config template names every credential path unconditionally
so an operator can see where each one goes. `serve` then read
auth.token_file whether or not bearer auth was on, so a deployment with
auth.mode = "none" -- the shipped default -- failed at startup with
"/etc/rustingface/client-tokens does not exist". The documentation was a
landmine.
Reads a credential only when the running configuration actually consults
it: the client token file only under bearer auth, and the object-store
keys only when there is an S3 endpoint rather than a local path. A
credential that *is* needed and missing still fails loudly and by name;
that is the behaviour worth keeping.
Adds tests/startup.rs, which drives the real binary. The first case
spawns `serve` rather than a subcommand that never touches the file --
running `list` there would have passed either way. Verified the test
catches the bug: reverting the fix reproduces the exact production
error.
Observed on deploy run 8, which got the whole pipeline through sysusers,
rsync, firewalld and restart before the service exited on this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
Guarantee 1 is "durably stored in the configured bucket before the
response completes". The tee sent every chunk straight through, so the
client saw the last byte before the multipart upload was completed and
well before the manifest referenced it. A client that read a file and
immediately failed over to a sealed instance could be told the file had
never been seen -- and if either storage phase failed, it had already
received a clean 200 over bytes nothing recorded.
The tee now runs one chunk behind and releases the final chunk only
after the upload completes and the manifest write lands. Content-Length
is already on the wire, so a client cannot treat the transfer as
complete without those bytes; a storage or manifest failure now ends the
body short instead, which is what the client detects and retries on.
Costs one chunk of extra latency at the end of a transfer and nothing
else: the withheld chunk is bounded by the upstream chunk size, and a
disconnected client still has its transfer finished without it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
A snapshot_download fans out across a repo's files, so several transfers
finish at once and each rewrites the same manifest document. The
read-merge-write had no conditional put behind it: two writers that read
the same document both wrote their own version, and the second silently
dropped the first's entry. The verify-after-write retry caught most of
that, but the write happens in the detached transfer task -- after the
client already has its bytes -- so a lost entry surfaced only later, as a
sealed instance claiming never to have seen a file it had served. That
breaks the fetch-once retention guarantee.
Adds a compare-and-set pair to the Store port (get_versioned /
put_if_version, conditional on the etag last read) and uses it for
manifest updates, so a racing write is refused and retried rather than
merged away. The local filesystem store cannot do conditional updates,
so it reports supports_compare_and_set() = false and the caller keeps
the verify-after-write path for it.
Also serialises manifest writes per key within the process. CAS alone is
correct but turns eight parallel workers into eight writers contending
for one document, burning the retry budget on self-inflicted collisions;
the lock makes the common case uncontended and leaves the retry loop for
genuinely concurrent writers -- another instance, or the CLI against the
same bucket.
Regression test fetches twelve files of one repo concurrently and
asserts every one survives into the manifest and replays when sealed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
The Hub answers a resolve request with a redirect: 307 to its own
resolve-cache for an ordinary file, 302 to a CDN for an LFS-backed one.
X-Repo-Commit, X-Linked-Etag and X-Linked-Size ride on that redirect
response; the CDN it points at carries none of them and sets an ETag of
its own.
reqwest was following the redirect itself, so head_of() only ever saw
the final hop. Every recorded etag was the CDN's, X-Linked-Etag and
X-Linked-Size were lost, and no file was ever recognised as LFS -- which
also meant expected_sha256() never fired, so nothing was verified
against upstream's oid and every transfer took the staging path instead
of writing straight to its content-addressed key.
Follow redirects by hand instead, absorbing replayable metadata only
while still on the Hub's origin and taking the content length from
whichever hop actually serves the bytes. The operator's upstream token
is dropped when a redirect crosses origins, since a presigned CDN URL
needs no authorization of ours and forwarding one leaks a credential.
Also derives the lfs flag from X-Linked-Size rather than the presence of
X-Linked-Etag: the Hub sends a linked etag for every file (a git blob
SHA-1 for ordinary ones), so the etag alone marked everything as LFS.
Found by running the conformance suite against the MinIO on caveman.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
Scaffolds the workspace per architecture/generic.md §1 and implements
phases 0-3 of doc/spec.md §12.
Crates:
entities manifest/ref/repo schemas, bucket key layout, config, the
X-Error-Code taxonomy. No I/O.
core resolver, freeze pinning, single-flight, the streaming tee,
policy, gc/verify/refresh. Defines the Store and Upstream
ports.
data object_store (S3 + local) and reqwest Hub adapters.
api the axum surface: resolve, model/dataset info, tree, refs,
whoami, metrics, bearer auth, range handling.
bin one binary: serve plus fetch/pin/refresh/list/show/rm/gc/
verify/doctor.
Deployment targets bob.hanzalova.internal:20482 (port derived per
architecture/port-allocations.md §3), storing to the MinIO on
caveman.kosherinata.internal, fronted by hanzalova at rf.internal.
Ships the sysusers drop-in, hardened unit, firewalld service, nginx
vhost, config template, infra-setup.sh and the Gitea Actions
ci/deploy/conformance workflows.
Testing: 112 unit and integration tests, including the sovereignty
suite (cold fetch, sealed replay, single-flight, client disconnect,
range resume, freeze stability, gc-after-rm, digest mismatch), plus a
conformance suite driving a pinned huggingface_hub against a real Hub.
Deviations from the spec, all deliberate:
- one binary with subcommands (spec §9) rather than generic.md's
separate -api and -cli binaries; the library split is unchanged.
- a dedicated sysusers account and hardened unit (generic.md §8)
rather than the spec's illustrative DynamicUser unit.
- manifests carry an optional repo_tree recorded verbatim, resolving
spec §13's "record whole, filter on read" question for the tree
endpoint as well as model-info.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
grenadeandClaude Opus 5