`systemctl restart` during an in-flight blob transfer hung the drain until
`TimeoutStopSec` expired, at which point systemd escalated to SIGABRT and
the process core-dumped. Observed on bob during deploy 28:
rustingface.service: Main process exited, code=dumped, status=6/ABRT
rustingface.service: Failed with result 'timeout'.
axum's graceful shutdown waits for in-flight requests, and a blob response
*is* an in-flight request lasting size/bandwidth -- seven minutes for a
3.3GB shard at 8MB/s, against a 120s stop timeout. That wait cannot
converge, so the escalation was not an edge case but the normal outcome.
It also got more likely the moment #1 was fixed: a proxied blob previously
could not stream for longer than ~180s, so the size ceiling was doing the
job of a stop timeout by accident.
Cap the drain with `server.shutdown_grace` (15s) and exit cleanly when it
expires. Cutting a download deliberately is the better failure: the client
retries, which is a normal event, and nothing in the bucket depends on the
response, because a manifest entry is only written once its blob is
durable. TimeoutStopSec drops to 60s and is now only a backstop.
Verified against the live bucket with a transfer in flight: SIGTERM ->
clean exit 0 after exactly 15s, with a warning naming the reason. Before,
the same shape produced a two-minute stall and a core dump.
Closes#3
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
Scaffolds the workspace per architecture/generic.md §1 and implements
phases 0-3 of doc/spec.md §12.
Crates:
entities manifest/ref/repo schemas, bucket key layout, config, the
X-Error-Code taxonomy. No I/O.
core resolver, freeze pinning, single-flight, the streaming tee,
policy, gc/verify/refresh. Defines the Store and Upstream
ports.
data object_store (S3 + local) and reqwest Hub adapters.
api the axum surface: resolve, model/dataset info, tree, refs,
whoami, metrics, bearer auth, range handling.
bin one binary: serve plus fetch/pin/refresh/list/show/rm/gc/
verify/doctor.
Deployment targets bob.hanzalova.internal:20482 (port derived per
architecture/port-allocations.md §3), storing to the MinIO on
caveman.kosherinata.internal, fronted by hanzalova at rf.internal.
Ships the sysusers drop-in, hardened unit, firewalld service, nginx
vhost, config template, infra-setup.sh and the Gitea Actions
ci/deploy/conformance workflows.
Testing: 112 unit and integration tests, including the sovereignty
suite (cold fetch, sealed replay, single-flight, client disconnect,
range resume, freeze stability, gc-after-rm, digest mismatch), plus a
conformance suite driving a pinned huggingface_hub against a real Hub.
Deviations from the spec, all deliberate:
- one binary with subcommands (spec §9) rather than generic.md's
separate -api and -cli binaries; the library split is unchanged.
- a dedicated sysusers account and hardened unit (generic.md §8)
rather than the spec's illustrative DynamicUser unit.
- manifests carry an optional repo_tree recorded verbatim, resolving
spec §13's "record whole, filter on read" question for the tree
endpoint as well as model-info.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
