rob thijssen
b41964fae0
`systemctl restart` during an in-flight blob transfer hung the drain until
`TimeoutStopSec` expired, at which point systemd escalated to SIGABRT and
the process core-dumped. Observed on bob during deploy 28:
rustingface.service: Main process exited, code=dumped, status=6/ABRT
rustingface.service: Failed with result 'timeout'.
axum's graceful shutdown waits for in-flight requests, and a blob response
*is* an in-flight request lasting size/bandwidth -- seven minutes for a
3.3GB shard at 8MB/s, against a 120s stop timeout. That wait cannot
converge, so the escalation was not an edge case but the normal outcome.
It also got more likely the moment #1 was fixed: a proxied blob previously
could not stream for longer than ~180s, so the size ceiling was doing the
job of a stop timeout by accident.
Cap the drain with `server.shutdown_grace` (15s) and exit cleanly when it
expires. Cutting a download deliberately is the better failure: the client
retries, which is a normal event, and nothing in the bucket depends on the
response, because a manifest entry is only written once its blob is
durable. TimeoutStopSec drops to 60s and is now only a backstop.
Verified against the live bucket with a transfer in flight: SIGTERM ->
clean exit 0 after exactly 15s, with a warning naming the reason. Before,
the same shape produced a two-minute stall and a core dump.
Closes #3
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqNtYNhov3fukx46KS9R7L
63 lines
2.1 KiB
Desktop File
63 lines
2.1 KiB
Desktop File
[Unit]
|
|
Description=rustingface sovereign model registry
|
|
Documentation=https://git.lair.cafe/grenade/rustingface
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
# Type=exec, not notify: the binary sends no sd_notify, and a notify unit that
|
|
# never sends READY=1 blocks `systemctl restart` until TimeoutStartSec.
|
|
Type=exec
|
|
User=rustingface
|
|
Group=rustingface
|
|
ExecStart=/usr/local/bin/rustingface serve --config /etc/rustingface/config.toml
|
|
Restart=on-failure
|
|
RestartSec=5s
|
|
|
|
# Credentials are passed by systemd rather than read from disk by the service,
|
|
# so the secret files stay root-owned and never appear in the unit's namespace
|
|
# except under $CREDENTIALS_DIRECTORY. The config names these basenames.
|
|
LoadCredential=s3-access-key:/etc/rustingface/s3-access-key
|
|
LoadCredential=s3-secret-key:/etc/rustingface/s3-secret-key
|
|
LoadCredential=hf-token:/etc/rustingface/hf-token
|
|
|
|
# A backstop, not the mechanism. The service caps its own drain at
|
|
# server.shutdown_grace and then exits cleanly, because a blob response lasts
|
|
# size/bandwidth and waiting for one to finish does not converge -- letting this
|
|
# expire instead means SIGABRT and a core dump. Keep it comfortably above the
|
|
# configured grace so systemd never escalates first.
|
|
TimeoutStopSec=60
|
|
|
|
# Hardening. The process holds no durable state — the bucket is the whole of it
|
|
# — so there is no StateDirectory to protect and ProtectSystem can stay strict
|
|
# with no ReadWritePaths at all.
|
|
NoNewPrivileges=true
|
|
ProtectSystem=strict
|
|
ProtectHome=true
|
|
PrivateTmp=true
|
|
PrivateDevices=true
|
|
ProtectKernelTunables=true
|
|
ProtectKernelModules=true
|
|
ProtectControlGroups=true
|
|
ProtectClock=true
|
|
ProtectHostname=true
|
|
ProtectProc=invisible
|
|
RestrictRealtime=true
|
|
RestrictSUIDSGID=true
|
|
RestrictNamespaces=true
|
|
LockPersonality=true
|
|
MemoryDenyWriteExecute=true
|
|
SystemCallFilter=@system-service
|
|
SystemCallArchitectures=native
|
|
CapabilityBoundingSet=
|
|
AmbientCapabilities=
|
|
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
|
|
UMask=0077
|
|
|
|
# Concurrent multi-gigabyte transfers each hold a handful of sockets and part
|
|
# buffers; the default 1024 is comfortable but the headroom costs nothing.
|
|
LimitNOFILE=16384
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|