A deploy during a long download kills the service with SIGABRT #3

New Issue

grenade · 2026-09-02T11:13:37Z

grenade commented

2026-09-02 11:13:37 +00:00

Summary

systemctl restart during an in-flight blob transfer hangs the drain until
TimeoutStopSec expires, at which point systemd escalates to SIGABRT and
the process core-dumps.

Observed on bob during deploy run 28, while a 500MiB read was in progress:

rustingface.service: Main process exited, code=dumped, status=6/ABRT
rustingface.service: Failed with result 'timeout'.
rustingface.service: Consumed 7min 24.264s CPU time, 490.1M memory peak.

The unit sets TimeoutStopSec=120.

Cause

shutdown() in crates/rustingface/src/cmd/serve.rs hands off to
axum::serve(...).with_graceful_shutdown(...), which waits for in-flight
requests to finish. A blob response is an in-flight request, and it lasts
size / bandwidth — 420s for a 3.3GB shard at 8MB/s, well past the
120-second stop timeout.

The docstring on shutdown() reasons about detached transfers not being
waited on, which is correct, but says nothing about client-facing streams,
which are what actually hold the drain open.

Why it matters more now

Before #1 was fixed, a proxied blob could not stream for longer than ~180
seconds — the ceiling was doing the job of a stop timeout by accident. With
that ceiling gone, transfers legitimately run for many minutes, so every
deploy that lands during one now risks this.

Nothing is lost from the bucket when it happens: the upload is a detached
task and a manifest entry is only written after the blob is durable, so the
worst case is an orphan blob that gc reclaims. The cost is that every
client mid-download is cut, and the service dies dirtily enough to write a
core dump.

Suggested direction

Either of, or both:

Let the drain stop waiting on blob response streams specifically, so a
restart cuts downloads deliberately and promptly rather than by timeout
escalation. Clients retry; that is a normal, recoverable event.
Raise TimeoutStopSec to something that reflects a real transfer, and
accept slower restarts.

The first is better: a bounded, intentional cut beats a two-minute stall
followed by SIGABRT either way, and it keeps deploys quick.

## Summary `systemctl restart` during an in-flight blob transfer hangs the drain until `TimeoutStopSec` expires, at which point systemd escalates to `SIGABRT` and the process core-dumps. Observed on bob during deploy run 28, while a 500MiB read was in progress: ``` rustingface.service: Main process exited, code=dumped, status=6/ABRT rustingface.service: Failed with result 'timeout'. rustingface.service: Consumed 7min 24.264s CPU time, 490.1M memory peak. ``` The unit sets `TimeoutStopSec=120`. ## Cause `shutdown()` in `crates/rustingface/src/cmd/serve.rs` hands off to `axum::serve(...).with_graceful_shutdown(...)`, which waits for in-flight requests to finish. A blob response *is* an in-flight request, and it lasts `size / bandwidth` — 420s for a 3.3GB shard at 8MB/s, well past the 120-second stop timeout. The docstring on `shutdown()` reasons about detached transfers not being waited on, which is correct, but says nothing about client-facing streams, which are what actually hold the drain open. ## Why it matters more now Before #1 was fixed, a proxied blob could not stream for longer than ~180 seconds — the ceiling was doing the job of a stop timeout by accident. With that ceiling gone, transfers legitimately run for many minutes, so every deploy that lands during one now risks this. Nothing is lost from the bucket when it happens: the upload is a detached task and a manifest entry is only written after the blob is durable, so the worst case is an orphan blob that `gc` reclaims. The cost is that every client mid-download is cut, and the service dies dirtily enough to write a core dump. ## Suggested direction Either of, or both: - Let the drain stop waiting on blob response streams specifically, so a restart cuts downloads deliberately and promptly rather than by timeout escalation. Clients retry; that is a normal, recoverable event. - Raise `TimeoutStopSec` to something that reflects a real transfer, and accept slower restarts. The first is better: a bounded, intentional cut beats a two-minute stall followed by `SIGABRT` either way, and it keeps deploys quick.

grenade referenced this issue from a commit

2026-09-02 11:16:55 +00:00

fix(serve): bound the drain so a restart cannot escalate to SIGABRT

grenade closed this issue

2026-09-02 11:24:53 +00:00

grenade referenced this issue from a commit

2026-09-02 11:24:54 +00:00

Merge branch 'fix/drain-does-not-wait-on-blob-streams'

Sign in to join this conversation.