grenade
c9f7383d91
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m46s
CI / Format (push) Successful in 11s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 6m53s
CI / CUDA type-check (push) Successful in 19m6s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
beast served nothing for 70 minutes on 2026-08-17 and recovered only
because a human noticed. The unit already had Restart=on-failure and
RestartSec=5; the process simply never exited, so systemd saw a healthy
daemon — correctly, by its own rules — while the listener held 27
unaccepted connections, both GPUs sat at 0%, and clients got 503s.
The watchdog already diagnosed the remedy in its own log text ("recovery
may stall until a process restart") and then waited for someone to apply
it.
Exiting on that Err branch would not have helped. There is no
`ncclCommAbort succeeded` line and no `... failed` line in the incident:
the abort call itself blocked and never returned, taking its thread with
it. Control never reached either branch. So:
- the abort runs on its own named thread, where hanging costs one thread
rather than the daemon;
- an independent deadman waits NEURON_TP_ABORT_DEADLINE_S (default 60)
for it to return, and does NOT depend on it returning;
- if the deadline elapses, log the forensics and exit 70 (EX_SOFTWARE).
`wait_for_abort` is split out so the decision is testable without a GPU —
the branch that ends the process is both the most important to get right
and the hardest to reach by accident. Tests cover the abort returning,
never returning, and landing in the last instant (exiting on that race
would restart a process that had just recovered).
Forensics are logged BEFORE the exit, including a live nvidia-smi read
rather than the cached VRAM value, since the cache is refreshed by the
machinery that is wedged. A self-healing outage nobody can diagnose is a
poor trade for one that at least stayed put.
The unit gains StartLimitIntervalSec=600 / StartLimitBurst=4. Self-
restarting a deterministic wedge would otherwise flap, and a flapping
node is worse than a down one: cortex marks it healthy between restarts
and keeps routing to it, so every client sees intermittent failures
instead of one node cleanly out. Four failures in ten minutes leaves the
unit `failed` and visible.
process::exit skips destructors and the drain path deliberately —
normally the wrong instinct, but draining cannot work when the collective
is stuck, and TimeoutStopSec=120s would add two minutes to an outage that
is already total (#256).
Refs #265.