rob thijssen
c9f7383d91
All checks were successful
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m46s
CI / Format (push) Successful in 11s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 6m53s
CI / CUDA type-check (push) Successful in 19m6s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
beast served nothing for 70 minutes on 2026-08-17 and recovered only
because a human noticed. The unit already had Restart=on-failure and
RestartSec=5; the process simply never exited, so systemd saw a healthy
daemon — correctly, by its own rules — while the listener held 27
unaccepted connections, both GPUs sat at 0%, and clients got 503s.
The watchdog already diagnosed the remedy in its own log text ("recovery
may stall until a process restart") and then waited for someone to apply
it.
Exiting on that Err branch would not have helped. There is no
`ncclCommAbort succeeded` line and no `... failed` line in the incident:
the abort call itself blocked and never returned, taking its thread with
it. Control never reached either branch. So:
- the abort runs on its own named thread, where hanging costs one thread
rather than the daemon;
- an independent deadman waits NEURON_TP_ABORT_DEADLINE_S (default 60)
for it to return, and does NOT depend on it returning;
- if the deadline elapses, log the forensics and exit 70 (EX_SOFTWARE).
`wait_for_abort` is split out so the decision is testable without a GPU —
the branch that ends the process is both the most important to get right
and the hardest to reach by accident. Tests cover the abort returning,
never returning, and landing in the last instant (exiting on that race
would restart a process that had just recovered).
Forensics are logged BEFORE the exit, including a live nvidia-smi read
rather than the cached VRAM value, since the cache is refreshed by the
machinery that is wedged. A self-healing outage nobody can diagnose is a
poor trade for one that at least stayed put.
The unit gains StartLimitIntervalSec=600 / StartLimitBurst=4. Self-
restarting a deterministic wedge would otherwise flap, and a flapping
node is worse than a down one: cortex marks it healthy between restarts
and keeps routing to it, so every client sees intermittent failures
instead of one node cleanly out. Four failures in ten minutes leaves the
unit `failed` and visible.
process::exit skips destructors and the drain path deliberately —
normally the wrong instinct, but draining cannot work when the collective
is stuck, and TimeoutStopSec=120s would add two minutes to an outage that
is already total (#256).
Refs #265.
53 lines
2.2 KiB
Desktop File
53 lines
2.2 KiB
Desktop File
[Unit]
|
|
Description=Neuron — per-node GPU discovery and harness daemon for cortex
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
# Start-limit bound for the self-restart path (#265). When a TP forward
|
|
# wedges and the NCCL abort that would unblock it also hangs, neuron
|
|
# exits 70 so Restart=on-failure recovers it in seconds instead of the
|
|
# 70-minute outage that needed a human on 2026-08-17.
|
|
#
|
|
# That only helps if the wedge is transient. A deterministic one — a bad
|
|
# build, a sick card, a model that wedges on load — would otherwise
|
|
# restart forever, and a flapping node is WORSE than a down one: cortex
|
|
# marks it healthy between restarts and keeps routing to it, so every
|
|
# client sees intermittent failures instead of one node cleanly out.
|
|
#
|
|
# Four failures in ten minutes stops the unit and leaves it `failed`,
|
|
# where `systemctl status` and an alert can see it. Visibly broken beats
|
|
# invisibly cycling.
|
|
StartLimitIntervalSec=600
|
|
StartLimitBurst=4
|
|
|
|
[Service]
|
|
Type=simple
|
|
ExecStart=/usr/bin/neuron --config /etc/neuron/neuron.toml
|
|
Restart=on-failure
|
|
RestartSec=5
|
|
User=neuron
|
|
Group=neuron
|
|
# /var/lib/neuron is the neuron user's $HOME — hf-hub writes its
|
|
# default cache there (~/.cache/huggingface/hub). Without this directive
|
|
# systemd doesn't create the directory and hf-hub downloads fail with
|
|
# "fetch GGUF <file>: failed to create cache dir".
|
|
StateDirectory=neuron
|
|
StateDirectoryMode=0755
|
|
# Loading default_models from neuron.toml happens before the HTTP
|
|
# listener binds; large models can take many minutes to download and
|
|
# materialise on first activation. systemd's default TimeoutStartSec
|
|
# (90s) is far too short; allow 30 minutes.
|
|
TimeoutStartSec=1800s
|
|
# On stop, neuron drains in-flight requests then unloads every model
|
|
# to release CUDA contexts cleanly. Allow generous time for big-model
|
|
# unloads; systemd will SIGKILL after this bound.
|
|
#
|
|
# Note this bound does NOT apply to the self-restart path (#265): a
|
|
# wedged process exits directly rather than draining, precisely because
|
|
# draining cannot work when the collective is stuck — and waiting this
|
|
# out would add two minutes to an outage that is already total.
|
|
TimeoutStopSec=120s
|
|
KillSignal=SIGTERM
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|