helexa/helexa
1
0
Fork 0
Code Issues 80 Pull Requests Actions Packages Projects Releases Wiki Activity

VRAM backstop rejects before admission, killing long agentic sessions — admission counts requests, capacity is KV bytes #257

New Issue
Open
opened 2026-08-15 12:43:42 +00:00 by grenade · 3 comments
grenade commented 2026-08-15 12:43:42 +00:00
Owner
Copy Link

Killed a 30-minute, 26-step dsh agentic session on 2026-08-15 with two
503s 1.2 s apart. The harness retries twice and gives up, so the entire
session was discarded with no conclusion surfaced to the user.

What happened

Session on Qwen/Qwen3.8-27B (beast, TP-2), context grown to 67,881
tokens over 26 tool-calling turns, two turns concurrently in flight.

12:25:15  inbound  /v1/responses  (turn N, ~5 min, no stream end logged)
12:30:16  inbound → 503 in 225 ms
12:30:17  inbound → 503 in 222 ms      ← dsh retry #2, gives up

neuron logged only TP chat_completion (stream): starting prompt_len=67881 max_new=8192 … vram_free_mb=4835 and then rejected
~7 ms later, logging nothing. cortex does not record upstream error
bodies, so the reason appears in neither journal. The guard was
identifiable only by reading the source and reproducing its arithmetic.

The rejection is arithmetically correct

check_vram's length-aware backstop (#65), candle.rs:1676:

tokens      = 67,881 prompt + 8,192 output_reserve_tokens  = 76,073
kv_mb       = 32,768 B/token/card × 76,073 / 1 MiB         =  2,377 MB
required_mb = 2,377 + 2,048 activation + 1,500 floor       =  5,925 MB
vram_free                                                  =  4,835 MB

Short by 1,090 MB. A third 68k sequence genuinely did not fit beside the
two already resident.

Why it is still a bug

  1. The check runs before admission control. In
    stream_tp_chat_completion, validate_request(...)? rejects at
    candle.rs:4737; tp.admission.enter(...) is not reached until
    :4749. max_queue_depth: 8 was entirely empty
    (queue_depth: 0, all rejected_* counters 0) — there was capacity
    to wait, and the code rejected instead of queueing.
  2. Reordering alone would not fix it. Admission grants on
    in_flight (2) < max_in_flight (8) and would pass the request
    straight through to the same failure. Admission bounds request
    count; capacity is bounded by KV bytes.     On long-context sessions
    those two disagree badly — 8 concurrent 68k sequences need ~19 GB of
    KV that does not exist.
  3. Retry-After: 5 is fiction. The code comments it as transient —
    "VRAM frees as the in-flight request(s) complete" — but those
    requests were 68k-token turns taking 70–150 s. 5 s was never enough,
    and dsh's two ~1 s retries were doomed before they were sent.
  4. Silent. No log line on the rejection path. Same failure shape as
    #253: a fast reject with nothing an operator can read.

Chosen fix (operator decision, 2026-08-15): VRAM-aware admission

Admission accounts for each admitted sequence's KV footprint, so a
request that does not fit waits in the existing queue until one that
does, rather than being rejected while the queue sits empty.

Sketch:

  • AdmissionController gains a KV budget as a second gate — a
    Semaphore whose permits are MB, acquired with acquire_many_owned
    (FIFO, so a large request is not starved by a stream of small ones).
  • Budget is captured at model load, when no sequences are running:
    budget_mb = free_tightest_mb − activation_headroom_mb − min_free_vram_mb. It must not be re-derived from live free VRAM,
    which already excludes in-flight KV and would double-count.
  • A request reserving more than the whole budget can never be
    admitted — that is the genuine "too long for this node" case and must
    fast-reject with a permanent error, not queue forever.
  • The permit holds both gates for the request's lifetime; dropping it
    frees both.

Also in scope, since they are what made this undiagnosable:

  • Log the rejection with the arithmetic (prompt_len, kv_mb,
    required_mb, free_mb).
  • Derive Retry-After from something real, or drop the claim of
    transience.

max_wait_secs default (30 s) is shorter than a single 150 s turn, so
the wait deadline for the VRAM gate needs its own consideration —
otherwise the queue merely converts a fast 503 into a slow one.

Killed a 30-minute, 26-step dsh agentic session on 2026-08-15 with two 503s 1.2 s apart. The harness retries twice and gives up, so the entire session was discarded with no conclusion surfaced to the user. ## What happened Session on `Qwen/Qwen3.8-27B` (beast, TP-2), context grown to 67,881 tokens over 26 tool-calling turns, two turns concurrently in flight. ``` 12:25:15 inbound /v1/responses (turn N, ~5 min, no stream end logged) 12:30:16 inbound → 503 in 225 ms 12:30:17 inbound → 503 in 222 ms ← dsh retry #2, gives up ``` neuron logged only `TP chat_completion (stream): starting prompt_len=67881 max_new=8192 … vram_free_mb=4835` and then rejected ~7 ms later, **logging nothing**. cortex does not record upstream error bodies, so the reason appears in neither journal. The guard was identifiable only by reading the source and reproducing its arithmetic. ## The rejection is arithmetically correct `check_vram`'s length-aware backstop (#65), `candle.rs:1676`: ``` tokens = 67,881 prompt + 8,192 output_reserve_tokens = 76,073 kv_mb = 32,768 B/token/card × 76,073 / 1 MiB = 2,377 MB required_mb = 2,377 + 2,048 activation + 1,500 floor = 5,925 MB vram_free = 4,835 MB ``` Short by 1,090 MB. A third 68k sequence genuinely did not fit beside the two already resident. ## Why it is still a bug 1. **The check runs before admission control.** In `stream_tp_chat_completion`, `validate_request(...)?` rejects at `candle.rs:4737`; `tp.admission.enter(...)` is not reached until `:4749`. `max_queue_depth: 8` was **entirely empty** (`queue_depth: 0`, all `rejected_*` counters 0) — there was capacity to wait, and the code rejected instead of queueing. 2. **Reordering alone would not fix it.** Admission grants on `in_flight (2) < max_in_flight (8)` and would pass the request straight through to the same failure. **Admission bounds request count; capacity is bounded by KV bytes.** On long-context sessions those two disagree badly — 8 concurrent 68k sequences need ~19 GB of KV that does not exist. 3. **`Retry-After: 5` is fiction.** The code comments it as transient — "VRAM frees as the in-flight request(s) complete" — but those requests were 68k-token turns taking 70–150 s. 5 s was never enough, and dsh's two ~1 s retries were doomed before they were sent. 4. **Silent.** No log line on the rejection path. Same failure shape as #253: a fast reject with nothing an operator can read. ## Chosen fix (operator decision, 2026-08-15): VRAM-aware admission Admission accounts for each admitted sequence's KV footprint, so a request that does not fit **waits in the existing queue** until one that does, rather than being rejected while the queue sits empty. Sketch: - `AdmissionController` gains a KV budget as a second gate — a `Semaphore` whose permits are MB, acquired with `acquire_many_owned` (FIFO, so a large request is not starved by a stream of small ones). - Budget is captured **at model load**, when no sequences are running: `budget_mb = free_tightest_mb − activation_headroom_mb − min_free_vram_mb`. It must not be re-derived from live free VRAM, which already excludes in-flight KV and would double-count. - A request reserving more than the **whole** budget can never be admitted — that is the genuine "too long for this node" case and must fast-reject with a permanent error, not queue forever. - The permit holds both gates for the request's lifetime; dropping it frees both. Also in scope, since they are what made this undiagnosable: - Log the rejection with the arithmetic (`prompt_len`, `kv_mb`, `required_mb`, `free_mb`). - Derive `Retry-After` from something real, or drop the claim of transience. `max_wait_secs` default (30 s) is shorter than a single 150 s turn, so the wait deadline for the VRAM gate needs its own consideration — otherwise the queue merely converts a fast 503 into a slow one.
grenade commented 2026-08-15 13:05:43 +00:00
Author
Owner
Copy Link

A caveat on the fix, found by watching beast at idle after the session
that prompted this issue. Recording it because it is what live
validation has to check.

Driver-visible free VRAM drifts badly from the load-time reading, and
asymmetrically across TP ranks.

At load, after a clean restart, the two cards were near-symmetric:

device 0   22824 MiB used   9286 free
device 1   22760 MiB used   9350 free

At idle after the session — in_flight: 0, queue_depth: 0, 0%
utilisation, temps back to 50 °C:

device 0   17770 MiB used  14340 free      pid 510897 (leader)   17744 MiB
device 1   28170 MiB used   3940 free      pid 510942 (rank 1)   28144 MiB

Rank 1 grew ~5.4 GiB and rank 0 shrank ~5 GiB, and neither returned at
idle. Consistent with cudarc allocator-pool retention: freed KV blocks
stay in the process's pool rather than going back to the driver, so
nvidia-smi counts them used while they remain reusable by the process.

Why this argues for the fix rather than against it. The old check
read live vram_free_mb and rejected against it — a number depressed
by pool retention, i.e. it was refusing requests over memory the process
could actually have reused. That is an additional, independent reason
the pre-admission backstop was the wrong instrument. The new gate tracks
reservations rather than driver readings, so pool retention cannot
corrupt its accounting.

Where it is a genuine risk. The budget is derived from
free_tightest_mb at load. Two assumptions behind that need live
confirmation:

  1. Retained pool memory is reusable for new KV. If it is, the budget
    is sound. If any of that ~5 GiB is genuinely unreachable rather than
    pooled, the budget over-admits by that much — the exact failure this
    issue is about.
  2. The tightest rank at load stays the tightest. It did not here:
    the ranks inverted over one session. A per-card budget derived from
    whichever card was tightest at load may be the wrong card later.

Neither is resolvable from outside the process. The check on deploy:
run a long-context session to saturation and confirm requests queue
(rejected_kv_timeout flat, kv_available_mb falling to near zero and
recovering) rather than failing a prefill. A prefill OOM under the new
gate would indict assumption 1.

A caveat on the fix, found by watching beast at idle after the session that prompted this issue. Recording it because it is what live validation has to check. **Driver-visible free VRAM drifts badly from the load-time reading, and asymmetrically across TP ranks.** At load, after a clean restart, the two cards were near-symmetric: ``` device 0 22824 MiB used 9286 free device 1 22760 MiB used 9350 free ``` At idle after the session — `in_flight: 0`, `queue_depth: 0`, 0% utilisation, temps back to 50 °C: ``` device 0 17770 MiB used 14340 free pid 510897 (leader) 17744 MiB device 1 28170 MiB used 3940 free pid 510942 (rank 1) 28144 MiB ``` Rank 1 grew ~5.4 GiB and rank 0 shrank ~5 GiB, and neither returned at idle. Consistent with cudarc allocator-pool retention: freed KV blocks stay in the process's pool rather than going back to the driver, so `nvidia-smi` counts them used while they remain reusable by the process. **Why this argues for the fix rather than against it.** The old check read *live* `vram_free_mb` and rejected against it — a number depressed by pool retention, i.e. it was refusing requests over memory the process could actually have reused. That is an additional, independent reason the pre-admission backstop was the wrong instrument. The new gate tracks reservations rather than driver readings, so pool retention cannot corrupt its accounting. **Where it is a genuine risk.** The budget is derived from `free_tightest_mb` at load. Two assumptions behind that need live confirmation: 1. **Retained pool memory is reusable for new KV.** If it is, the budget is sound. If any of that ~5 GiB is genuinely unreachable rather than pooled, the budget over-admits by that much — the exact failure this issue is about. 2. **The tightest rank at load stays the tightest.** It did not here: the ranks inverted over one session. A per-card budget derived from whichever card was tightest at load may be the wrong card later. Neither is resolvable from outside the process. The check on deploy: run a long-context session to saturation and confirm requests **queue** (`rejected_kv_timeout` flat, `kv_available_mb` falling to near zero and recovering) rather than failing a prefill. A prefill OOM under the new gate would indict assumption 1.
grenade commented 2026-08-15 15:50:46 +00:00
Author
Owner
Copy Link

Device fault under the workload this change admits — do not treat #257 as validated

Roughly an hour after adebbd42 deployed, on beast, during a sustained
long-context dsh session:

15:41:00  ERROR neuron::harness::engine: batch engine: device fault, model marked poisoned
          model=Qwen/Qwen3.8-27B error=TP assemble_kv_batch: AssembleKvBatch: leader assembly failed
15:41:02  WARN  auto-recovery: poisoned, enqueueing rebuild
15:41:02  WARN  auto-recovery: unload+reload starting

Auto-recovery worked and cortex passed no error to the client, so the
user-visible impact was nil. That is the only good news here.

Why this is probably ours

It landed ~7 minutes into sustained saturation with 2–3 concurrent
53–67k-token sequences — a regime this node had never sustained,
because the pre-admission backstop refused exactly those requests. The
gate admitted work that was previously rejected, and that work faulted
in KV assembly.

Two candidate causes, and the log cannot distinguish them:

  1. The budget over-admits. This is assumption 1 from the earlier
    comment: the budget is derived from free_tightest_mb at load, but
    ~5 GiB of driver-visible VRAM is held by the cudarc allocator pool.
    If that pool memory is not in fact reusable for new KV, the budget is
    too large by roughly that much and AssembleKvBatch is where it
    surfaces.
  2. A pre-existing batch-engine defect under multi-long-sequence
    load, previously unreachable because the backstop shed that load
    before it got here.

What blocks the diagnosis

AssembleKvBatch: leader assembly failed carries no underlying
cause — the device error is discarded on the poison path. Same failure
family as #253 (422 with no log) and the 503 that opened this issue: the
rejection/fault path produces nothing an operator can act on. Whatever
else happens, that error should propagate the CUDA status.

What was proven before the fault

Worth keeping, because it is independently useful:

  • Budget: free_at_load_mb=9285 → kv_budget_mb=4713.
  • Reservation exact: an 18,020-token prompt reserved 819 MiB
    ((18020+8192)×32 KiB) and returned it in full.
  • Under saturation: queue_depth=1, kv_available_mb=0, and the queued
    request was admitted when a sequence finished (QUEUE CLEARED, kv_available=2060). Requests queue and are served rather than 503ing.
  • Across the whole episode: rejected_kv_timeout=0,
    rejected_kv_unservable=0, rejected_queue_full=0, zero non-2xx.

So the mechanism is right and the queueing is real. What is now in doubt
is whether the budget is sized safely.

Options

  • Shrink the budget by the observed pool retention (~5 GiB) — crude, and
    would roughly halve long-context concurrency.
  • max_in_flight = 1 on beast as a stopgap: keeps the gate honest while
    removing concurrency as a variable.
  • Propagate the device error first and reproduce, so this is diagnosed
    rather than guessed at.

Operator decision pending. Until then #257 should be considered
deployed but unvalidated, not done.

## Device fault under the workload this change admits — do not treat #257 as validated Roughly an hour after `adebbd42` deployed, on beast, during a sustained long-context dsh session: ``` 15:41:00 ERROR neuron::harness::engine: batch engine: device fault, model marked poisoned model=Qwen/Qwen3.8-27B error=TP assemble_kv_batch: AssembleKvBatch: leader assembly failed 15:41:02 WARN auto-recovery: poisoned, enqueueing rebuild 15:41:02 WARN auto-recovery: unload+reload starting ``` Auto-recovery worked and cortex passed no error to the client, so the user-visible impact was nil. That is the only good news here. ### Why this is probably ours It landed ~7 minutes into **sustained saturation with 2–3 concurrent 53–67k-token sequences** — a regime this node had never sustained, because the pre-admission backstop refused exactly those requests. The gate admitted work that was previously rejected, and that work faulted in KV assembly. Two candidate causes, and the log cannot distinguish them: 1. **The budget over-admits.** This is assumption 1 from the earlier comment: the budget is derived from `free_tightest_mb` at load, but ~5 GiB of driver-visible VRAM is held by the cudarc allocator pool. If that pool memory is not in fact reusable for new KV, the budget is too large by roughly that much and `AssembleKvBatch` is where it surfaces. 2. **A pre-existing batch-engine defect** under multi-long-sequence load, previously unreachable because the backstop shed that load before it got here. ### What blocks the diagnosis `AssembleKvBatch: leader assembly failed` carries **no underlying cause** — the device error is discarded on the poison path. Same failure family as #253 (422 with no log) and the 503 that opened this issue: the rejection/fault path produces nothing an operator can act on. Whatever else happens, that error should propagate the CUDA status. ### What was proven before the fault Worth keeping, because it is independently useful: - Budget: `free_at_load_mb=9285 → kv_budget_mb=4713`. - Reservation exact: an 18,020-token prompt reserved 819 MiB (`(18020+8192)×32 KiB`) and returned it in full. - Under saturation: `queue_depth=1`, `kv_available_mb=0`, and the queued request was **admitted when a sequence finished** (`QUEUE CLEARED, kv_available=2060`). Requests queue and are served rather than 503ing. - Across the whole episode: `rejected_kv_timeout=0`, `rejected_kv_unservable=0`, `rejected_queue_full=0`, zero non-2xx. So the mechanism is right and the queueing is real. What is now in doubt is whether the budget is *sized* safely. ### Options - Shrink the budget by the observed pool retention (~5 GiB) — crude, and would roughly halve long-context concurrency. - `max_in_flight = 1` on beast as a stopgap: keeps the gate honest while removing concurrency as a variable. - Propagate the device error first and reproduce, so this is diagnosed rather than guessed at. Operator decision pending. Until then #257 should be considered **deployed but unvalidated**, not done.
grenade commented 2026-08-15 15:54:11 +00:00
Author
Owner
Copy Link

Auto-recovery from the fault completed on its own — status: loaded,
serving again, no intervention. Worth noting because it means the device
fault is survivable, not fatal to the node.

The rebuilt model published a different budget, which is a data point
against the current derivation:

original load:   free_at_load_mb 9285  →  kv_budget_mb 4713
after recovery:  free_at_load_mb ~9123 →  kv_budget_mb 4551

162 MiB lower for the same model on the same card, minutes apart. The
unload/reload did not return everything to the driver — the same
allocator-pool retention as the ~5 GiB drift noted earlier, just smaller.

The implication for the design: the budget is currently a function of
when the model happened to load, not a property of the hardware. Every
reload lands on a slightly different number, always drifting down, and
nothing re-derives it. A long-lived node that recovers a few times would
ratchet its own concurrency downward for no reason a user could see.

Worth considering as a fourth option alongside the three above: derive
the budget from total_vram − resident_weights − reserves rather than
from a driver free-VRAM reading at one instant. That is stable across
reloads and immune to pool retention, at the cost of needing the
resident weight size — which the loader knows.

Auto-recovery from the fault completed on its own — `status: loaded`, serving again, no intervention. Worth noting because it means the device fault is survivable, not fatal to the node. The rebuilt model published a **different budget**, which is a data point against the current derivation: ``` original load: free_at_load_mb 9285 → kv_budget_mb 4713 after recovery: free_at_load_mb ~9123 → kv_budget_mb 4551 ``` 162 MiB lower for the same model on the same card, minutes apart. The unload/reload did not return everything to the driver — the same allocator-pool retention as the ~5 GiB drift noted earlier, just smaller. The implication for the design: **the budget is currently a function of when the model happened to load, not a property of the hardware.** Every reload lands on a slightly different number, always drifting down, and nothing re-derives it. A long-lived node that recovers a few times would ratchet its own concurrency downward for no reason a user could see. Worth considering as a fourth option alongside the three above: derive the budget from `total_vram − resident_weights − reserves` rather than from a driver free-VRAM reading at one instant. That is stable across reloads and immune to pool retention, at the cost of needing the resident weight size — which the loader knows.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
out-of-scope
Outside helexa's constraining principles: near-frontier models on consumer hardware, served transparently and predictably
 p1-now
Active pain or unblocking work — pick up first
 p2-next
Engine levers — start once P1 baselines exist
 p3-later
Dependent on earlier work, or demand-driven
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
grenade
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: helexa/helexa#257
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?