helexa/helexa
1
0
Fork 0
Code Issues 104 Pull Requests Actions Packages Projects Releases Wiki Activity

beast reserves KV for eight tenants to serve one — max_in_flight is starving the single deep session it actually gets #291

New Issue
Open
opened 2026-08-26 08:28:09 +00:00 by grenade · 2 comments
grenade commented 2026-08-26 08:28:09 +00:00
Owner
Copy Link

Measured while asking why a 27B on 64 GB of RTX 5090 struggles for room
when the same model surpasses expectation on a single 24 GB card.

The capacity is there. It is being spent on breadth we do not use.

Observed concurrency: one request at a time, 90% of the time

Seven days of beast's journal, start/done paired by req_id (an earlier
unpaired count drifted to a nonsense peak of 322 — 312 starts never log a
done, from client_gone, errors, and the window edge). 1,397 paired
requests:

in_flight % of busy time cumulative
1 89.6% 89.6%
2 4.0% 93.6%
3–7 3.0% 96.6%
8 2.5% 99.1%
9 0.9% 100%

Utilisation: 4.9% — 8.3 busy hours in a week.

And most of the concurrency that exists is ours: helexa-bench's
concurrency:2/4/8 scenarios are synthetic bursts, and they produce the
4/8/9 spikes. Real multi-user demand on beast is close to nil.

(The peak of 9 against max_in_flight = 8 is the queue draining, not a
violation — but the effective ceiling is not exactly 8.)

What that costs the session we do get

47.1 GB used / 64.2 GB total        (17.1 GB free)
  ~42 GB   weights (q6k, TP=2, ~21 GB/card)
   4.7 GB  KV budget per card

At 32 KB/token/card that budget is 150,816 tokens, divided by
max_in_flight = 8 → 18,852 tokens per slot.

Meanwhile we advertise a 131,072-token context. Per card:

context KV, 1 slot KV, 8 slots
8,192 0.25 GB 2.0 GB
32,768 1.00 GB 8.0 GB
131,072 4.00 GB 32.0 GB

A card holds 31 GB and already carries ~21 GB of weights. The
advertised context and the concurrency setting are arithmetically
incompatible by roughly 7×. An agentic session gets ~19k tokens of KV
while seven idle slots hold reservations for callers who are not there.

This is also why #290's xhigh blowout was so easy to hit: a model
instructed to deliberate exhaustively, inside an output ceiling set
within a KV allocation sized for eight tenants.

Recommendation: max_in_flight = 2

setting covers per-slot KV
1 89.6% of busy time ~150k tokens (matches the advertised context)
2 93.6% ~75k tokens
8 (today) 99.1% 18.8k tokens

2 keeps the batch engine meaningful — measured at 1.79× at two
slots, and 5.24× at eight is throughput we are not being asked for —
gives the 27B a genuinely deep window, and leaves admission to queue the
~6% above two slots, which it already does gracefully with Retry-After
(#53).

Config-only, on a file CI now ships, and reversible.

Reference implementation worth mining

https://github.com/syv-ai/qwen38-27b-rtx3090 — vLLM settings for
this exact model on a single 24 GB RTX 3090. It already encodes the
single-vs-multi-user split this issue is proposing, and several of its
choices map onto open issues here:

  • Explicit profiles. single-user/start_qwen.sh (latency, 64k
    context, MTP-4 speculation, ~120–124 tok/s greedy) versus
    batch/start_qwen.sh (64 concurrent, ~1,035 tok/s aggregate, 8
    seats). Two named configurations, not one compromise.
  • Context modes are KV-dtype modes. CTX=fast (bf16, ~64k),
    CTX=long (fp8 via FlashInfer, 114–150k), CTX=huge (4/2-bit KVarN,
    240k+, lossy). That is #289 promoted from a tuning knob to a serving
    profile, and it is how they buy context rather than by adding VRAM.
  • MTP speculation as standard (SPEC=mtp, k=4). That is #79, where
    the draft head is already confirmed present in our deployed weights
    (849 MB, mtp_num_hidden_layers = 1).
  • A warning for the adaptive design below: with SPEC=dflash2 they
    measure 137 tok/s at 1 stream and 97 tok/s per-stream at 2, noting
    "the limit is the pool rather than MAX_SEQS". Slot count is not
    the only thing that binds, and an adaptive scheduler that only moves
    max_in_flight may find the pool binds first.

For scale: they get 120–124 tok/s single-user on one 3090; beast
measures 32 tok/s on two 5090s. Their weights are W4A16 with int8
lm_head against our q6k, and they speculate where we do not — so the gap
is configuration, not silicon. Same observation as the llama.cpp writeup
on #282.

The harder half: adapting at runtime

The obvious conclusion is that a single user should be able to use the
capacity idle tenants are reserving. One constraint has to be stated
before anyone designs it:

KV is planned at load. max_in_flight is both the admission bound
and the batch engine's slot count, and the KV budget is derived from it
when the model loads. So "use the spare capacity when nobody else is
here" is not a per-request knob — it implies re-planning the KV budget,
which today means a model reload (~5–6 min of 27B cold-load on beast).

Two shapes, in increasing cost:

  1. Admit fewer, deeper sessions and let admission queue the rest.
    Nearly free — it is the max_in_flight = 2 change above — and
    captures most of the benefit given the measured distribution. A third
    caller waits rather than being served a shallow window.
  2. Genuine runtime re-planning. Watch for sustained idleness, then
    re-plan KV for a deeper single session, and re-plan back under
    contention. Needs: an idleness signal that cannot thrash, a way to
    re-plan without dropping in-flight work, and an answer for what
    happens to the advertised limit.context when it changes underneath
    a client that has already read it. The advertised context changing
    mid-session is a client-visible contract change, not just an internal
    one.

(2) is the interesting version and should not be attempted before (1) is
measured — if max_in_flight = 2 restores the deep window, the
remaining upside from adaptivity is the 6% tail.

Suggested order

  1. Set max_in_flight = 2 on beast; re-measure the tower-defence run
    and helexa-bench concurrency:*. Cheap, reversible, testable.
  2. Re-run the concurrency measurement above afterwards, excluding
    bench    , so the distribution reflects users rather than our own
    instrument (#288's lesson).
  3. Then decide whether (2) is worth building, or whether KV
    quantisation (#289) and speculation (#79) return more for the effort.

Related

#290 (the reasoning work that surfaced this), #289 (KV quantisation),
#79 (MTP speculation — draft head confirmed in our weights), #282
(single-stream decode gap vs llama.cpp), #98 (the batch engine these
slots feed), #53/#54 (admission and fair-share), #257 (KV budget).

Measured while asking why a 27B on 64 GB of RTX 5090 struggles for room when the same model surpasses expectation on a single 24 GB card. The capacity is there. It is being spent on breadth we do not use. ## Observed concurrency: one request at a time, 90% of the time Seven days of beast's journal, start/done paired by `req_id` (an earlier unpaired count drifted to a nonsense peak of 322 — 312 starts never log a `done`, from `client_gone`, errors, and the window edge). 1,397 paired requests: | in_flight | % of busy time | cumulative | |---|---|---| | **1** | **89.6%** | 89.6% | | 2 | 4.0% | 93.6% | | 3–7 | 3.0% | 96.6% | | 8 | 2.5% | 99.1% | | 9 | 0.9% | 100% | **Utilisation: 4.9%** — 8.3 busy hours in a week. And most of the concurrency that exists is **ours**: helexa-bench's `concurrency:2/4/8` scenarios are synthetic bursts, and they produce the 4/8/9 spikes. Real multi-user demand on beast is close to nil. (The peak of 9 against `max_in_flight = 8` is the queue draining, not a violation — but the effective ceiling is not exactly 8.) ## What that costs the session we do get ``` 47.1 GB used / 64.2 GB total (17.1 GB free) ~42 GB weights (q6k, TP=2, ~21 GB/card) 4.7 GB KV budget per card ``` At 32 KB/token/card that budget is **150,816 tokens**, divided by `max_in_flight = 8` → **18,852 tokens per slot**. Meanwhile we advertise a **131,072-token context**. Per card: | context | KV, 1 slot | KV, 8 slots | |---|---|---| | 8,192 | 0.25 GB | 2.0 GB | | 32,768 | 1.00 GB | 8.0 GB | | 131,072 | **4.00 GB** | **32.0 GB** | A card holds 31 GB and already carries ~21 GB of weights. **The advertised context and the concurrency setting are arithmetically incompatible by roughly 7×.** An agentic session gets ~19k tokens of KV while seven idle slots hold reservations for callers who are not there. This is also why #290's `xhigh` blowout was so easy to hit: a model instructed to deliberate exhaustively, inside an output ceiling set within a KV allocation sized for eight tenants. ## Recommendation: `max_in_flight = 2` | setting | covers | per-slot KV | |---|---|---| | 1 | 89.6% of busy time | ~150k tokens (matches the advertised context) | | **2** | **93.6%** | **~75k tokens** | | 8 (today) | 99.1% | 18.8k tokens | `2` keeps the batch engine meaningful — measured at **1.79× at two slots**, and 5.24× at eight is throughput we are not being asked for — gives the 27B a genuinely deep window, and leaves admission to queue the ~6% above two slots, which it already does gracefully with `Retry-After` (#53). Config-only, on a file CI now ships, and reversible. ## Reference implementation worth mining **<https://github.com/syv-ai/qwen38-27b-rtx3090>** — vLLM settings for *this exact model* on a single 24 GB RTX 3090. It already encodes the single-vs-multi-user split this issue is proposing, and several of its choices map onto open issues here: - **Explicit profiles.** `single-user/start_qwen.sh` (latency, 64k context, MTP-4 speculation, ~120–124 tok/s greedy) versus `batch/start_qwen.sh` (64 concurrent, ~1,035 tok/s aggregate, 8 seats). Two named configurations, not one compromise. - **Context modes are KV-dtype modes.** `CTX=fast` (bf16, ~64k), `CTX=long` (fp8 via FlashInfer, 114–150k), `CTX=huge` (4/2-bit KVarN, 240k+, lossy). That is #289 promoted from a tuning knob to a serving profile, and it is how they buy context rather than by adding VRAM. - **MTP speculation as standard** (`SPEC=mtp`, k=4). That is #79, where the draft head is already confirmed present in our deployed weights (849 MB, `mtp_num_hidden_layers = 1`). - **A warning for the adaptive design below**: with `SPEC=dflash2` they measure 137 tok/s at 1 stream and 97 tok/s per-stream at 2, noting *"the limit is the pool rather than `MAX_SEQS`"*. Slot count is not the only thing that binds, and an adaptive scheduler that only moves `max_in_flight` may find the pool binds first. For scale: they get **120–124 tok/s single-user on one 3090**; beast measures **32 tok/s** on two 5090s. Their weights are W4A16 with int8 lm_head against our q6k, and they speculate where we do not — so the gap is configuration, not silicon. Same observation as the llama.cpp writeup on #282. ## The harder half: adapting at runtime The obvious conclusion is that a single user should be able to use the capacity idle tenants are reserving. One constraint has to be stated before anyone designs it: **KV is planned at load.** `max_in_flight` is both the admission bound and the batch engine's slot count, and the KV budget is derived from it when the model loads. So "use the spare capacity when nobody else is here" is not a per-request knob — it implies re-planning the KV budget, which today means a model reload (~5–6 min of 27B cold-load on beast). Two shapes, in increasing cost: 1. **Admit fewer, deeper sessions and let admission queue the rest.** Nearly free — it is the `max_in_flight = 2` change above — and captures most of the benefit given the measured distribution. A third caller waits rather than being served a shallow window. 2. **Genuine runtime re-planning.** Watch for sustained idleness, then re-plan KV for a deeper single session, and re-plan back under contention. Needs: an idleness signal that cannot thrash, a way to re-plan without dropping in-flight work, and an answer for what happens to the advertised `limit.context` when it changes underneath a client that has already read it. The advertised context changing mid-session is a client-visible contract change, not just an internal one. (2) is the interesting version and should not be attempted before (1) is measured — if `max_in_flight = 2` restores the deep window, the remaining upside from adaptivity is the 6% tail. ## Suggested order 1. Set `max_in_flight = 2` on beast; re-measure the tower-defence run and `helexa-bench` `concurrency:*`. Cheap, reversible, testable. 2. Re-run the concurrency measurement above afterwards, **excluding bench**, so the distribution reflects users rather than our own instrument (#288's lesson). 3. Then decide whether (2) is worth building, or whether KV quantisation (#289) and speculation (#79) return more for the effort. ## Related #290 (the reasoning work that surfaced this), #289 (KV quantisation), #79 (MTP speculation — draft head confirmed in our weights), #282 (single-stream decode gap vs llama.cpp), #98 (the batch engine these slots feed), #53/#54 (admission and fair-share), #257 (KV budget).
grenade commented 2026-08-27 09:24:21 +00:00
Author
Owner
Copy Link

Fixed in 43e873f8, deployed and verified (batch engine started max_slots=2 on beast). 00acd283 corrects the docs.

beast now runs max_in_flight = 2. Each slot gets ~59,000 tokens of KV
instead of a fraction of the advertised context, so a session can
actually reach the window we publish.

The justification sits in asset/neuron/beast.toml beside the value,
because a number like this is worthless without the evidence that chose
it: seven days of paired request spans showed 89.6% of busy time was
one request in flight and 93.6% two or fewer. Eight slots were
sizing for traffic that does not arrive, and charging every real session
for the privilege.

00acd283 fixes the docs example that advertised 8 — an operator
following it would have reproduced this bug on their own hardware, which
makes the doc the more durable half of the fix.

Fixed in `43e873f8`, deployed and verified (`batch engine started max_slots=2` on beast). `00acd283` corrects the docs. beast now runs `max_in_flight = 2`. Each slot gets ~59,000 tokens of KV instead of a fraction of the advertised context, so a session can actually reach the window we publish. The justification sits in `asset/neuron/beast.toml` beside the value, because a number like this is worthless without the evidence that chose it: seven days of paired request spans showed **89.6%** of busy time was one request in flight and **93.6%** two or fewer. Eight slots were sizing for traffic that does not arrive, and charging every real session for the privilege. `00acd283` fixes the docs example that advertised `8` — an operator following it would have reproduced this bug on their own hardware, which makes the doc the more durable half of the fix.
grenade commented 2026-08-27 20:25:18 +00:00
Author
Owner
Copy Link

Correction — reopening. The premise of this issue is wrong, and the
fix has been reverted (21eefe7e).

The issue and its closing comment both rest on:

the KV budget is divided by it at load … at 8 that gave each slot
18,852 tokens out of 150,816

That has not been true since eca42510 (2026-08-15), eleven days
before this issue was filed, which made admission reserve KV per
request:

admission.enter_with_kv(
    principal,
    kv_reservation_mb(prompt_len, requested_output, profile, cfg),
)

kv_budget is a semaphore of MiB permits. Each request takes what its
own prompt needs and waits for bytes when the pool is short — which is
precisely what kv_max_wait_secs exists for. grep finds nothing in
the tree dividing that budget by the slot count, and the engine's
max_slots is only the batch width.

So the arithmetic in the issue body — "the advertised context and the
concurrency setting are arithmetically incompatible by roughly 7×" — is
not a real constraint. Eight slots do not give a session a 19k window.
The session gets whatever it needs from a shared 118,000-token pool, and
concurrent sessions contend for bytes rather than being partitioned.

The measurement stands; the conclusion does not. Seven days of
paired spans really did show 89.6% of busy time at one request in flight
and 4.9% utilisation. But that data cannot demonstrate starvation that
the code does not implement — and it described traffic that no longer
exists. While it was being taken, the router's helexa/balanced alias
pointed the authenticated web chat at Qwen/Qwen3.6-27B while beast's
resident model was the 3.8 (see 3b0b3a22), so per-model concurrency
was split across two models that were busy evicting each other. Both
public tiers now resolve to the flagship, so chat concurrency lands on
one model.

Two things worth keeping from this:

  1. The docs contradicted themselves and nobody noticed — including
    me, while editing them. kv_max_wait_secs was documented as "how
    long it waits for enough KV budget" twelve lines above the claim that
    the budget is divided per slot. One of those describes a shared pool
    and the other a partition. Both were written down and shipped.
  2. A closing comment full of measurements is not the same as a
    verified mechanism.     This one had seven days of production spans, a
    trade table, and a live batch engine started max_slots=2
    confirmation — every number correct, attached to a causal claim that
    was never checked against the code it described.

21eefe7e restores max_in_flight = 8, corrects the operating docs,
and fixes the prefix-cache trade table, whose "tokens per slot" column
had silently divided the pool by two.

What remains genuinely open here: 00acd283 also changed the docs'
example from 8 to 2 on this reasoning, and #295/#292's closing comments
quote per-slot figures. Those are corrected in the same commit. If
anyone acted on the 2-slot guidance on their own hardware, this is the
retraction.

**Correction — reopening. The premise of this issue is wrong, and the fix has been reverted (`21eefe7e`).** The issue and its closing comment both rest on: > the KV budget is divided by it at load … at 8 that gave each slot > 18,852 tokens out of 150,816 That has not been true since **`eca42510` (2026-08-15)**, *eleven days before this issue was filed*, which made admission reserve KV per request: ```rust admission.enter_with_kv( principal, kv_reservation_mb(prompt_len, requested_output, profile, cfg), ) ``` `kv_budget` is a semaphore of MiB permits. Each request takes what its own prompt needs and waits for bytes when the pool is short — which is precisely what `kv_max_wait_secs` exists for. `grep` finds nothing in the tree dividing that budget by the slot count, and the engine's `max_slots` is only the batch width. So the arithmetic in the issue body — "the advertised context and the concurrency setting are arithmetically incompatible by roughly 7×" — is not a real constraint. Eight slots do not give a session a 19k window. The session gets whatever it needs from a shared 118,000-token pool, and concurrent sessions contend for bytes rather than being partitioned. **The measurement stands; the conclusion does not.** Seven days of paired spans really did show 89.6% of busy time at one request in flight and 4.9% utilisation. But that data cannot demonstrate starvation that the code does not implement — and it described traffic that no longer exists. While it was being taken, the router's `helexa/balanced` alias pointed the authenticated web chat at `Qwen/Qwen3.6-27B` while beast's resident model was the 3.8 (see `3b0b3a22`), so per-model concurrency was split across two models that were busy evicting each other. Both public tiers now resolve to the flagship, so chat concurrency lands on one model. Two things worth keeping from this: 1. **The docs contradicted themselves and nobody noticed** — including me, while editing them. `kv_max_wait_secs` was documented as "how long it waits for enough KV budget" twelve lines above the claim that the budget is divided per slot. One of those describes a shared pool and the other a partition. Both were written down and shipped. 2. **A closing comment full of measurements is not the same as a verified mechanism.** This one had seven days of production spans, a trade table, and a live `batch engine started max_slots=2` confirmation — every number correct, attached to a causal claim that was never checked against the code it described. `21eefe7e` restores `max_in_flight = 8`, corrects the operating docs, and fixes the prefix-cache trade table, whose "tokens per slot" column had silently divided the pool by two. What remains genuinely open here: `00acd283` also changed the docs' example from 8 to 2 on this reasoning, and #295/#292's closing comments quote per-slot figures. Those are corrected in the same commit. If anyone acted on the 2-slot guidance on their own hardware, this is the retraction.
grenade reopened this issue 2026-08-27 20:25:22 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
out-of-scope
Outside helexa's constraining principles: near-frontier models on consumer hardware, served transparently and predictably
 p1-now
Active pain or unblocking work — pick up first
 p2-next
Engine levers — start once P1 baselines exist
 p3-later
Dependent on earlier work, or demand-driven
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
grenade
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: helexa/helexa#291
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?