Block a user
grenade
created branch fix/291-max-in-flight-and-config-only-ci in helexa/helexa
2026-08-26 08:40:08 +00:00
beast reserves KV for eight tenants to serve one — max_in_flight is starving the single deep session it actually gets
grenade
created branch feat/290-reasoning-rungs-from-the-model in helexa/helexa
2026-08-26 06:14:55 +00:00
grenade
pushed to feat/290-reasoning-rungs-from-the-model at helexa/helexa
2026-08-26 06:14:55 +00:00
Reasoning effort: we invented a ladder the model rejects, never tell the model which rung was chosen, and guillotine its thinking at a budget it was told to exceed
Evaluate speculative decoding for neuron (EAGLE-3 first, bench-gated) — batch-1 decode latency
Step (1) done: the MTP head is in the weights we already serve
Inspected the deployed Qwen/Qwen3.8-27B snapshot on beast directly, so
this is no longer a claim inherited from a blog…
Evaluate KV-cache quantisation — a third party reports q4_0 is both faster and smaller than f16
Evaluate speculative decoding for neuron (EAGLE-3 first, bench-gated) — batch-1 decode latency
Qwen3.8-27B appears to ship an MTP draft head, and llama.cpp reports 1.81× from it
Source: https://kgptalkie.com/tutorials/generative-ai/qwen-3-8-27b-llama-cpp-speed-settings — Qwen3.8-27B…
[epic] Decode efficiency: we are at ~16% of roofline on beast and do not know where the rest goes
A third-party llama.cpp benchmark on the same model and GPU class — read with care
Source: https://kgptalkie.com/tutorials/generative-ai/qwen-3-8-27b-llama-cpp-speed-settings — Qwen3.8-27B…