Serve Qwen3.8-27B #250
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Qwen3.8-27B goes open-weights at 2026-08-12 18:00 (UTC+03:00). This issue captures the preparation done ahead of the drop and the runbook for the session itself.
What is known
Confirmed: 27B ships open-weights alongside Qwen3.8-Max (2.4T-A95B, 1M context). Unconfirmed: dense vs sparse, context length, modality, licence. There is no
transformersPR and no spec sheet.Qwen/Qwen3.8-27Bon the HF API returns 401 — the same response as a repo id that does not exist — so nothing has been published yet.The decisive fact is
config.json#/model_type, and it does not exist until the drop. All preparation is therefore aimed at collapsing that unknown to minutes rather than removing it.Why the outcome is binary
qwen3_5(6,139 lines) already serves Qwen3.6-27B, TP-2 on beast. The harness allow-lists today:llama, qwen3, qwen3_5, qwen3_moe, qwen3_nextqwen3, qwen3_5, qwen3_nextIf 3.8-27B keeps the Qwen3.6 shape, support is close to a catalogue entry. If the architecture moved, it is a port on the scale of #92. Nothing in between is likely.
Prepared
script/await-model.sh— watches HF and ModelScope forQwen/Qwen3.8-27B{,-Instruct,-Thinking}and downloads the first to appear. Armed on beast since 2026-08-11, logging to~grenade/await-qwen38.log, into/models/llm-cachewith a symlink into the cache the daemon reads. Refuses to fetch below an 80 GB floor, so watching three candidate ids cannot fill the volume overnight.script/arch-triage.py— reportsmodel_type, allow-list membership, a field-level config diff against Qwen3.6-27B, parameter count, per-precision weight size, and per-token KV cost, from the small files alone. The allow-lists are parsed out ofcandle.rsrather than copied, so it cannot claim support the code lacks.--renderrenders the chat template over a tool-calling exchange — the tool-call wire format is documented nowhere else, and a mismatch shows up only as tools silently never being called.Both on branch
feat/new-model-triage-tooling.Two findings that change the plan
1. The cold-swap strategy requires unpinning Qwen3.6-27B. Two bf16 27Bs cannot be co-resident (54 + 54 GB against 64 GB of card), so the operator decision is cold-swap. But
router.rs:339andevictor.rs:38both exclude catalogue-pinned models from the evictable set, and Qwen3.6-27B carriespinned_on = ["beast"]. With the pin in place beast's evictable estimate is empty,fits_after_evictis false, and 3.8-27B can never land there. Removing the pin makes beast rank as evictable-fit (14.3 GB free + 54 GB evictable ≥ 54 GB need) and the swap proceeds. benjy and quadbrat stay correctly infeasible viamin_devices = 2.This is worth applying as part of the flip rather than in advance: unpinning also exposes the flagship to displacement by anything else that needs beast.
2. The same pin may already be blocking Qwen3-Coder-Next. Its catalogue comment states it is "NOT co-resident with the 27B (cold-swap per F4e #99)" — but it is subject to the same exclusion, so cortex-driven placement should not be able to load it either. Worth confirming whether it only ever loads via neuron's
default_models. If so, the pin has been in tension with the documented cold-swap design for some time, independent of 3.8.Separately, the fit signal is imprecise for TP models: ranking compares the whole-model
vram_mbagainst a single card's free VRAM, so an entirely empty beast still computesfits_free = falsefor a TP-2 27B. It reaches the right node anyway via the most-free fallback, and the evictable comparison is consistent on both sides, so this does not bite here — but the signal means less than it appears to for any multi-device model.Environment
/models/llm-cachehad no default ACL — the Qwen3.6-27B directory haduser:neuron:rwxapplied by hand after the fact. A fresh download would have landed unreadable by the service user and failed the load with a confusing permission error. An inheritable ACL is now set on the parent.Capacity:
/modelshas 174 GB free against a ~52 GB bf16 checkpoint. The candle fork pin is still open upstream (candle#3798, last touched 29 July) — if the new arch needs a candle bump we would be rebasing the fork under time pressure.Baseline
helexa-benchis version-aware and already holds a full sweep for Qwen3.6-27B at 5 samples/cell; the servability merge changes the build SHA, so a fresh sweep triggers on deploy without manual work. Numbers to beat:Runbook
await-qwen38.log); otherwise fetch by hand.script/arch-triage.py Qwen/Qwen3.8-27B --render→ decides everything below.model_typeis already supported: catalogue entry, unpin 3.6-27B, load, smoke, bench.harness/testdata/, add the render test, then port —script/dump_reference.py+numerical_reference.rsis the established parity method.Correction: beast serves the 27B at q6k, not bf16
The VRAM framing in the issue body used the bf16 checkpoint size. beast's
default_modelsentry is actually:Live footprint is 17.8 GB on GPU0 and 22.9 GB on GPU1 — ~40.6 GB total, not 54. Two co-resident copies would need ~35.6 and ~45.7 GB per card against 32.6 GB available, so the conclusion is unchanged: they cannot co-reside and cold-swap is the right strategy. The evictable-fit arithmetic also still works, because the router uses the catalogue's
vram_mbconsistently on both sides of the comparison.Two consequences worth carrying into the session:
vram_mb = 54000is a bf16 figure describing a model that is served quantised. Placement therefore reasons about a footprint ~13 GB larger than reality. It is conservative rather than wrong, but if Qwen3.8-27B is also served at q6k, the catalogue entry should say what it actually costs.await-model.shremains correct.Coder-Next: suggestive, not proven
beast's
default_modelscontains only the 27B, so Qwen3-Coder-Next can only ever arrive via cortex-driven placement — the path the pin should block.helexa-benchshows it served 175 runs on 2026-07-02/03 and nothing since; Qwen3-Next-80B-A3B-Thinking shows 234 runs on 2026-07-02 and nothing since. Both frontier models stopped abruptly in early July.That is consistent with the pin blocking placement, but it does not prove it: both were loaded for the F3 A/B evaluation (#94) and may simply never have been requested again. Distinguishing the two needs a placement attempt against a live cortex, which is a guarded-window exercise rather than something to infer from the bench table.
The MoE branch of the decision tree
Qwen3.8-Max is a sparse MoE (2.4T/95B active), so the 27B may be too. Triaging
Qwen/Qwen3-30B-A3Bto exercise that path surfaces a case worth knowing about before the drop rather than during it:qwen3_moeis inDENSE_SUPPORTED_MODEL_TYPESbut not inTP_SUPPORTED_MODEL_TYPES— there is notp_qwen3_moe.rs. So if Qwen3.8-27B ships as a plainqwen3_moe, beast cannot run it TP-2 the way it runs the 3.6-27B; it would be single-GPU only, and would have to fit one 32.6 GB card. At q6k that is plausible (~23 GB of weights plus KV) but it is a different deployment shape, a different catalogue entry (min_devices = 1), and a different performance profile from today's flagship.Note this does not apply if it ships as
qwen3_5/qwen3_next— those are hybrid MoE architectures that are TP-supported, and they are what Qwen3.6-27B and Coder-Next already use. The distinction is the specificmodel_type, which is exactly what step 2 of the runbook resolves.Correction: the candle pin is not a release risk
The issue body flags that a candle bump under time pressure would be painful. That overstates it in two ways.
First, there is nothing to bump to: our fork is based on candle 0.11.0, which is the current published release (26 June 2026). We are not on a stale base.
Second, and more to the point — a new architecture is a new module in our own tree (
crates/neuron/src/harness/arch/), not a candle change.qwen3_5lives there, all 6,139 lines of it. candle supplies primitives (kernels, flash-attn,VarBuilder), so it only becomes a blocker if the new arch needs a primitive that does not exist yet — a much narrower risk than "we might need to rebase the fork".The fork pin remains worth dropping when candle#3798 lands, but that is z-image hygiene and unrelated to this release.
Runbook change: do NOT unpin Qwen3.6-27B
The issue body says the cold-swap requires removing
pinned_on = ["beast"]from the 27B. That is no longer true, and doing it would now be wrong.Branch
feat/residency-priority-displacementsplits the two jobspinned_onwas doing. It now means affinity only — where a model may run. Eviction immunity moved to a newresidency_priority, and the evictor takes the incoming model so "may X displace Y" is a question about the pair rather than a property of one.So the 27B keeps
pinned_on = ["beast"](correct: it is the only node with two big cards) and carriesresidency_priority = 300.Qwen3.8-27B should be catalogued at
residency_priority = 300— the same number, not a higher one. Equal rank means the same residency class, and models in a class may displace each other in both directions. That is what makes the A/B possible: whichever generation is asked for takes beast, and the other can take it back on the next request.A higher number would be actively harmful here. Ranking 3.8 above 3.6 lets it claim beast once and hold it permanently — 3.6 could never come back via cold-load, and the comparison the cold-swap decision was made for could not be run. This was a real bug in the first cut of the change (fixed in
158a89fa); worth stating explicitly so nobody "helpfully" bumps the number during the session.The everyday class sits at 200 (Z-Image and Qwen3-8B, which take turns on benjy). 200 < 300 in both directions of comparison, so image generation still cannot take beast however idle the flagship is — which was the original reason the pin existed.
Revised step 3: catalogue Qwen3.8-27B with
pinned_on = ["beast"],min_devices = 2,residency_priority = 300; load; smoke; bench against the 3.6 baseline. No unpinning, no eviction-rule changes at flip time.Note
models.tomlis gitignored — the fleet priorities are a deploy-time config change that has to reach the gateway host separately from the merge.Release slipped to 2026-08-14
The 2026-08-12 18:00 date passed without publication.
Qwen/Qwen3.8-27Bis now expected a day later.The watcher has been restarted with a 7-day window (was 48 hours). The original deadline would have expired at 2026-08-14 13:54 UTC — 66 minutes before an 18:00 UTC+03 release, so it would have stopped watching just as the weights landed. Worth noting for any future use of
await-model.sh: pick the window against the announced time plus room for slippage, because announced dates for this family have now moved once.Presence on a registry has three states, not two
Both registries now serve something for this repo id without serving weights, and they fail differently:
huggingface.co/Qwen/Qwen3.8-27B(page)huggingface.co/api/models/Qwen/Qwen3.8-27Bmodelscope.cn/api/v1/models/…modelscope.cn/api/v1/models/…/repo/filesThe distinction matters because two of these look like success.
await-model.shqueries the HF API endpoint and the ModelScope file tree — the two that discriminate. Checking the HF page URL, which is the obvious thing to reach for, would report this model as available right now.The ModelScope variant of this already bit: a 200 on its model endpoint fired a false "APPEARED" an hour before the original release date and would have repeated every minute until the real drop. Fixed in
a97eb7e5(file-tree check, and sightings announced once per repo rather than once per poll).Nothing else changes: the fetch is still gated on the HF API check, which has never fired.
Implementation now tracked separately in #251, which carries the decision tree for what happens once
model_typeis known.This issue stays as the preparation record — the tooling built, the environment traps found and fixed, the fleet measurements, and the two design findings that changed the plan (the pinning trap, and the q6k correction). #251 references it rather than repeating it.