Serve Qwen3.8-27B #250

Open
opened 2026-08-11 08:23:44 +00:00 by grenade · 5 comments
Owner

Qwen3.8-27B goes open-weights at 2026-08-12 18:00 (UTC+03:00). This issue captures the preparation done ahead of the drop and the runbook for the session itself.

What is known

Confirmed: 27B ships open-weights alongside Qwen3.8-Max (2.4T-A95B, 1M context). Unconfirmed: dense vs sparse, context length, modality, licence. There is no transformers PR and no spec sheet. Qwen/Qwen3.8-27B on the HF API returns 401 — the same response as a repo id that does not exist — so nothing has been published yet.

The decisive fact is config.json#/model_type, and it does not exist until the drop. All preparation is therefore aimed at collapsing that unknown to minutes rather than removing it.

Why the outcome is binary

qwen3_5 (6,139 lines) already serves Qwen3.6-27B, TP-2 on beast. The harness allow-lists today:

  • dense: llama, qwen3, qwen3_5, qwen3_moe, qwen3_next
  • tensor-parallel: qwen3, qwen3_5, qwen3_next

If 3.8-27B keeps the Qwen3.6 shape, support is close to a catalogue entry. If the architecture moved, it is a port on the scale of #92. Nothing in between is likely.

Prepared

script/await-model.sh — watches HF and ModelScope for Qwen/Qwen3.8-27B{,-Instruct,-Thinking} and downloads the first to appear. Armed on beast since 2026-08-11, logging to ~grenade/await-qwen38.log, into /models/llm-cache with a symlink into the cache the daemon reads. Refuses to fetch below an 80 GB floor, so watching three candidate ids cannot fill the volume overnight.

script/arch-triage.py — reports model_type, allow-list membership, a field-level config diff against Qwen3.6-27B, parameter count, per-precision weight size, and per-token KV cost, from the small files alone. The allow-lists are parsed out of candle.rs rather than copied, so it cannot claim support the code lacks. --render renders the chat template over a tool-calling exchange — the tool-call wire format is documented nowhere else, and a mismatch shows up only as tools silently never being called.

Both on branch feat/new-model-triage-tooling.

Two findings that change the plan

1. The cold-swap strategy requires unpinning Qwen3.6-27B. Two bf16 27Bs cannot be co-resident (54 + 54 GB against 64 GB of card), so the operator decision is cold-swap. But router.rs:339 and evictor.rs:38 both exclude catalogue-pinned models from the evictable set, and Qwen3.6-27B carries pinned_on = ["beast"]. With the pin in place beast's evictable estimate is empty, fits_after_evict is false, and 3.8-27B can never land there. Removing the pin makes beast rank as evictable-fit (14.3 GB free + 54 GB evictable ≥ 54 GB need) and the swap proceeds. benjy and quadbrat stay correctly infeasible via min_devices = 2.

This is worth applying as part of the flip rather than in advance: unpinning also exposes the flagship to displacement by anything else that needs beast.

2. The same pin may already be blocking Qwen3-Coder-Next. Its catalogue comment states it is "NOT co-resident with the 27B (cold-swap per F4e #99)" — but it is subject to the same exclusion, so cortex-driven placement should not be able to load it either. Worth confirming whether it only ever loads via neuron's default_models. If so, the pin has been in tension with the documented cold-swap design for some time, independent of 3.8.

Separately, the fit signal is imprecise for TP models: ranking compares the whole-model vram_mb against a single card's free VRAM, so an entirely empty beast still computes fits_free = false for a TP-2 27B. It reaches the right node anyway via the most-free fallback, and the evictable comparison is consistent on both sides, so this does not bite here — but the signal means less than it appears to for any multi-device model.

Environment

/models/llm-cache had no default ACL — the Qwen3.6-27B directory had user:neuron:rwx applied by hand after the fact. A fresh download would have landed unreadable by the service user and failed the load with a confusing permission error. An inheritable ACL is now set on the parent.

Capacity: /models has 174 GB free against a ~52 GB bf16 checkpoint. The candle fork pin is still open upstream (candle#3798, last touched 29 July) — if the new arch needs a candle bump we would be rebasing the fork under time pressure.

Baseline

helexa-bench is version-aware and already holds a full sweep for Qwen3.6-27B at 5 samples/cell; the servability merge changes the build SHA, so a fresh sweep triggers on deploy without manual work. Numbers to beat:

scenario TTFT tok/s
chat:128 1.78 s 33.0
chat:4096 0.09 s 31.2
concurrency:8 0.48 s 167.2 (aggregate)
capability:rust-plan 1.23 s 32.3

Runbook

  1. Confirm the fetch landed (await-qwen38.log); otherwise fetch by hand.
  2. script/arch-triage.py Qwen/Qwen3.8-27B --render → decides everything below.
  3. If model_type is already supported: catalogue entry, unpin 3.6-27B, load, smoke, bench.
  4. If not: capture the chat template into harness/testdata/, add the render test, then port — script/dump_reference.py + numerical_reference.rs is the established parity method.
  5. Live-validate through the public chain, not just neuron, before calling it served.
Qwen3.8-27B goes open-weights at **2026-08-12 18:00 (UTC+03:00)**. This issue captures the preparation done ahead of the drop and the runbook for the session itself. ## What is known Confirmed: 27B ships open-weights alongside Qwen3.8-Max (2.4T-A95B, 1M context). Unconfirmed: dense vs sparse, context length, modality, licence. There is no `transformers` PR and no spec sheet. `Qwen/Qwen3.8-27B` on the HF API returns 401 — the same response as a repo id that does not exist — so nothing has been published yet. The decisive fact is `config.json#/model_type`, and it does not exist until the drop. All preparation is therefore aimed at collapsing that unknown to minutes rather than removing it. ## Why the outcome is binary `qwen3_5` (6,139 lines) already serves Qwen3.6-27B, TP-2 on beast. The harness allow-lists today: - dense: `llama, qwen3, qwen3_5, qwen3_moe, qwen3_next` - tensor-parallel: `qwen3, qwen3_5, qwen3_next` If 3.8-27B keeps the Qwen3.6 shape, support is close to a catalogue entry. If the architecture moved, it is a port on the scale of #92. Nothing in between is likely. ## Prepared **`script/await-model.sh`** — watches HF and ModelScope for `Qwen/Qwen3.8-27B{,-Instruct,-Thinking}` and downloads the first to appear. **Armed on beast since 2026-08-11**, logging to `~grenade/await-qwen38.log`, into `/models/llm-cache` with a symlink into the cache the daemon reads. Refuses to fetch below an 80 GB floor, so watching three candidate ids cannot fill the volume overnight. **`script/arch-triage.py`** — reports `model_type`, allow-list membership, a field-level config diff against Qwen3.6-27B, parameter count, per-precision weight size, and per-token KV cost, from the small files alone. The allow-lists are parsed out of `candle.rs` rather than copied, so it cannot claim support the code lacks. `--render` renders the chat template over a tool-calling exchange — the tool-call wire format is documented nowhere else, and a mismatch shows up only as tools silently never being called. Both on branch `feat/new-model-triage-tooling`. ## Two findings that change the plan **1. The cold-swap strategy requires unpinning Qwen3.6-27B.** Two bf16 27Bs cannot be co-resident (54 + 54 GB against 64 GB of card), so the operator decision is cold-swap. But `router.rs:339` and `evictor.rs:38` both exclude catalogue-pinned models from the evictable set, and Qwen3.6-27B carries `pinned_on = ["beast"]`. With the pin in place beast's evictable estimate is empty, `fits_after_evict` is false, and **3.8-27B can never land there**. Removing the pin makes beast rank as evictable-fit (14.3 GB free + 54 GB evictable ≥ 54 GB need) and the swap proceeds. benjy and quadbrat stay correctly infeasible via `min_devices = 2`. This is worth applying as part of the flip rather than in advance: unpinning also exposes the flagship to displacement by anything else that needs beast. **2. The same pin may already be blocking Qwen3-Coder-Next.** Its catalogue comment states it is "NOT co-resident with the 27B (cold-swap per F4e #99)" — but it is subject to the same exclusion, so cortex-driven placement should not be able to load it either. Worth confirming whether it only ever loads via neuron's `default_models`. If so, the pin has been in tension with the documented cold-swap design for some time, independent of 3.8. Separately, the fit signal is imprecise for TP models: ranking compares the whole-model `vram_mb` against a *single* card's free VRAM, so an entirely empty beast still computes `fits_free = false` for a TP-2 27B. It reaches the right node anyway via the most-free fallback, and the evictable comparison is consistent on both sides, so this does not bite here — but the signal means less than it appears to for any multi-device model. ## Environment `/models/llm-cache` had **no default ACL** — the Qwen3.6-27B directory had `user:neuron:rwx` applied by hand after the fact. A fresh download would have landed unreadable by the service user and failed the load with a confusing permission error. An inheritable ACL is now set on the parent. Capacity: `/models` has 174 GB free against a ~52 GB bf16 checkpoint. The candle fork pin is still open upstream ([candle#3798](https://github.com/huggingface/candle/pull/3798), last touched 29 July) — if the new arch needs a candle bump we would be rebasing the fork under time pressure. ## Baseline `helexa-bench` is version-aware and already holds a full sweep for Qwen3.6-27B at 5 samples/cell; the servability merge changes the build SHA, so a fresh sweep triggers on deploy without manual work. Numbers to beat: | scenario | TTFT | tok/s | |---|---|---| | chat:128 | 1.78 s | 33.0 | | chat:4096 | 0.09 s | 31.2 | | concurrency:8 | 0.48 s | 167.2 (aggregate) | | capability:rust-plan | 1.23 s | 32.3 | ## Runbook 1. Confirm the fetch landed (`await-qwen38.log`); otherwise fetch by hand. 2. `script/arch-triage.py Qwen/Qwen3.8-27B --render` → decides everything below. 3. If `model_type` is already supported: catalogue entry, unpin 3.6-27B, load, smoke, bench. 4. If not: capture the chat template into `harness/testdata/`, add the render test, then port — `script/dump_reference.py` + `numerical_reference.rs` is the established parity method. 5. Live-validate through the public chain, not just neuron, before calling it served.
Author
Owner

Correction: beast serves the 27B at q6k, not bf16

The VRAM framing in the issue body used the bf16 checkpoint size. beast's default_models entry is actually:

model_id = "Qwen/Qwen3.6-27B"
quant = "q6k"
tensor_parallel = 2
devices = [0, 1]

Live footprint is 17.8 GB on GPU0 and 22.9 GB on GPU1 — ~40.6 GB total, not 54. Two co-resident copies would need ~35.6 and ~45.7 GB per card against 32.6 GB available, so the conclusion is unchanged: they cannot co-reside and cold-swap is the right strategy. The evictable-fit arithmetic also still works, because the router uses the catalogue's vram_mb consistently on both sides of the comparison.

Two consequences worth carrying into the session:

  • The catalogue's vram_mb = 54000 is a bf16 figure describing a model that is served quantised. Placement therefore reasons about a footprint ~13 GB larger than reality. It is conservative rather than wrong, but if Qwen3.8-27B is also served at q6k, the catalogue entry should say what it actually costs.
  • The disk estimate is unaffected — the checkpoint still downloads as ~52 GB of bf16 safetensors and is quantised in situ at load. The 80 GB free-space floor in await-model.sh remains correct.

Coder-Next: suggestive, not proven

beast's default_models contains only the 27B, so Qwen3-Coder-Next can only ever arrive via cortex-driven placement — the path the pin should block. helexa-bench shows it served 175 runs on 2026-07-02/03 and nothing since; Qwen3-Next-80B-A3B-Thinking shows 234 runs on 2026-07-02 and nothing since. Both frontier models stopped abruptly in early July.

That is consistent with the pin blocking placement, but it does not prove it: both were loaded for the F3 A/B evaluation (#94) and may simply never have been requested again. Distinguishing the two needs a placement attempt against a live cortex, which is a guarded-window exercise rather than something to infer from the bench table.

## Correction: beast serves the 27B at q6k, not bf16 The VRAM framing in the issue body used the bf16 checkpoint size. beast's `default_models` entry is actually: ```toml model_id = "Qwen/Qwen3.6-27B" quant = "q6k" tensor_parallel = 2 devices = [0, 1] ``` Live footprint is 17.8 GB on GPU0 and 22.9 GB on GPU1 — ~40.6 GB total, not 54. Two co-resident copies would need ~35.6 and ~45.7 GB per card against 32.6 GB available, so **the conclusion is unchanged**: they cannot co-reside and cold-swap is the right strategy. The evictable-fit arithmetic also still works, because the router uses the catalogue's `vram_mb` consistently on both sides of the comparison. Two consequences worth carrying into the session: - The catalogue's `vram_mb = 54000` is a bf16 figure describing a model that is served quantised. Placement therefore reasons about a footprint ~13 GB larger than reality. It is conservative rather than wrong, but if Qwen3.8-27B is also served at q6k, the catalogue entry should say what it actually costs. - The disk estimate is unaffected — the checkpoint still downloads as ~52 GB of bf16 safetensors and is quantised in situ at load. The 80 GB free-space floor in `await-model.sh` remains correct. ## Coder-Next: suggestive, not proven beast's `default_models` contains **only** the 27B, so Qwen3-Coder-Next can only ever arrive via cortex-driven placement — the path the pin should block. `helexa-bench` shows it served 175 runs on 2026-07-02/03 and **nothing since**; Qwen3-Next-80B-A3B-Thinking shows 234 runs on 2026-07-02 and nothing since. Both frontier models stopped abruptly in early July. That is consistent with the pin blocking placement, but it does not prove it: both were loaded for the F3 A/B evaluation (#94) and may simply never have been requested again. Distinguishing the two needs a placement attempt against a live cortex, which is a guarded-window exercise rather than something to infer from the bench table.
Author
Owner

The MoE branch of the decision tree

Qwen3.8-Max is a sparse MoE (2.4T/95B active), so the 27B may be too. Triaging Qwen/Qwen3-30B-A3B to exercise that path surfaces a case worth knowing about before the drop rather than during it:

model_type      qwen3_moe
dense path      SUPPORTED
tensor parallel NOT SUPPORTED

qwen3_moe is in DENSE_SUPPORTED_MODEL_TYPES but not in TP_SUPPORTED_MODEL_TYPES — there is no tp_qwen3_moe.rs. So if Qwen3.8-27B ships as a plain qwen3_moe, beast cannot run it TP-2 the way it runs the 3.6-27B; it would be single-GPU only, and would have to fit one 32.6 GB card. At q6k that is plausible (~23 GB of weights plus KV) but it is a different deployment shape, a different catalogue entry (min_devices = 1), and a different performance profile from today's flagship.

Note this does not apply if it ships as qwen3_5 / qwen3_next — those are hybrid MoE architectures that are TP-supported, and they are what Qwen3.6-27B and Coder-Next already use. The distinction is the specific model_type, which is exactly what step 2 of the runbook resolves.

Correction: the candle pin is not a release risk

The issue body flags that a candle bump under time pressure would be painful. That overstates it in two ways.

First, there is nothing to bump to: our fork is based on candle 0.11.0, which is the current published release (26 June 2026). We are not on a stale base.

Second, and more to the point — a new architecture is a new module in our own tree (crates/neuron/src/harness/arch/), not a candle change. qwen3_5 lives there, all 6,139 lines of it. candle supplies primitives (kernels, flash-attn, VarBuilder), so it only becomes a blocker if the new arch needs a primitive that does not exist yet — a much narrower risk than "we might need to rebase the fork".

The fork pin remains worth dropping when candle#3798 lands, but that is z-image hygiene and unrelated to this release.

## The MoE branch of the decision tree Qwen3.8-Max is a sparse MoE (2.4T/95B active), so the 27B may be too. Triaging `Qwen/Qwen3-30B-A3B` to exercise that path surfaces a case worth knowing about before the drop rather than during it: ``` model_type qwen3_moe dense path SUPPORTED tensor parallel NOT SUPPORTED ``` `qwen3_moe` is in `DENSE_SUPPORTED_MODEL_TYPES` but **not** in `TP_SUPPORTED_MODEL_TYPES` — there is no `tp_qwen3_moe.rs`. So if Qwen3.8-27B ships as a plain `qwen3_moe`, beast cannot run it TP-2 the way it runs the 3.6-27B; it would be single-GPU only, and would have to fit one 32.6 GB card. At q6k that is plausible (~23 GB of weights plus KV) but it is a different deployment shape, a different catalogue entry (`min_devices = 1`), and a different performance profile from today's flagship. Note this does **not** apply if it ships as `qwen3_5` / `qwen3_next` — those are hybrid MoE architectures that *are* TP-supported, and they are what Qwen3.6-27B and Coder-Next already use. The distinction is the specific `model_type`, which is exactly what step 2 of the runbook resolves. ## Correction: the candle pin is not a release risk The issue body flags that a candle bump under time pressure would be painful. That overstates it in two ways. First, there is nothing to bump to: our fork is based on **candle 0.11.0, which is the current published release** (26 June 2026). We are not on a stale base. Second, and more to the point — a new architecture is a new module in **our own tree** (`crates/neuron/src/harness/arch/`), not a candle change. `qwen3_5` lives there, all 6,139 lines of it. candle supplies primitives (kernels, flash-attn, `VarBuilder`), so it only becomes a blocker if the new arch needs a primitive that does not exist yet — a much narrower risk than "we might need to rebase the fork". The fork pin remains worth dropping when [candle#3798](https://github.com/huggingface/candle/pull/3798) lands, but that is z-image hygiene and unrelated to this release.
Author
Owner

Runbook change: do NOT unpin Qwen3.6-27B

The issue body says the cold-swap requires removing pinned_on = ["beast"] from the 27B. That is no longer true, and doing it would now be wrong.

Branch feat/residency-priority-displacement splits the two jobs pinned_on was doing. It now means affinity only — where a model may run. Eviction immunity moved to a new residency_priority, and the evictor takes the incoming model so "may X displace Y" is a question about the pair rather than a property of one.

So the 27B keeps pinned_on = ["beast"] (correct: it is the only node with two big cards) and carries residency_priority = 300.

Qwen3.8-27B should be catalogued at residency_priority = 300 — the same number, not a higher one. Equal rank means the same residency class, and models in a class may displace each other in both directions. That is what makes the A/B possible: whichever generation is asked for takes beast, and the other can take it back on the next request.

A higher number would be actively harmful here. Ranking 3.8 above 3.6 lets it claim beast once and hold it permanently — 3.6 could never come back via cold-load, and the comparison the cold-swap decision was made for could not be run. This was a real bug in the first cut of the change (fixed in 158a89fa); worth stating explicitly so nobody "helpfully" bumps the number during the session.

The everyday class sits at 200 (Z-Image and Qwen3-8B, which take turns on benjy). 200 < 300 in both directions of comparison, so image generation still cannot take beast however idle the flagship is — which was the original reason the pin existed.

Revised step 3: catalogue Qwen3.8-27B with pinned_on = ["beast"], min_devices = 2, residency_priority = 300; load; smoke; bench against the 3.6 baseline. No unpinning, no eviction-rule changes at flip time.

Note models.toml is gitignored — the fleet priorities are a deploy-time config change that has to reach the gateway host separately from the merge.

## Runbook change: do NOT unpin Qwen3.6-27B The issue body says the cold-swap requires removing `pinned_on = ["beast"]` from the 27B. That is no longer true, and doing it would now be wrong. Branch `feat/residency-priority-displacement` splits the two jobs `pinned_on` was doing. It now means affinity only — *where* a model may run. Eviction immunity moved to a new `residency_priority`, and the evictor takes the incoming model so "may X displace Y" is a question about the pair rather than a property of one. So the 27B keeps `pinned_on = ["beast"]` (correct: it is the only node with two big cards) **and** carries `residency_priority = 300`. **Qwen3.8-27B should be catalogued at `residency_priority = 300` — the same number, not a higher one.** Equal rank means the same residency class, and models in a class may displace each other in both directions. That is what makes the A/B possible: whichever generation is asked for takes beast, and the other can take it back on the next request. A higher number would be actively harmful here. Ranking 3.8 above 3.6 lets it claim beast once and hold it permanently — 3.6 could never come back via cold-load, and the comparison the cold-swap decision was made for could not be run. This was a real bug in the first cut of the change (fixed in `158a89fa`); worth stating explicitly so nobody "helpfully" bumps the number during the session. The everyday class sits at 200 (Z-Image and Qwen3-8B, which take turns on benjy). 200 < 300 in both directions of comparison, so image generation still cannot take beast however idle the flagship is — which was the original reason the pin existed. Revised step 3: *catalogue Qwen3.8-27B with `pinned_on = ["beast"]`, `min_devices = 2`, `residency_priority = 300`; load; smoke; bench against the 3.6 baseline.* No unpinning, no eviction-rule changes at flip time. Note `models.toml` is gitignored — the fleet priorities are a deploy-time config change that has to reach the gateway host separately from the merge.
Author
Owner

Release slipped to 2026-08-14

The 2026-08-12 18:00 date passed without publication. Qwen/Qwen3.8-27B is now expected a day later.

The watcher has been restarted with a 7-day window (was 48 hours). The original deadline would have expired at 2026-08-14 13:54 UTC — 66 minutes before an 18:00 UTC+03 release, so it would have stopped watching just as the weights landed. Worth noting for any future use of await-model.sh: pick the window against the announced time plus room for slippage, because announced dates for this family have now moved once.

Presence on a registry has three states, not two

Both registries now serve something for this repo id without serving weights, and they fail differently:

Check Now Means
huggingface.co/Qwen/Qwen3.8-27B (page) 200 staged placeholder
huggingface.co/api/models/Qwen/Qwen3.8-27B 401 not published
modelscope.cn/api/v1/models/… 200, empty payload reserved id
modelscope.cn/api/v1/models/…/repo/files fails no files

The distinction matters because two of these look like success. await-model.sh queries the HF API endpoint and the ModelScope file tree — the two that discriminate. Checking the HF page URL, which is the obvious thing to reach for, would report this model as available right now.

The ModelScope variant of this already bit: a 200 on its model endpoint fired a false "APPEARED" an hour before the original release date and would have repeated every minute until the real drop. Fixed in a97eb7e5 (file-tree check, and sightings announced once per repo rather than once per poll).

Nothing else changes: the fetch is still gated on the HF API check, which has never fired.

## Release slipped to 2026-08-14 The 2026-08-12 18:00 date passed without publication. `Qwen/Qwen3.8-27B` is now expected a day later. The watcher has been restarted with a **7-day** window (was 48 hours). The original deadline would have expired at 2026-08-14 13:54 UTC — **66 minutes before** an 18:00 UTC+03 release, so it would have stopped watching just as the weights landed. Worth noting for any future use of `await-model.sh`: pick the window against the announced time *plus* room for slippage, because announced dates for this family have now moved once. ## Presence on a registry has three states, not two Both registries now serve something for this repo id without serving weights, and they fail differently: | Check | Now | Means | |---|---|---| | `huggingface.co/Qwen/Qwen3.8-27B` (page) | **200** | staged placeholder | | `huggingface.co/api/models/Qwen/Qwen3.8-27B` | **401** | not published | | `modelscope.cn/api/v1/models/…` | **200**, empty payload | reserved id | | `modelscope.cn/api/v1/models/…/repo/files` | fails | no files | The distinction matters because two of these look like success. `await-model.sh` queries the HF **API** endpoint and the ModelScope **file tree** — the two that discriminate. Checking the HF *page* URL, which is the obvious thing to reach for, would report this model as available right now. The ModelScope variant of this already bit: a 200 on its model endpoint fired a false "APPEARED" an hour before the original release date and would have repeated every minute until the real drop. Fixed in `a97eb7e5` (file-tree check, and sightings announced once per repo rather than once per poll). Nothing else changes: the fetch is still gated on the HF API check, which has never fired.
Author
Owner

Implementation now tracked separately in #251, which carries the decision tree for what happens once model_type is known.

This issue stays as the preparation record — the tooling built, the environment traps found and fixed, the fleet measurements, and the two design findings that changed the plan (the pinning trap, and the q6k correction). #251 references it rather than repeating it.

Implementation now tracked separately in #251, which carries the decision tree for what happens once `model_type` is known. This issue stays as the preparation record — the tooling built, the environment traps found and fixed, the fleet measurements, and the two design findings that changed the plan (the pinning trap, and the q6k correction). #251 references it rather than repeating it.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: helexa/helexa#250