Compare commits

...
Author SHA1 Message Date
grenadeandClaude Opus 5 15c664c5c3 fix(quant): use llama.cpp's fallback table, which spends bits rather than saving them
CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 20s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 11s
CI / Clippy (push) Successful in 2m31s
CI / Test (push) Successful in 8m59s
CI / CUDA type-check (push) Successful in 19m19s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
`bf1e80ed` said it met the 640-wide reduction "the way llama.cpp meets
the same wall" and did not. Upstream's `tensor_type_fallback`
(src/llama-quant.cpp) maps Q4K -> Q5_0 and Q5K -> Q5_1; that commit
used Q4_0 and Q5_0. Only Q2K/Q3K -> Q4_0 and Q6K -> Q8_0 matched.

The difference is a principle, not two table cells. Every upstream
fallback *increases* bits per weight: a k-quant carries a two-level
scale hierarchy the legacy 32-block types do not, so it is better than
a legacy type of equal width, and dropping Q4K (4.5 bpw) to Q4_0 (also
4.5) holds memory constant by taking the quality hit in silence.
Upstream pays a bit instead. We were taking the silent trade, in the
one direction that flatters the constraint we are straining against,
having asserted conformance to a table nobody had read.

It is not free. `down_proj` is exactly a third of the 120.8 B routed
parameters, so Q5_0 rather than Q4_0 costs +5.0 GB across the expert
bank: 67.9 -> 73.0 GB. Every published number that rests on the expert
footprint moves with it, and the model was already over both cards
combined, so this makes host residency more necessary rather than less.
That is the right way round — the cheaper table did not make the model
fit, it just made the shortfall look smaller.

The undivisible-by-32 case now falls back to F16 as upstream does,
rather than erroring. It is unreachable for this checkpoint (640 and
2560 are both multiples of 32) and vanishingly rare in general, but a
3.5x blowup on a tensor must be loud rather than fatal, so it warns.

Also corrects the record: nothing here is GGUF-incompatible. GGUF
stores a ggml_type per tensor and mixed types within a file are the
norm — "Q4_K_M" names a mixture, not a uniform type. A per-row
substitution is just the tensor's type. What still differs from a file
llama.cpp would write is the *selection* policy above this function
(their per-tensor upgrades and imatrix weighting), which is a quality
question for #315, not a format one.

Refs #307, #315, #322.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 08:22:59 +03:00
grenade 7593ab3d5d Merge branch 'fix/qwen4-exp-expert-row-width'
build-prerelease / Resolve version stamps + change detection (push) Successful in 22s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m39s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m35s
build-prerelease / Build neuron-ampere (push) Successful in 2m39s
build-prerelease / Test (push) Successful in 9m16s
build-prerelease / Build neuron-blackwell (push) Successful in 14m42s
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m15s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m15s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m14s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 28s
2026-09-06 21:21:11 +03:00
grenadeandClaude Opus 5 bf1e80ed55 fix(qwen4_exp): 640 is not a whole number of k-quant blocks
CI / Config consistency (push) Successful in 11s
CI / Classify changes (push) Successful in 23s
CI / Format (push) Successful in 12s
CI / Clippy (push) Successful in 2m37s
CI / Test (push) Successful in 7m2s
CI / CUDA type-check (push) Successful in 19m17s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The first real cold load on beast got through routing, resolved all
131 shards, parsed the config, reached decoder layer 0 — and died
there:

    quantize experts to Q4K: quantized tensor must have their last dim
    divisible by block size [2560, 640] 256

ggml quantises along the last dimension and a row must be a whole
number of blocks. `gate_up` reduces over `hidden_size` = 2560, which
is ten blocks of 256. `down_proj` reduces over `moe_intermediate_size`
= 640, which is two and a half. So `quant = "q4k"` could never have
loaded this model, on any card, whatever else was fixed first.

It is not an orientation mistake — `[2560, 640]` is the `[out, in]`
layout a `QMatMul` wants, and 640 is genuinely the reduction. The
constraint is the number itself. llama.cpp meets the same wall and
answers it the same way: substitute a 32-block type for the row that
will not divide. Q4_0 and Q4K are both 4.5 bpw, so the memory
arithmetic every placement decision rests on is unchanged; the trade
is quality, on one of the three expert matrices, against not loading
at all. A row that divides neither 256 nor 32 is an error, because
silently leaving those weights dense would be a memory surprise rather
than a mercy.

The substitution is decided once per layer from the shapes rather than
per expert -- 512 identical lines a layer would bury the one fact
worth reading.

The test uses the checkpoint's own widths, 2560 and 640, scaled down
only in expert count. The fixture that missed this was 256 in both
dimensions -- the narrowest a k-quant accepts -- so every row divided
and the two halves were indistinguishable. A test whose fixture cannot
express the failure is not evidence about it.

Refs #307, #315, #320.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-06 20:59:39 +03:00
grenade a14d1af447 Merge branch 'fix/qwen4-exp-load-reporting-and-pool-trim'
build-prerelease / Resolve version stamps + change detection (push) Successful in 21s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m43s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m35s
build-prerelease / Build neuron-ampere (push) Successful in 2m36s
build-prerelease / Test (push) Successful in 10m54s
build-prerelease / Build neuron-blackwell (push) Successful in 14m28s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m7s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m16s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m8s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 30s
2026-09-06 19:58:00 +03:00
grenadeandClaude Opus 5 f1af926c30 fix(qwen4_exp): route the arch on the path that actually loads it
CI / Config consistency (push) Successful in 12s
CI / Classify changes (push) Successful in 27s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 15s
CI / Clippy (push) Successful in 2m39s
CI / Test (push) Successful in 6m49s
CI / CUDA type-check (push) Successful in 19m10s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
`2f2f306c` added qwen4_exp to `DENSE_SUPPORTED_MODEL_TYPES` and to
`candle::load_arch_dense`. That is the CPU fallback. Every CUDA load
goes through the device worker's own `load_dense_inner`, which had no
arm for it — so on a GPU host the model passed preflight and then hit

    unrouted supported model_type 'qwen4_exp' — DENSE_SUPPORTED_MODEL_TYPES
    and load_dense_inner must stay in sync

in about 200 ms, having read none of the 360 GB. The bail's own comment
is the specification this violated, and it was written before there was
a second loader to violate it.

The arm mirrors the CPU one, and the job now carries the parsed
`quant`. It has to: the worker load never had a quant parameter, so
even a routed qwen4_exp would have loaded BF16 experts — 241.6 GB of
them, measured from the checkpoint's own safetensors headers, against
a 32.6 GB card. In-situ quantisation reaching only the CPU path is the
same defect #320 fixed one layer up, and it would have looked like
#320's fix had not worked.

The test asserts every model_type the preflight admits has an arm here,
by driving a bare `{"model_type": ...}` at the loader and requiring the
failure to be anything other than "unrouted". Adding a name to the list
with no loader behind it now fails in CI rather than on a GPU host at
load time — which is the only place the difference between the two
loaders was observable.

Refs #307, #315, #320.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-06 19:36:46 +03:00
grenadeandClaude Opus 5 430b6be62b fix(neuron): let the loaded model answer for itself, and give the pool back
CI / Test (push) Blocked by required conditions
CI / Config consistency (push) Successful in 9s
CI / Classify changes (push) Successful in 21s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / CUDA type-check (push) Successful in 19m23s
CI / Format (push) Successful in 12s
CI / Clippy (push) Successful in 2m41s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Three things the first qwen4_exp cold load would have got wrong, none
of which the load itself has to reach to matter.

The single-GPU path decided prefix-snapshot capability by matching
config.json's model_type against a list of two, while
ModelArch::supports_kv_snapshot had grown to three. So qwen4_exp
loaded with snapshots reported unavailable and batched decode silently
off — the work that landed the snapshot was reachable only through a
list somewhere else that nobody had reason to look at. The dense load
reply now carries the arch's own answer (DenseLoad), taken while the
model is still in hand on the worker thread, which is the only place
that can be asked. The TP path still re-derives it; that is correct
for the two archs it serves today and it is commented as the hazard it
is, because #319 will add a third.

Nothing built a context profile for qwen4_exp either, so derive_limit
had no physics and the model would have advertised the static prompt
cap. The profile is the qwen3_5 one plus a term: the QSA indexer keeps
a second cache on every full-attention layer, 12 x 1 x 128 x 2 bytes =
3 KiB/token beside the main cache's 24. Advertising against the main
cache alone overstates what the card can hold by an eighth, which is
the #281 failure mode. The test drives both builders off the
checkpoint's verbatim config.json and asserts each declines the
other's, since the load path now tries them in turn.

And a load that fails part-way left its bytes in cudarc's pool. The
image path learned this at the benjy cold-swap and trims after every
load and drop; the text path never did. A dense load that runs out of
VRAM has typically allocated tens of gigabytes first, and the model
that was evicted to make room for it needs them back — so LoadGguf,
LoadDense, DropArch and DropTp now trim the way LoadImage and
DropImage always have.

Refs #307, #310, #281.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-06 19:10:08 +03:00
grenade af40bac9b8 Merge branch 'fix/320-isq-reachable-and-log-levels'
build-prerelease / Resolve version stamps + change detection (push) Successful in 18s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Lint (fmt + clippy) (push) Successful in 4m7s
build-prerelease / Test (push) Successful in 9m15s
build-prerelease / Build neuron-ampere (push) Successful in 2m11s
build-prerelease / Build neuron-ada (push) Successful in 2m14s
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Successful in 14m18s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m6s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 29s
2026-09-06 18:03:15 +03:00
grenade d5738bfb89 fix(neuron): read quant as ISQ when the repo has no GGUF (#320)
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 14s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m36s
CI / Test (push) Successful in 9m44s
CI / CUDA type-check (push) Successful in 19m12s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The single-GPU load paths treated a non-empty `quant` as "select the
GGUF whose filename contains this string", while the TP path treats the
same field as in-situ quantisation of safetensors. A model published
only as safetensors is therefore loadable quantised on two cards and
not on one, which is backwards — the single card is the one that needs
the quantisation.

Fall through to the dense loader (which applies ISQ) when the repo has
no GGUF at all but does have safetensors. A repo that does carry GGUFs
keeps the old behaviour exactly, including the old error when none of
them match: that case is a typo, not an ISQ request, and silently
turning it into a full-precision load would be a worse answer than the
error.

Also drop the per-model context-limit line from debug to trace. Every
input to it is fixed at load, so it repeats byte-identically on every
`GET /models` poll — roughly 8,600 lines a day per model saying the
same thing, on top of whatever you were actually reading the journal
for.

And set RUST_LOG=debug for neuron fleet-wide from deploy.yml, in the
model.conf drop-in it already manages. The code default is `info`,
which is too coarse to reconstruct a load or a routing decision after
the fact; before this, one host had `debug` because it was set there by
hand, and the others quietly did not. A verbosity you have to remember
to set is one you discover you did not have at the moment you needed
it. It sorts after local.conf, so the managed value wins.
2026-09-03 16:22:19 +03:00
grenade ecbfe22124 Merge branch 'feat/qwen4-exp-quantised-experts'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m45s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m27s
build-prerelease / Build neuron-ampere (push) Successful in 2m28s
build-prerelease / Test (push) Successful in 10m36s
build-prerelease / Build neuron-blackwell (push) Successful in 14m47s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m3s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m11s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m10s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 24s
2026-09-02 19:52:10 +03:00
grenade 3e5d39807d fix(qwen4_exp): point the CUDA leader load at the moved parse_quant_string
CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 16s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m32s
CI / Test (push) Successful in 9m9s
CI / CUDA type-check (push) Successful in 19m20s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Moving the function out of tp/worker.rs left dispatch.rs calling it
through worker's `use` import, which is private — so the call compiled
nowhere except in my head. The CUDA type-check caught it; the CPU
build cannot, because that call site is cfg(cuda) and every local check
I ran was blind to it.

That is what the job is for, and it is the second time this port has
relied on it. Worth remembering that "fmt, clippy and test are green"
means green on roughly half of this crate.
2026-09-02 19:30:08 +03:00
grenade 05a5aff25e feat(qwen4_exp): quantise the experts in situ, which is what makes it loadable
CI / Config consistency (push) Successful in 10s
CI / Classify changes (push) Successful in 20s
CI / Format (push) Successful in 12s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Clippy (push) Successful in 2m46s
CI / Test (push) Successful in 12m52s
CI / CUDA type-check (push) Failing after 19m32s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
120.8 B routed parameters are 241.6 GB at BF16 and 67.9 GB at q4k,
against 64 GB of VRAM and roughly 110 GB of usable host RAM. Everything
else in this port has been waiting on that arithmetic.

The experts are quantised out of the banked tensors rather than out of
the checkpoint, which is what bounds the peak: one layer's fused pair is
5.03 GB, and it is dropped once its 512 experts are quantised, so the
full 241.6 GB never exists at once. That is the reason the previous
commit banked them.

q4k was chosen on evidence rather than taste. fp8 experts are excluded
by arithmetic — 120.8 GB of experts plus 51.2 GB of PLE tables is
176.9 GB, which does not fit even counting host RAM. NVFP4 and IQ4_XS
are memory-equivalent to q4k at ~4-4.5 bpw, so they separate only on
kernels, and neuron already implements the ggml k-quants through the
same in-situ path the 27B and the image model use in production. NVFP4
is the optimisation to race later, against a *batched* measurement:
FreeToken measured it losing to plain Triton at M=1 for expert widths
below 1024, and this model is 640.

parse_quant_string moves out from behind cfg(feature = "cuda"). A
catalogue quant name means the same thing whichever loader reads it, and
it was unavailable to precisely the path that now needs it.

One trap worth the comment it got: QTensor::quantize reads
src.storage() — the whole backing buffer, ignoring shape and offset — so
a view into the fused tensor hands it every expert's bytes while the
shape claims one, and candle panics on a block count deep inside
from_float. Neither contiguous() nor copy() helps, because a slice along
dim 0 is already contiguous and both are then free to keep sharing the
parent buffer. The source has to own its storage outright.

The test uses 256-wide tensors because the k-quants block on 256
elements, so a toy fixture cannot be quantised at all — and it compares
against an unrelated expert as well as against the dense one, so a
quantised path that returned something smoothly wrong would not pass.

Refs #307, #315, #259.
2026-09-02 18:52:24 +03:00
grenade cd74eeaf1c Merge branch 'feat/qwen4-exp-expert-bank'
build-prerelease / Resolve version stamps + change detection (push) Successful in 22s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m36s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Test (push) Successful in 9m29s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 2m25s
build-prerelease / Build neuron-ada (push) Successful in 2m28s
build-prerelease / Build neuron-blackwell (push) Successful in 17m0s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m5s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m9s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m5s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 29s
2026-09-02 18:16:31 +03:00
grenade 60bbe88ba7 feat(qwen4_exp): bank the experts instead of slicing them apart
CI / Config consistency (push) Successful in 14s
CI / Classify changes (push) Successful in 28s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 12s
CI / Clippy (push) Successful in 2m59s
CI / Test (push) Successful in 10m36s
CI / CUDA type-check (push) Successful in 20m31s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Slicing the checkpoint's two fused tensors into 512 per-expert modules
copies 241.6 GB at BF16, 5.03 GB per layer. That fits in neither
beast's 64 GB of VRAM nor its 123 GB of host RAM, and no quantisation
rescues it — #309 measured 59.4 GB of 63.7 used at 4.25 bpw with only
four layers' experts offloaded. The loader was the reason nothing could
be loaded.

So the block now takes the experts in whichever shape the checkpoint
gives, and views one per routed token. The routing is untouched: it was
already correct and duplicating it to get a second storage layout would
have invited exactly the kind of drift that is hard to see — softmax in
f32 over all 512, top-k, renormalise, sigmoid-gated shared expert.

Banking costs nothing per call. A slice along dim 0 of a contiguous
tensor is contiguous, and x.matmul(w.t()) is what Linear does anyway,
so the views feed the same GEMMs without a copy.

Because "we only changed where the bytes live" is a claim rather than a
fact, the same weights are expressed both ways and asserted to compute
the same function. Reading the gate and up halves in the wrong order is
pinned separately, since the banked path would get that wrong as
silently as the sliced one did.

It is also the layout a host-resident expert bank with a device-side
gather wants (#318), so that work starts from this rather than having
to undo the slicing first.

One consequence worth noting: MLP::from_weights has no caller outside
tests any more, so it goes back behind cfg(test).

Refs #307, #312, #318.
2026-09-02 17:06:44 +03:00
grenade 499ed5013e Merge branch 'feat/qwen4-exp-mtp-head'
build-prerelease / Resolve version stamps + change detection (push) Successful in 26s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m59s
build-prerelease / Test (push) Successful in 9m52s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 3m15s
build-prerelease / Build neuron-ada (push) Successful in 3m22s
build-prerelease / Build neuron-blackwell (push) Successful in 16m5s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m6s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m14s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m24s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 29s
2026-09-02 14:43:04 +03:00
grenade 0ab159e144 Merge branch 'docs/mtp-wiring-settled' 2026-09-02 14:43:04 +03:00
grenade e3a82e412d Merge branch 'feat/qwen4-exp-snapshot' 2026-09-02 14:43:04 +03:00
grenade 2c852c05cb feat(qwen4_exp): the MTP draft head
CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 18s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m32s
CI / Test (push) Successful in 9m9s
CI / CUDA type-check (push) Successful in 19m30s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The checkpoint ships a trained draft model — a second full-attention
decoder layer with its own indexer, its own 512-expert MoE and both
hyper-connections — so speculative decoding here is a wiring problem
rather than a training one. It reuses embed_tokens and lm_head, and
mtp.layers.0 is exactly the decoder layer we already have, so what is
new is only how the target's state and the next token's embedding
become an input for it.

Three parts of that were read from vLLM rather than inferred, and none
are recoverable from the tensor shapes.

pre_fc_norm_hidden is ungrouped: one normalisation over the whole
10240, where every other norm that wide in this architecture does four
of 2560. vLLM has a GroupedGemmaRMSNorm and deliberately does not use
it here. Our port spec asserted the opposite; grouping it rescales the
draft input in four independent pieces and shows up as a poor
acceptance rate rather than as an error, so it is pinned by value on an
input the two readings cannot agree on.

The hidden state is the pre-final-mixer stream, not the 2560 the LM
head sees — which is why the model now returns it. Both are "the hidden
state" and only one is right, so the head refuses the collapsed one by
width instead of broadcasting it into something plausible.

And the head emits both streams: the collapse for lm_head, and the
pre-mixer stream that its own next step consumes. A head returning only
the first cannot be chained past one token.

Building the test also settled something the loop will need: the head
owns its own KV and indexer caches, so it has to walk the prompt
alongside the target before it can draft. A head fed only the newest
token attends over an empty past, and here that is caught rather than
served, because the indexer's divergence check fires. Prefill is part
of speculation's cost, not just decode.

Refs #307, #313.
2026-09-02 12:43:32 +03:00
grenade 1862f0ad2d Merge branch 'docs/mtp-wiring-settled' into feat/qwen4-exp-mtp-head 2026-09-02 12:38:30 +03:00
grenade 559ff46f2a doc(qwen4_exp): settle the MTP wiring, and correct a norm this got wrong
CI / Classify changes (push) Successful in 16s
CI / Config consistency (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / CUDA type-check (push) Has been skipped
CI / Format (push) Has been skipped
CI / Clippy (push) Has been skipped
CI / Test (push) Has been skipped
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
§8 said every 10240-wide norm in the model is grouped over 2560 and
named mtp.pre_fc_norm_hidden among them. It is not. vLLM applies a
plain GemmaRMSNorm over the whole 10240 there, while using its
GroupedGemmaRMSNorm for the hyper-connection — both classes exist in
that tree, so the distinction is deliberate. Grouping it would rescale
the draft head's input in four independent pieces and surface as a poor
acceptance rate rather than as an error, which is the kind of wrong this
document exists to prevent.

§9 now carries the wiring instead of describing it as clear and leaving
it. Three parts of it are not recoverable from the tensor shapes: the
norm above; that the hidden state is the pre-final-mixer multi-stream
rather than the 2560 the LM head sees; and that the head emits both, the
collapsed stream for the head and the multi stream for its own next
step.

It also separates the wiring from the accept rule, which §9 had
conflated. The accept rule is greedy prefix match or the Leviathan
scheme and is not specific to this architecture.

Read from vllm/models/qwen4_exp/nvidia/mtp.py at 76ba321.

Refs #307, #313.
2026-09-02 12:34:19 +03:00
grenade bc57e8423e test(qwen4_exp): round-trip the snapshot, not the logits
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 15s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m30s
CI / Test (push) Successful in 10m19s
CI / CUDA type-check (push) Successful in 19m56s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Taken from llama.cpp's test for this architecture (their #27941), which
fixed the indexer cache dropping ext.x/ext.y on restore. Their note on
why the test is shaped that way is the point: comparing generated output
cannot see a field dropped on the way back in, because a dropped field
need not change the next token. Their fix moved 198 bytes of 335,692.

The existing test here compares logits after a restore and has exactly
that blind spot. This one snapshots, restores, snapshots again, and
requires the two to agree field for field — so anything
restore_kv_cache silently fails to put back shows up as a shape or a
value difference rather than as nothing at all.

The indexer cache gets its own assertion, since it is the field their
bug was in and the one this architecture adds.

Refs #307, #312.
2026-09-02 12:26:54 +03:00
grenade 25157ca473 feat(qwen4_exp): snapshot the state, all of it
CI / CUDA type-check (push) Blocked by required conditions
CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 17s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 12s
CI / Clippy (push) Successful in 2m30s
CI / Test (push) Successful in 10m3s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Prefix caching (#11) needs everything clear_kv_cache destroys, captured
at one token boundary. For this architecture that is more than the
attention K/V.

The QSA indexer keeps a second cache — one head of 128 dims per
position — and a snapshot holding only the K/V restores a layer whose
indexer still holds the previous request's keys. It then selects blocks
of somebody else's text and returns prose about it. So the two travel
together in a single FullSparse variant rather than as a variant plus
an optional field: the type makes capturing one without the other
unrepresentable, which is what clear_kv_cache already does at the other
end.

PLE contributes two things of different natures. The dilated conv's
left context is nine positions of a 10240-wide signal, deep-copied
rather than shared — the buffer is replaced by assignment today, so a
shallow clone would be sound, but narrow().contiguous() on an
already-contiguous slice is a view, which makes aliasing a property of
the data rather than of the code. That is the conditional the
GatedDeltaNet states are deep-copied to avoid. The carried n-gram ids
are two integers per sequence and decide which of 320 million rows the
next token reads.

Batched decode is explicitly refused rather than half-supported:
assemble_batch and extract_row bail on a FullSparse layer instead of
assembling it as plain attention, which would drop the indexer cache
and hand every row a selection computed from another request's keys.
That is the next piece of work, not a gap to be discovered by a wrong
answer.

The test moves the live state two decode steps past the snapshot before
restoring, so it measures a rewind rather than a coincidence, and its
control asserts that a cleared model cannot take the step at all —
which is what distinguishes restore from clear, and is now recorded
rather than assumed.

Refs #307, #312.
2026-09-02 12:13:17 +03:00
grenade b303a453cb refactor: move the cache snapshot out of arch/qwen3_5
The snapshot types are not qwen3_5's. tp_qwen3_5 already constructs
them, ModelArch::snapshot_kv_cache hands one back whatever the
architecture, and qwen4_exp is about to capture state through them
too. They lived under arch/qwen3_5/ because that was once the only arch
with a snapshot.

Pure move plus import updates — no behaviour change, and none intended.
It is a separate commit so the qwen4_exp extension that follows is
reviewable as the change it actually is, rather than as a diff mixed
with a rename.
2026-09-02 12:07:14 +03:00
grenade da749be5d7 Merge branch 'feat/rustingface-always'
build-prerelease / Resolve version stamps + change detection (push) Successful in 20s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m42s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m29s
build-prerelease / Build neuron-ampere (push) Successful in 2m32s
build-prerelease / Test (push) Successful in 6m40s
build-prerelease / Build neuron-blackwell (push) Successful in 15m11s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m3s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m8s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m11s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 24s
2026-09-02 09:48:13 +03:00
grenade c62bfccb3a Merge branch 'feat/qwen4-exp-harness' 2026-09-02 09:48:13 +03:00
grenade 2eb683441a feat(deploy): fetch every model through rustingface, not huggingface.co
CI / Config consistency (push) Successful in 10s
CI / Classify changes (push) Successful in 15s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 9s
CI / CUDA type-check (push) Successful in 19m1s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 9m40s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
rustingface has been running at rf.internal since before this change,
with a client token provisioned for exactly this purpose — the token
file carries the comment "helexa neurons -> rf.internal as HF_ENDPOINT
(added 2026-08-31)". Nothing on the fleet ever used it. neuron honours
HF_ENDPOINT (d99b4d91) and no host sets it, so every neuron has been
pulling from huggingface.co directly and caching only to its own disk.

This makes the registry a deploy-managed fact rather than a per-host
one. registry.conf sorts after local.conf, so the managed values win
over a hand-set HF_TOKEN, which is the point: the registry stops being
something each host is separately talked into (#287).

It also makes rustingface a hard dependency for every cold load on the
fleet, and that is the substance of the change rather than a cost of it.
An outage on our own host is one we can end; an outage at huggingface.co
is one we can only wait out. A serving stack that stops when someone
else's CDN does is not sovereign in any sense worth the word. The
comment in the workflow says so, and says not to add an upstream
fallback — that would read as a robustness improvement while deleting
the property the change exists for.

Three properties worth stating, because each is a way this could go
wrong quietly.

It is written only when the secret is non-empty. Pointing HF_ENDPOINT at
a registry we cannot authenticate to turns every cold load into a 401,
so a missing secret warns and changes nothing.

It is mode 0600, unlike model.conf's 0644, because it carries a
credential and a world-readable drop-in is a world-readable token. It is
staged under umask 077 so it is never briefly readable on the way in.
Deciding whether it changed therefore needs root to read it back, which
is a second sudoers line, scoped to that one path.

And the token never reaches a log: the applied-message prints the
endpoint only.

Refs #287.
2026-09-02 09:17:38 +03:00
28 changed files with 2125 additions and 237 deletions
+85 -15
View File
@@ -317,21 +317,82 @@ jobs:
# restart decision below, exactly like the drop-in.
neuron_toml_changed="${{ steps.neuronconf.outputs.changed }}"
# ── Desired per-model systemd drop-in ─────────────────────────
# model.conf carries NEURON_MAX_PROMPT_TOKENS so the context cap
# is deterministic per host and rolled out (with a restart) by
# ── Desired per-host systemd drop-in ──────────────────────────
# model.conf carries the environment neuron is started with, so
# it is deterministic per host and rolled out (with a restart) by
# this workflow, not hand-edited. It sorts after local.conf, so a
# deploy-managed value wins over any manual local override of the
# same variable. See doc/context-limits.md.
# same variable.
#
# RUST_LOG is fleet-uniform and unconditional. The code default
# is `info`, which is too coarse to diagnose a load or a routing
# decision after the fact, and before this it was `debug` on
# exactly one host because someone set it there by hand. A
# verbosity you have to remember to set is a verbosity you find
# out you did not have at the moment you needed it. Bumping a
# module (`debug,neuron::harness::candle=trace`) is a one-line
# change here, applied everywhere, with an audit trail.
#
# NEURON_MAX_PROMPT_TOKENS is per-host and stays conditional;
# see doc/context-limits.md.
conf=/etc/systemd/system/neuron.service.d/model.conf
config_changed=${neuron_toml_changed:-0}
rust_log="debug"
desired=$(printf '%s\n%s\n%s' \
"# Managed by .gitea/workflows/deploy.yml - do not edit by hand." \
"[Service]" \
"Environment=RUST_LOG=${rust_log}")
if [ -n "${max_prompt_tokens}" ]; then
desired=$(printf '%s\n%s\n%s\n%s' \
"# Managed by .gitea/workflows/deploy.yml - do not edit by hand." \
"# Per-model context cap; see doc/context-limits.md." \
"[Service]" \
desired=$(printf '%s\n%s' "${desired}" \
"Environment=NEURON_MAX_PROMPT_TOKENS=${max_prompt_tokens}")
[ "${desired}" = "$(cat "${conf}" 2>/dev/null || true)" ] || config_changed=1
fi
[ "${desired}" = "$(cat "${conf}" 2>/dev/null || true)" ] || config_changed=1
# ── Desired registry drop-in ──────────────────────────────────
# Every model fetch goes through rustingface rather than
# huggingface.co: it holds the upstream credential, and it
# caches to the operator's store, so the second host to want a
# checkpoint does not re-cross the internet for it.
#
# This makes rustingface a hard dependency for every cold load
# on the fleet. That is the point, not a side effect. An outage
# on our own host is one we can end; an outage at huggingface.co
# is one we can only wait out, and a serving stack that stops
# when someone else's CDN does is not sovereign in any sense
# worth the word.
#
# So do NOT add a fallback to huggingface.co here. It would read
# as a robustness improvement and would delete the property this
# exists for: the fleet would silently resume depending on a
# third party at exactly the moments that dependency matters.
#
# registry.conf sorts after local.conf, so the deploy-managed
# values win over a hand-set HF_TOKEN — which is the point:
# this is the mechanism that stops the registry being a
# hand-synced per-host decision (#287).
#
# Mode 0600, and compared by hash under sudo, because it
# carries a token. Written ONLY when the secret is non-empty:
# pointing HF_ENDPOINT at a registry we cannot authenticate to
# turns every cold load into a 401, which is strictly worse
# than fetching from upstream.
rf_conf=/etc/systemd/system/neuron.service.d/registry.conf
rf_token='${{ secrets.RUSTINGFACE_TOKEN }}'
rf_endpoint='https://rf.internal'
if [ -n "${rf_token}" ]; then
rf_desired=$(printf '%s\n%s\n%s\n%s\n%s\n%s' \
"# Managed by .gitea/workflows/deploy.yml - do not edit by hand." \
"# All model fetches go through rustingface, which holds the" \
"# upstream Hugging Face credential and caches to our store." \
"[Service]" \
"Environment=HF_ENDPOINT=${rf_endpoint}" \
"Environment=HF_TOKEN=${rf_token}")
rf_have=$(sudo /usr/bin/sha256sum "${rf_conf}" 2>/dev/null | cut -d' ' -f1 || true)
rf_want=$(printf '%s\n' "${rf_desired}" | sha256sum | cut -d' ' -f1)
[ "${rf_have}" = "${rf_want}" ] || config_changed=1
else
echo "WARNING: RUSTINGFACE_TOKEN is empty - leaving ${rf_conf} as-is."
echo " Fetches use whatever HF_ENDPOINT/HF_TOKEN the host already has."
fi
# ── Package version gate (manifest rationale: see deploy-cortex) ──
@@ -364,15 +425,24 @@ jobs:
echo "installed=${installed} published=${latest:-unknown} pkg_changed=${pkg_changed} config_changed=${config_changed} — deploying"
# Write the drop-in (staged in gitea_ci's dir, installed root-owned).
# Guarded on max_prompt_tokens as well as config_changed: since
# a neuron.toml change also sets config_changed, an entry with
# no max_prompt_tokens would otherwise reach this with `desired`
# unset.
if [ "${config_changed}" -eq 1 ] && [ -n "${max_prompt_tokens}" ]; then
# `desired` is always set now that RUST_LOG is unconditional, so
# config_changed alone is the guard.
if [ "${config_changed}" -eq 1 ]; then
printf '%s\n' "${desired}" > /var/lib/gitea_ci/model.conf
sudo /usr/bin/install -o root -g root -m 0644 -D /var/lib/gitea_ci/model.conf "${conf}"
rm -f /var/lib/gitea_ci/model.conf
echo "applied ${conf}: NEURON_MAX_PROMPT_TOKENS=${max_prompt_tokens}"
echo "applied ${conf}: RUST_LOG=${rust_log} NEURON_MAX_PROMPT_TOKENS=${max_prompt_tokens:-<unset>}"
fi
# Staged 0600 in gitea_ci's own dir before install, so the token
# is never briefly world-readable on the way in.
if [ -n "${rf_token}" ] && [ "${rf_have:-}" != "${rf_want:-}" ]; then
umask 077
printf '%s\n' "${rf_desired}" > /var/lib/gitea_ci/registry.conf
sudo /usr/bin/install -o root -g root -m 0600 -D /var/lib/gitea_ci/registry.conf "${rf_conf}"
rm -f /var/lib/gitea_ci/registry.conf
umask 022
echo "applied ${rf_conf}: HF_ENDPOINT=${rf_endpoint} (token not logged)"
fi
if systemctl is-active --quiet neuron.service; then
+8
View File
@@ -18,6 +18,14 @@ gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/neuron/neuron.toml
# NEURON_MAX_PROMPT_TOKENS: gitea_ci stages it in its own dir, then
# installs it root-owned. Exact source/dest paths; see doc/context-limits.md.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/install -o root -g root -m 0644 -D /var/lib/gitea_ci/model.conf /etc/systemd/system/neuron.service.d/model.conf
# deploy.yml also writes the registry drop-in, which points every model
# fetch at rustingface and carries the client token for it. Mode 0600
# rather than model.conf's 0644: this one holds a credential, and a
# world-readable systemd drop-in is a world-readable token.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/install -o root -g root -m 0600 -D /var/lib/gitea_ci/registry.conf /etc/systemd/system/neuron.service.d/registry.conf
# Reading it back to decide whether it changed needs root too, since
# 0600 excludes gitea_ci. Restricted to the one file.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/systemd/system/neuron.service.d/registry.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start neuron.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop neuron.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now neuron.service
+1
View File
@@ -22,3 +22,4 @@
pub mod qwen3_5;
pub mod qwen4_exp;
pub mod snapshot;
@@ -18,6 +18,7 @@ use candle_core::{Module, Tensor};
use candle_nn::var_builder::ShardedVarBuilder;
use std::sync::Arc;
use super::super::snapshot::LayerKvSnapshot;
use super::TextConfig;
use super::full_attn::Qwen3_5Attention;
use super::linear_attn::GatedDeltaNet;
@@ -25,7 +26,6 @@ use super::mlp::Qwen3_5MLP;
use super::moe::Qwen3_5MoeBlock;
use super::rmsnorm::Qwen3_5RmsNorm;
use super::rope::RotaryEmbedding;
use super::snapshot::LayerKvSnapshot;
/// One of the two attention flavours sitting in a decoder layer's
/// attention slot. Full-attention layers need the rotary table and
@@ -19,9 +19,12 @@ pub struct Qwen3_5MLP {
impl Qwen3_5MLP {
/// Construct directly from pre-built projections.
///
/// `qwen4_exp` ships its routed experts as fused 3D tensors rather
/// than per-expert modules, so its loader slices them and builds the
/// experts here — same SwiGLU, different storage.
/// Test-only again: `qwen4_exp` briefly built its 512 experts this
/// way before the fused tensors stayed banked (`Experts::Banked`),
/// which removed the last caller outside tests. The differential
/// test that proves banking neutral still needs it, to express the
/// same weights in the layout it is being compared against.
#[cfg(test)]
pub(crate) fn from_weights(gate_proj: Linear, up_proj: Linear, down_proj: Linear) -> Self {
Self {
gate_proj,
+11 -7
View File
@@ -79,7 +79,6 @@ pub mod mlp;
pub mod moe;
pub mod rmsnorm;
pub mod rope;
pub mod snapshot;
pub mod vision;
use decoder::Qwen3_5DecoderLayer;
@@ -534,15 +533,18 @@ impl Qwen3_5Model {
/// counter as one consistent prefix snapshot (#11). Only valid at
/// a token boundary — i.e. between forward calls, which is the
/// only time the caller can reach this anyway.
pub fn snapshot_kv_cache(&self) -> candle_core::Result<snapshot::KvCacheSnapshot> {
pub fn snapshot_kv_cache(
&self,
) -> candle_core::Result<crate::harness::arch::snapshot::KvCacheSnapshot> {
let layers = self
.layers
.iter()
.map(|l| l.snapshot_kv())
.collect::<candle_core::Result<Vec<_>>>()?;
Ok(snapshot::KvCacheSnapshot {
Ok(crate::harness::arch::snapshot::KvCacheSnapshot {
layers,
rope_delta: self.rope_delta,
ple: None,
})
}
@@ -550,7 +552,7 @@ impl Qwen3_5Model {
/// snapshot. The snapshot stays valid for further restores.
pub fn restore_kv_cache(
&mut self,
snap: &snapshot::KvCacheSnapshot,
snap: &crate::harness::arch::snapshot::KvCacheSnapshot,
) -> candle_core::Result<()> {
if snap.layers.len() != self.layers.len() {
candle_core::bail!(
@@ -582,7 +584,7 @@ impl Qwen3_5Model {
/// new token per batch row — with each row at its own sequence
/// position `positions[i]` (typically `prefix_lens[i] + step`).
/// The cache must hold batched state (see
/// `snapshot::assemble_batch`); `attn_mask` is the padding mask
/// `crate::harness::arch::snapshot::assemble_batch`); `attn_mask` is the padding mask
/// from [`Self::batch_decode_mask`] (or `None` when no row is
/// padded). Text-only: `rope_delta` is ignored — positions are
/// explicit and vision requests never enter the batch path.
@@ -1084,14 +1086,16 @@ impl Qwen3_5ForCausalLM {
}
/// See [`Qwen3_5Model::snapshot_kv_cache`].
pub fn snapshot_kv_cache(&self) -> candle_core::Result<snapshot::KvCacheSnapshot> {
pub fn snapshot_kv_cache(
&self,
) -> candle_core::Result<crate::harness::arch::snapshot::KvCacheSnapshot> {
self.base.snapshot_kv_cache()
}
/// See [`Qwen3_5Model::restore_kv_cache`].
pub fn restore_kv_cache(
&mut self,
snap: &snapshot::KvCacheSnapshot,
snap: &crate::harness::arch::snapshot::KvCacheSnapshot,
) -> candle_core::Result<()> {
self.base.restore_kv_cache(snap)
}
+447 -13
View File
@@ -29,18 +29,238 @@
//! replaces the loop behind the same `forward` signature.
use anyhow::{Context, Result};
use candle_core::{DType, Module, Tensor};
use candle_core::quantized::{GgmlDType, QMatMul, QTensor};
use candle_core::{DType, IndexOp, Module, Tensor};
use candle_nn::Linear;
use candle_nn::var_builder::ShardedVarBuilder;
use super::TextConfig;
use super::mlp::Qwen3_5MLP;
/// How a layer's routed experts are stored.
///
/// Two shapes, because the right one depends on whether the model fits.
///
/// `PerExpert` is the Qwen3-Next layout: one module per expert, sliced
/// out at load. Fine when the weights are device-resident anyway.
///
/// `Banked` keeps the checkpoint's own fused tensors whole and takes a
/// view per routed expert. `qwen4_exp` needs it: slicing its 512
/// experts into modules copies **241.6 GB** at BF16 — 5.03 GB per
/// layer — which fits in neither 64 GB of VRAM nor 123 GB of host RAM,
/// at any quantisation. It is also the layout a host-resident bank with
/// a device-side gather wants (#318), so the fused form is where that
/// work starts rather than something it would have to undo.
///
/// Banking costs nothing per call: a slice along dim 0 of a contiguous
/// tensor is contiguous, and `x.matmul(w.t())` is what `Linear` does
/// anyway, so the views feed the same GEMMs without a copy.
pub(crate) enum Experts {
PerExpert(Vec<Qwen3_5MLP>),
/// Quantised in situ at load (#315). One `(gate, up, down)` triple
/// per expert.
///
/// This is what makes `qwen4_exp` loadable at all: 120.8 B routed
/// parameters are 241.6 GB at BF16 and 67.9 GB at q4k, against
/// beast's 64 GB of VRAM and ~110 GB of usable host RAM. Nothing
/// about the arithmetic works until they shrink.
///
/// Quantising *from* the banked tensor is what keeps the peak
/// bounded: one layer's fused BF16 tensor is 5.03 GB, and it is
/// dropped once its 512 experts are quantised, so the full 241.6 GB
/// never exists at once.
Quantized(Vec<QuantizedExpert>),
Banked {
/// `(num_experts, 2 * intermediate, hidden)` — gate rows first,
/// then up. Reversing them computes `silu(up) * gate`, which is
/// a different function of the same weights (#312).
gate_up: Tensor,
/// `(num_experts, hidden, intermediate)`
down: Tensor,
intermediate: usize,
},
}
/// One expert's three projections, quantised.
pub(crate) struct QuantizedExpert {
gate: QMatMul,
up: QMatMul,
down: QMatMul,
}
/// The quantisation type actually usable for rows of `row` elements.
///
/// ggml quantises along the last dimension, and a row must be a whole
/// number of blocks. The k-quants (`Q2K`..`Q8K`) block on 256; the
/// older types block on 32. `qwen4_exp`'s `down_proj` reduces over
/// `moe_intermediate_size = 640`, which is 2.5 blocks of 256 — so a
/// `quant = "q4k"` load fails on the very first layer with
/// `quantized tensor must have their last dim divisible by block size
/// [2560, 640] 256`, having read 131 shards to get there.
///
/// **This table is llama.cpp's**, from `tensor_type_fallback` in
/// `src/llama-quant.cpp`, and the principle behind it is the part
/// worth stating: **every fallback spends bits rather than saving
/// them.** A k-quant carries a two-level scale hierarchy that the
/// legacy 32-block types do not, so it is better than a legacy type at
/// equal width — dropping `Q4K` (4.5 bpw) to `Q4_0` (also 4.5) would
/// hold the memory constant and take the quality hit silently.
/// Upstream declines that trade and pays a bit instead:
///
/// | requested | fallback | bpw |
/// |---|---|---|
/// | `Q2K`, `Q3K` | `Q4_0` | 2.63 / 3.44 -> 4.5 |
/// | `Q4K` | `Q5_0` | 4.5 -> 5.5 |
/// | `Q5K` | `Q5_1` | 5.5 -> 6.0 |
/// | `Q6K`, `Q8K` | `Q8_0` | 6.56 -> 8.5 |
///
/// For `qwen4_exp` that is not free: `down_proj` is exactly a third of
/// the 120.8 B routed parameters, so `Q5_0` rather than `Q4_0` costs
/// **+5.0 GB** across the expert bank (67.9 -> 73.0 GB). We take it
/// anyway. The cheaper table was an unmeasured divergence in the one
/// direction that flatters the constraint we are straining against,
/// and this matrix is the one a mixed k-quant scheme normally treats
/// as quality-critical. If the 5 GB turns out to buy nothing, #315 is
/// where that gets measured and reversed on evidence.
///
/// A row that divides 32 either — vanishingly rare, and impossible for
/// this checkpoint since 640 and 2560 are both multiples of 32 — falls
/// back to F16 as upstream does, loudly. It is a 3.5x blowup on that
/// tensor, so it must never be quiet.
pub(crate) fn quant_for_row(requested: GgmlDType, row: usize) -> candle_core::Result<GgmlDType> {
if row.is_multiple_of(requested.block_size()) {
return Ok(requested);
}
let fallback = match requested {
GgmlDType::Q2K | GgmlDType::Q3K => GgmlDType::Q4_0,
GgmlDType::Q4K => GgmlDType::Q5_0,
GgmlDType::Q5K => GgmlDType::Q5_1,
GgmlDType::Q6K | GgmlDType::Q8K => GgmlDType::Q8_0,
// Already a 32-block type (or not a quant at all): there is no
// smaller block to demote to, so the F16 check below decides.
other => other,
};
if row.is_multiple_of(fallback.block_size()) {
return Ok(fallback);
}
tracing::warn!(
?requested,
?fallback,
row,
"row divides neither the requested block size nor its 32-wide \
fallback; storing this tensor as F16 — 3.5x the bytes"
);
Ok(GgmlDType::F16)
}
impl Experts {
pub(crate) fn len(&self) -> usize {
match self {
Experts::PerExpert(v) => v.len(),
Experts::Banked { gate_up, .. } => gate_up.dims()[0],
Experts::Quantized(v) => v.len(),
}
}
/// Quantise a banked pair into per-expert triples.
///
/// Takes the fused tensors by reference and drops nothing itself —
/// the caller owns the peak, and should hold one layer at a time.
///
/// The two halves may not end up the same dtype: see
/// [`quant_for_row`]. `qwen4_exp`'s `down_proj` reduces over 640,
/// which no k-quant can block.
pub(crate) fn quantize_banked(
gate_up: &Tensor,
down: &Tensor,
intermediate: usize,
dtype: GgmlDType,
) -> candle_core::Result<Self> {
let n = gate_up.dims()[0];
// Decided once, from the shapes, rather than per expert —
// every expert in a layer has the same two row widths, and
// 512 identical log lines per layer would bury the fact.
let gate_up_dtype = quant_for_row(dtype, gate_up.dims()[2])?;
let down_dtype = quant_for_row(dtype, down.dims()[2])?;
if gate_up_dtype != dtype || down_dtype != dtype {
tracing::info!(
requested = ?dtype,
gate_up_row = gate_up.dims()[2],
gate_up_dtype = ?gate_up_dtype,
down_row = down.dims()[2],
down_dtype = ?down_dtype,
"expert quantisation: a row width does not divide the requested \
block size; substituting a 32-block type for that half"
);
}
let mut out = Vec::with_capacity(n);
for e in 0..n {
let gu = gate_up.i(e)?;
// The source must own its storage outright.
// `QTensor::quantize` reads `src.storage()` — the whole
// backing buffer, ignoring shape and offset — so a view into
// the fused tensor hands it every expert's bytes while the
// shape claims one, and candle panics on a block-count
// mismatch inside `from_float`. Neither `contiguous()` nor
// `copy()` saves you here: a slice along dim 0 is already
// contiguous, so both are free to hand back something still
// sharing the parent's buffer. Round-tripping through a Vec
// is the part that actually allocates, and it is also where
// the f32 the quantiser wants comes from.
let q = |t: Tensor, dtype: GgmlDType| -> candle_core::Result<QMatMul> {
let dims = t.dims().to_vec();
let vals = t.to_dtype(DType::F32)?.flatten_all()?.to_vec1::<f32>()?;
let owned = Tensor::from_vec(vals, dims, t.device())?;
QMatMul::from_qtensor(QTensor::quantize(&owned, dtype)?)
};
out.push(QuantizedExpert {
gate: q(gu.narrow(0, 0, intermediate)?, gate_up_dtype)?,
up: q(gu.narrow(0, intermediate, intermediate)?, gate_up_dtype)?,
down: q(down.i(e)?, down_dtype)?,
});
}
Ok(Experts::Quantized(out))
}
/// Expert `e` applied to the rows routed to it.
pub(crate) fn forward_one(&self, e: usize, xs: &Tensor) -> candle_core::Result<Tensor> {
match self {
Experts::PerExpert(v) => v[e].forward(xs),
Experts::Banked {
gate_up,
down,
intermediate,
} => {
let gu = gate_up.i(e)?;
let gate_w = gu.narrow(0, 0, *intermediate)?;
let up_w = gu.narrow(0, *intermediate, *intermediate)?;
let lhs = candle_nn::ops::silu(&xs.matmul(&gate_w.t()?)?)?;
let rhs = xs.matmul(&up_w.t()?)?;
(lhs * rhs)?.matmul(&down.i(e)?.t()?)
}
Experts::Quantized(v) => {
let q = &v[e];
// The quantised kernels accumulate in f32 whatever the
// activation dtype, so the cast is explicit here rather
// than implied — and the result comes back in the
// caller's dtype so the routing arithmetic upstream
// does not have to know which layout it got.
let dtype = xs.dtype();
let xs32 = xs.to_dtype(DType::F32)?;
let lhs = candle_nn::ops::silu(&q.gate.forward(&xs32)?)?;
let rhs = q.up.forward(&xs32)?;
q.down.forward(&(lhs * rhs)?)?.to_dtype(dtype)
}
}
}
}
pub struct Qwen3_5MoeBlock {
/// Router: `(num_experts, hidden)`, checkpoint name `mlp.gate`.
gate: Linear,
/// Routed experts, checkpoint names `mlp.experts.{i}.{gate,up,down}_proj`.
experts: Vec<Qwen3_5MLP>,
/// Routed experts. Per-expert modules for `qwen3_5`; the
/// checkpoint's fused tensors for `qwen4_exp` — see [`Experts`].
experts: Experts,
/// Always-on expert, `mlp.shared_expert.*`. `None` when the config
/// declares no shared expert (Qwen3-30B-A3B style).
shared_expert: Option<Qwen3_5MLP>,
@@ -62,7 +282,7 @@ impl Qwen3_5MoeBlock {
/// `qwen3_5` config.
pub(crate) fn from_parts(
gate: Linear,
experts: Vec<Qwen3_5MLP>,
experts: Experts,
shared_expert: Option<Qwen3_5MLP>,
shared_expert_gate: Option<Linear>,
num_experts_per_tok: usize,
@@ -129,7 +349,7 @@ impl Qwen3_5MoeBlock {
Ok(Self {
gate,
experts,
experts: Experts::PerExpert(experts),
shared_expert,
shared_expert_gate,
num_experts_per_tok: cfg.num_experts_per_tok,
@@ -201,13 +421,13 @@ impl Module for Qwen3_5MoeBlock {
)?;
let mut ys = xs_flat.zeros_like()?;
for (e, expert) in self.experts.iter().enumerate() {
for e in 0..self.experts.len() {
if tokens_for[e].is_empty() {
continue;
}
let rows = Tensor::new(tokens_for[e].as_slice(), xs.device())?;
let picked = xs_flat.index_select(&rows, 0)?;
let out = expert.forward(&picked)?;
let out = self.experts.forward_one(e, &picked)?;
let w = Tensor::new(weights_for[e].as_slice(), xs.device())?
.to_dtype(out.dtype())?
.reshape(((), 1))?;
@@ -252,7 +472,7 @@ mod tests {
let block = Qwen3_5MoeBlock {
gate: Linear::new(randn(&[n_exp, hidden]), None),
experts: (0..n_exp).map(|_| rand_mlp(hidden, inter)).collect(),
experts: Experts::PerExpert((0..n_exp).map(|_| rand_mlp(hidden, inter)).collect()),
shared_expert: Some(rand_mlp(hidden, inter)),
shared_expert_gate: Some(Linear::new(randn(&[1, hidden]), None)),
num_experts_per_tok: top_k,
@@ -283,8 +503,9 @@ mod tests {
let mut expect = vec![0f32; hidden];
for &e in selected {
let w = probs[e] / denom;
let out: Vec<f32> = block.experts[e]
.forward(&row)
let out: Vec<f32> = block
.experts
.forward_one(e, &row)
.unwrap()
.flatten_all()
.unwrap()
@@ -336,7 +557,7 @@ mod tests {
let (hidden, inter, n_exp) = (4, 2, 3);
let block = Qwen3_5MoeBlock {
gate: Linear::new(randn(&[n_exp, hidden]), None),
experts: (0..n_exp).map(|_| rand_mlp(hidden, inter)).collect(),
experts: Experts::PerExpert((0..n_exp).map(|_| rand_mlp(hidden, inter)).collect()),
shared_expert: None,
shared_expert_gate: None,
num_experts_per_tok: 1,
@@ -368,8 +589,9 @@ mod tests {
.max_by(|&a, &b| exps[a].partial_cmp(&exps[b]).unwrap())
.unwrap();
let w = exps[best] / sum;
let out: Vec<f32> = block.experts[best]
.forward(&xs.reshape(((), hidden)).unwrap())
let out: Vec<f32> = block
.experts
.forward_one(best, &xs.reshape(((), hidden)).unwrap())
.unwrap()
.flatten_all()
.unwrap()
@@ -383,4 +605,216 @@ mod tests {
);
}
}
/// Banking is a storage change, so the two layouts must compute the
/// same function. Same weights, expressed both ways, asserted
/// identical — otherwise "we only changed where the bytes live" is
/// a claim rather than a fact.
#[test]
fn banked_experts_equal_per_expert_modules() {
let (n_experts, hidden, inter) = (3usize, 4usize, 2usize);
// One source of truth for the weights, laid out both ways.
let gate_up = randn(&[n_experts, inter * 2, hidden]);
let down = randn(&[n_experts, hidden, inter]);
let per_expert: Vec<Qwen3_5MLP> = (0..n_experts)
.map(|e| {
let gu = gate_up.i(e).unwrap();
Qwen3_5MLP::from_weights(
Linear::new(gu.narrow(0, 0, inter).unwrap().contiguous().unwrap(), None),
Linear::new(
gu.narrow(0, inter, inter).unwrap().contiguous().unwrap(),
None,
),
Linear::new(down.i(e).unwrap().contiguous().unwrap(), None),
)
})
.collect();
let sliced = Experts::PerExpert(per_expert);
let banked = Experts::Banked {
gate_up,
down,
intermediate: inter,
};
assert_eq!(sliced.len(), banked.len());
let xs = randn(&[5, hidden]);
for e in 0..n_experts {
let a: Vec<f32> = sliced
.forward_one(e, &xs)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
let b: Vec<f32> = banked
.forward_one(e, &xs)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
assert_eq!(a.len(), b.len());
for (x, y) in a.iter().zip(b.iter()) {
assert!(
(x - y).abs() < 1e-5,
"expert {e} differs between layouts: {a:?} vs {b:?}"
);
}
}
}
/// And the halves are not interchangeable: reading `gate_up` the
/// other way round computes silu(up) * gate, which the banked path
/// would do just as silently as the sliced one.
#[test]
fn banked_reads_the_gate_before_the_up() {
let dev = Device::Cpu;
// gate row picks channel 0, up row picks channel 1.
let gate_up = Tensor::from_vec(vec![1.0f32, 0.0, 0.0, 1.0], (1, 2, 2), &dev).unwrap();
let down = Tensor::from_vec(vec![1.0f32, 0.0], (1, 2, 1), &dev).unwrap();
let banked = Experts::Banked {
gate_up,
down,
intermediate: 1,
};
let xs = Tensor::from_vec(vec![1.0f32, 2.0], (1, 2), &dev).unwrap();
let got: Vec<f32> = banked
.forward_one(0, &xs)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
// silu(gate.x) * (up.x) = silu(1) * 2
let want = (1.0f32 / (1.0 + (-1.0f32).exp())) * 2.0;
assert!((got[0] - want).abs() < 1e-5, "got {got:?} want {want}");
// the swapped reading would be silu(2) * 1
let swapped = (2.0f32 / (1.0 + (-2.0f32).exp())) * 1.0;
assert!(
(got[0] - swapped).abs() > 1e-3,
"this is silu(up) * gate: {got:?}"
);
}
/// In-situ quantisation has to approximate the dense expert, not
/// merely produce numbers of the right shape.
///
/// Dimensions are 256 because the k-quants block on 256 elements —
/// a tiny fixture silently cannot be quantised at all, which is the
/// first thing that bites when this is tried on a toy model.
///
/// `qwen4_exp`'s real expert widths, which no k-quant can block.
///
/// `moe_intermediate_size` is 640, so `down_proj` is `[2560, 640]`
/// and its 640-element rows are 2.5 blocks of 256. The load died
/// on beast at decoder layer 0 with `quantized tensor must have
/// their last dim divisible by block size [2560, 640] 256`, after
/// resolving all 131 shards.
///
/// The fixture that missed it was 256 wide in both dimensions —
/// the smallest width a k-quant accepts — so every row divided
/// and the orientation could not matter. This one uses the
/// checkpoint's own numbers, scaled down only in expert count.
#[test]
fn the_real_expert_widths_quantise_despite_a_640_wide_reduction() {
assert_eq!(
quant_for_row(GgmlDType::Q4K, 2560).unwrap(),
GgmlDType::Q4K,
"gate_up reduces over hidden_size, which is 10 whole blocks"
);
assert_eq!(
quant_for_row(GgmlDType::Q4K, 640).unwrap(),
GgmlDType::Q5_0,
"down reduces over moe_intermediate_size, which is not"
);
// llama.cpp's `tensor_type_fallback` table, which spends bits
// rather than saving them: a legacy 32-block type is worse than
// a k-quant at equal width, so upstream pays one to avoid the
// quality cliff. Q4_0 above would be 4.5 bpw like Q4K and would
// save 5.0 GB across the expert bank — that is the trade we
// declined, and this is where it is pinned.
assert_eq!(quant_for_row(GgmlDType::Q5K, 640).unwrap(), GgmlDType::Q5_1);
assert_eq!(quant_for_row(GgmlDType::Q2K, 640).unwrap(), GgmlDType::Q4_0);
assert_eq!(quant_for_row(GgmlDType::Q3K, 640).unwrap(), GgmlDType::Q4_0);
assert_eq!(quant_for_row(GgmlDType::Q6K, 640).unwrap(), GgmlDType::Q8_0);
assert_eq!(
quant_for_row(GgmlDType::Q8_0, 640).unwrap(),
GgmlDType::Q8_0
);
// 100 divides neither 256 nor 32: F16, as upstream does.
assert_eq!(quant_for_row(GgmlDType::Q4K, 100).unwrap(), GgmlDType::F16);
// And the whole path runs at those widths.
let (n_experts, hidden, inter) = (2usize, 2560usize, 640usize);
let gate_up = randn(&[n_experts, inter * 2, hidden]);
let down = randn(&[n_experts, hidden, inter]);
let quantised = Experts::quantize_banked(&gate_up, &down, inter, GgmlDType::Q4K)
.expect("the real widths must quantise");
let xs = randn(&[2, hidden]);
let got = quantised.forward_one(0, &xs).expect("forward");
assert_eq!(got.dims(), &[2, hidden]);
assert!(
got.flatten_all()
.unwrap()
.to_vec1::<f32>()
.unwrap()
.iter()
.all(|v| v.is_finite()),
"a substituted dtype must still produce finite output"
);
}
/// The assertion is relative error against the dense output, with a
/// noise baseline: a q8_0 expert should be close, a q4k expert
/// looser, and both far nearer the truth than an unrelated expert
/// is. Without that last comparison a "quantised" path that
/// returned anything smoothly wrong would pass.
#[test]
fn quantised_experts_approximate_the_dense_ones() {
let (n_experts, hidden, inter) = (2usize, 256usize, 256usize);
let gate_up = randn(&[n_experts, inter * 2, hidden]);
let down = randn(&[n_experts, hidden, inter]);
let banked = Experts::Banked {
gate_up: gate_up.clone(),
down: down.clone(),
intermediate: inter,
};
let xs = randn(&[4, hidden]);
let rel_err = |a: &[f32], b: &[f32]| -> f32 {
let num: f32 = a.iter().zip(b).map(|(x, y)| (x - y) * (x - y)).sum();
let den: f32 = a.iter().map(|x| x * x).sum::<f32>().max(1e-12);
(num / den).sqrt()
};
let out = |e: &Experts, i: usize| -> Vec<f32> {
e.forward_one(i, &xs)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap()
};
let dense0 = out(&banked, 0);
// Baseline: a different expert on the same input. Any quantised
// reading must beat this by a wide margin.
let noise = rel_err(&dense0, &out(&banked, 1));
for (dtype, tol) in [(GgmlDType::Q8_0, 0.05f32), (GgmlDType::Q4K, 0.20f32)] {
let q = Experts::quantize_banked(&gate_up, &down, inter, dtype).unwrap();
assert_eq!(q.len(), n_experts);
let err = rel_err(&dense0, &out(&q, 0));
assert!(err < tol, "{dtype:?} relative error {err:.4} exceeds {tol}");
assert!(
err < noise / 2.0,
"{dtype:?} error {err:.4} is not clearly better than an unrelated \
expert ({noise:.4}) — the quantised path may not be reading \
this expert at all"
);
}
}
}
@@ -41,11 +41,14 @@ use crate::harness::arch::qwen3_5::linear_attn::GatedDeltaNet;
use crate::harness::arch::qwen3_5::moe::Qwen3_5MoeBlock;
use crate::harness::arch::qwen3_5::rope::RotaryEmbedding;
use candle_core::quantized::GgmlDType;
use super::config::TextConfig;
use super::full_attn::Attention;
use super::hyper::HyperConnection;
use super::ple::PleBlock;
use super::{linear_attn, moe};
use crate::harness::arch::snapshot::LayerKvSnapshot;
/// The attention slot: one layer in four is full attention with the QSA
/// indexer, the rest are the recurrent delta rule.
@@ -71,6 +74,7 @@ impl DecoderLayer {
cfg: &TextConfig,
rotary: Arc<RotaryEmbedding>,
layer_idx: usize,
quant: Option<GgmlDType>,
vb: &ShardedVarBuilder,
) -> Result<Self> {
let layer_type = cfg
@@ -83,6 +87,30 @@ impl DecoderLayer {
cfg.layer_types.len()
)
})?;
Self::load_typed(
cfg,
rotary,
layer_type,
cfg.ple_layers().contains(&layer_idx),
quant,
vb,
)
}
/// Load a layer whose type the caller knows, rather than one looked
/// up in `layer_types`.
///
/// The MTP head's layer (#313) needs this: it lives under
/// `mtp.layers.0` with its own single-entry type list, and it
/// carries no PLE wherever the main model puts one.
pub fn load_typed(
cfg: &TextConfig,
rotary: Arc<RotaryEmbedding>,
layer_type: &str,
with_ple: bool,
quant: Option<GgmlDType>,
vb: &ShardedVarBuilder,
) -> Result<Self> {
let mixer = match layer_type {
"full_attention" => {
Mixer::Full(Box::new(Attention::load(cfg, rotary, &vb.pp("self_attn"))?))
@@ -91,8 +119,8 @@ impl DecoderLayer {
Mixer::Linear(Box::new(linear_attn::load(cfg, &vb.pp("linear_attn"))?))
}
other => anyhow::bail!(
"unknown layer_type '{other}' for layer {layer_idx} (expected \
'full_attention' or 'linear_attention')"
"unknown layer_type '{other}' (expected 'full_attention' or \
'linear_attention')"
),
};
@@ -109,7 +137,7 @@ impl DecoderLayer {
)
};
let ple = if cfg.ple_layers().contains(&layer_idx) {
let ple = if with_ple {
Some(PleBlock::load(
&vb.pp("ple"),
cfg.hidden_size,
@@ -127,7 +155,7 @@ impl DecoderLayer {
attn_hc: hc("attn_hyper_connection")?,
mlp_hc: hc("mlp_hyper_connection")?,
mixer,
mlp: moe::load(cfg, &vb.pp("mlp"))?,
mlp: moe::load(cfg, quant, &vb.pp("mlp"))?,
ple,
})
}
@@ -190,6 +218,65 @@ impl DecoderLayer {
Ok(())
}
/// Capture this layer's attention-side state.
pub fn snapshot_kv(&self) -> candle_core::Result<LayerKvSnapshot> {
Ok(match &self.mixer {
Mixer::Full(attn) => {
let (kv, indexer_keys) = attn.snapshot_kv();
LayerKvSnapshot::FullSparse { kv, indexer_keys }
}
Mixer::Linear(net) => {
let (conv_state, recurrent_state) = net.snapshot_state()?;
LayerKvSnapshot::Linear {
conv_state,
recurrent_state,
}
}
})
}
/// Replace this layer's attention-side state. A variant mismatch
/// means the snapshot came from a different model, or from the same
/// model with a different `layer_types` — either way, restoring it
/// would put a recurrent state into an attention layer.
pub fn restore_kv(&mut self, snap: &LayerKvSnapshot) -> candle_core::Result<()> {
match (&mut self.mixer, snap) {
(Mixer::Full(attn), LayerKvSnapshot::FullSparse { kv, indexer_keys }) => {
attn.restore_kv(kv.as_ref(), indexer_keys.as_ref())
}
(
Mixer::Linear(net),
LayerKvSnapshot::Linear {
conv_state,
recurrent_state,
},
) => net.restore_state(conv_state.as_ref(), recurrent_state.as_ref()),
_ => candle_core::bail!(
"restore_kv: snapshot layer kind does not match this layer's mixer kind"
),
}
}
/// The PLE block's rolling conv context, if this layer carries one.
pub fn snapshot_ple(&self) -> candle_core::Result<Option<Tensor>> {
match &self.ple {
Some(block) => block.snapshot_state(),
None => Ok(None),
}
}
/// Restore the PLE conv context. Refuses a snapshot that disagrees
/// with this layer about whether PLE lives here at all.
pub fn restore_ple(&mut self, conv_state: Option<&Tensor>) -> candle_core::Result<()> {
match &mut self.ple {
Some(block) => block.restore_state(conv_state),
None if conv_state.is_none() => Ok(()),
None => candle_core::bail!(
"restore_ple: snapshot carries PLE state but this layer has no PLE block"
),
}
}
pub fn has_ple(&self) -> bool {
self.ple.is_some()
}
@@ -170,6 +170,34 @@ impl Attention {
Ok(self.o_proj.forward(&gated)?)
}
/// Capture both caches at one token boundary.
///
/// The same reason `clear_kv_cache` drops them together: a
/// snapshot holding only the attention K/V restores a layer whose
/// indexer still holds the previous request's keys, and the
/// selection it computes from them is wrong without being invalid.
pub fn snapshot_kv(&self) -> (Option<(Tensor, Tensor)>, Option<Tensor>) {
let kv = match (self.kv_cache.k(), self.kv_cache.v()) {
(Some(k), Some(v)) => Some((k.clone(), v.clone())),
_ => None,
};
(kv, self.indexer.snapshot_keys())
}
/// Replace both caches from a snapshot.
pub fn restore_kv(
&mut self,
kv: Option<&(Tensor, Tensor)>,
indexer_keys: Option<&Tensor>,
) -> candle_core::Result<()> {
self.kv_cache.reset();
if let Some((k, v)) = kv {
self.kv_cache.append(k, v)?;
}
self.indexer.restore_keys(indexer_keys);
Ok(())
}
/// Both caches, together. Dropping only one leaves the indexer
/// selecting blocks of a previous request's text.
pub fn clear_kv_cache(&mut self) {
@@ -25,6 +25,8 @@
//! from `qwen3_5` with two settings corrected.
//! - [`model`] — embeddings in, logits out; four streams, one
//! n-gram lookup, and no final norm.
//! - [`mtp`] — the shipped draft head: one more decoder layer, fed
//! the target's pre-mixer stream and the next token's embedding.
//! - [`moe`] — the FFN every layer has, loaded from fused expert
//! tensors instead of 512 modules.
//! - [`ple`] — the hashed n-gram table on layer 1: how its rows
@@ -39,5 +41,6 @@ pub mod hyper;
pub mod linear_attn;
pub mod model;
pub mod moe;
pub mod mtp;
pub mod ple;
pub mod qsa;
@@ -34,6 +34,7 @@
//! See `doc/qwen4_exp-port-spec.md`.
use anyhow::{Context, Result, ensure};
use candle_core::quantized::GgmlDType;
use candle_core::{DType, Device, IndexOp, Module, Tensor};
use candle_nn::var_builder::ShardedVarBuilder;
use candle_nn::{Embedding, Linear};
@@ -45,6 +46,7 @@ use super::config::{Config, TextConfig};
use super::decoder::DecoderLayer;
use super::hyper::HyperConnection;
use super::ple::{NGramHasher, NGramTable, ShardedNGramTable};
use crate::harness::arch::snapshot::{KvCacheSnapshot, PleSnapshot};
/// The hashed n-gram lookup, hoisted to the model because it is a
/// function of the input ids alone.
@@ -89,6 +91,18 @@ impl NGramEmbedding {
Ok(stacked)
}
/// The carried ids, for a prefix snapshot. Two integers per batch
/// row — small, but they decide which n-grams the next token
/// hashes, so a restore that forgets them silently addresses the
/// wrong rows of a 320-million-row table.
fn snapshot(&self) -> Vec<Vec<i64>> {
self.carried.clone()
}
fn restore(&mut self, carried: &[Vec<i64>]) {
self.carried = carried.to_vec();
}
fn reset(&mut self) {
for row in &mut self.carried {
row.clear();
@@ -111,9 +125,14 @@ pub struct Qwen4ExpForCausalLM {
}
impl Qwen4ExpForCausalLM {
/// `quant` quantises the routed experts in situ (#315). `None`
/// keeps the checkpoint's precision, which for this architecture is
/// useful only on a model small enough to hold: 241.6 GB of BF16
/// experts fit neither 64 GB of VRAM nor 123 GB of host RAM.
pub fn load(
cfg: &Config,
dtype: DType,
quant: Option<GgmlDType>,
device: &Device,
vb: &ShardedVarBuilder,
) -> Result<Self> {
@@ -134,7 +153,7 @@ impl Qwen4ExpForCausalLM {
let mut layers = Vec::with_capacity(text.num_hidden_layers);
for i in 0..text.num_hidden_layers {
layers.push(
DecoderLayer::load(text, rotary.clone(), i, &layers_vb.pp(i))
DecoderLayer::load(text, rotary.clone(), i, quant, &layers_vb.pp(i))
.with_context(|| format!("load decoder layer {i}"))?,
);
}
@@ -177,6 +196,24 @@ impl Qwen4ExpForCausalLM {
/// `input_ids` is `(B, T)`; `offset` is the sequence position the
/// first of them sits at. Returns logits `(B, T, vocab)`.
pub fn forward(&mut self, input_ids: &Tensor, offset: usize) -> Result<Tensor> {
Ok(self.forward_inner(input_ids, offset)?.0)
}
/// Logits, plus the four-stream residual as it stood *before* the
/// mixer collapsed it.
///
/// The MTP draft head consumes that pre-mixer stream, not the 2560
/// the LM head sees (#313), so it has to leave the model here rather
/// than be reconstructed from the logits — which it could not be.
pub fn forward_with_hidden(
&mut self,
input_ids: &Tensor,
offset: usize,
) -> Result<(Tensor, Tensor)> {
self.forward_inner(input_ids, offset)
}
fn forward_inner(&mut self, input_ids: &Tensor, offset: usize) -> Result<(Tensor, Tensor)> {
let (batch, seq_len) = input_ids.dims2()?;
// Four identical streams — the residual this architecture
@@ -209,7 +246,92 @@ impl Qwen4ExpForCausalLM {
// The mixer's hc_norm is the only normalisation before the head.
let x = self.mixer.collapse(&h)?;
Ok(self.lm_head.forward(&x)?)
Ok((self.lm_head.forward(&x)?, h))
}
/// Capture every piece of per-request state at one token boundary:
/// the attention K/V and the QSA indexer cache per full-attention
/// layer, the GatedDeltaNet conv and recurrent states per linear
/// layer, and PLE's conv context and carried ids.
///
/// There is no `rope_delta` here — this architecture's text path
/// takes positions from the caller's offset rather than tracking a
/// vision-induced skew, so the field is recorded as zero. When the
/// vision tower lands (#314) that stops being true.
pub fn snapshot_kv_cache(&self) -> candle_core::Result<KvCacheSnapshot> {
let layers = self
.layers
.iter()
.map(|l| l.snapshot_kv())
.collect::<candle_core::Result<Vec<_>>>()?;
let conv_state = self
.layers
.iter()
.map(|l| l.snapshot_ple())
.collect::<candle_core::Result<Vec<_>>>()?
.into_iter()
.flatten()
.next();
Ok(KvCacheSnapshot {
layers,
rope_delta: 0,
ple: self.ngram.as_ref().map(|n| PleSnapshot {
conv_state,
carried_ids: n.snapshot(),
}),
})
}
/// Replace the live state from a snapshot. The snapshot stays valid
/// for further restores.
pub fn restore_kv_cache(&mut self, snap: &KvCacheSnapshot) -> candle_core::Result<()> {
if snap.layers.len() != self.layers.len() {
candle_core::bail!(
"restore_kv_cache: snapshot has {} layers, model has {}",
snap.layers.len(),
self.layers.len()
);
}
// A snapshot from a model whose PLE layer sits elsewhere would
// otherwise restore the conv context onto the wrong layer and
// leave this one holding the previous request's.
if snap.ple.is_some() != self.ngram.is_some() {
candle_core::bail!(
"restore_kv_cache: snapshot {} PLE state, model {} a PLE layer",
if snap.ple.is_some() {
"carries"
} else {
"has no"
},
if self.ngram.is_some() {
"has"
} else {
"has none"
}
);
}
for (layer, layer_snap) in self.layers.iter_mut().zip(snap.layers.iter()) {
layer.restore_kv(layer_snap)?;
}
let conv = snap.ple.as_ref().and_then(|p| p.conv_state.as_ref());
for layer in self.layers.iter_mut() {
if layer.has_ple() {
layer.restore_ple(conv)?;
}
}
if let (Some(ngram), Some(ple)) = (&mut self.ngram, snap.ple.as_ref()) {
ngram.restore(&ple.carried_ids);
}
Ok(())
}
/// `embed_tokens` applied to `input_ids`.
///
/// The MTP head reuses this embedding
/// (`mtp_use_dedicated_embeddings: false`) but does not own it, so
/// the speculative loop borrows it from the target model.
pub fn embed(&self, input_ids: &Tensor) -> Result<Tensor> {
Ok(self.embed_tokens.forward(input_ids)?)
}
pub fn clear_kv_cache(&mut self) -> Result<()> {
@@ -320,6 +442,7 @@ fn rotary_for(cfg: &TextConfig, dtype: DType, device: &Device) -> Result<RotaryE
#[cfg(test)]
mod tests {
use super::*;
use crate::harness::arch::snapshot::LayerKvSnapshot;
use candle_nn::var_builder::ShardedSafeTensors;
use std::collections::HashMap;
@@ -535,6 +658,95 @@ mod tests {
}
}
}
// ---- the MTP draft head ----
// A second full-attention layer plus the two fc_* projections
// and their pre-norms. Emitted unconditionally: the head is what
// #313 builds on and the fixture is the only place its loader
// gets exercised.
{
let m = "mtp";
t.insert(format!("{m}.pre_fc_norm_embedding.weight"), rand(vec![h]));
t.insert(format!("{m}.pre_fc_norm_hidden.weight"), rand(vec![wide]));
t.insert(format!("{m}.fc_embedding.weight"), rand(vec![h, h]));
t.insert(format!("{m}.fc_hidden.weight"), rand(vec![h, h]));
t.insert(
format!("{m}.hyper_connection_mixer.hc_norm.weight"),
rand(vec![wide]),
);
t.insert(
format!("{m}.hyper_connection_mixer.input_mix_weight_down.weight"),
rand(vec![cfg.hc_lowrank, wide]),
);
t.insert(
format!("{m}.hyper_connection_mixer.input_mix_weight_up.weight"),
rand(vec![wide, cfg.hc_lowrank]),
);
let l = format!("{m}.layers.0");
for which in ["attn_hyper_connection", "mlp_hyper_connection"] {
t.insert(format!("{l}.{which}.hc_norm.weight"), rand(vec![wide]));
t.insert(
format!("{l}.{which}.input_mix_weight_down.weight"),
rand(vec![cfg.hc_lowrank, wide]),
);
t.insert(
format!("{l}.{which}.input_mix_weight_up.weight"),
rand(vec![wide, cfg.hc_lowrank]),
);
t.insert(
format!("{l}.{which}.block_inject_weight.weight"),
rand(vec![hc, wide]),
);
}
let a = format!("{l}.self_attn");
let heads = cfg.num_attention_heads * cfg.head_dim;
t.insert(format!("{a}.q_proj.weight"), rand(vec![heads * 2, h]));
t.insert(
format!("{a}.k_proj.weight"),
rand(vec![cfg.num_key_value_heads * cfg.head_dim, h]),
);
t.insert(
format!("{a}.v_proj.weight"),
rand(vec![cfg.num_key_value_heads * cfg.head_dim, h]),
);
t.insert(format!("{a}.o_proj.weight"), rand(vec![h, heads]));
t.insert(format!("{a}.q_norm.weight"), rand(vec![cfg.head_dim]));
t.insert(format!("{a}.k_norm.weight"), rand(vec![cfg.head_dim]));
let fused = (cfg.indexer_n_heads + cfg.indexer_kv_heads) * cfg.indexer_head_dim;
t.insert(
format!("{a}.indexer.index_qk_proj.weight"),
rand(vec![fused, h]),
);
t.insert(
format!("{a}.indexer.q_layernorm.weight"),
rand(vec![cfg.indexer_head_dim]),
);
t.insert(
format!("{a}.indexer.k_layernorm.weight"),
rand(vec![cfg.indexer_head_dim]),
);
let mm = format!("{l}.mlp");
t.insert(format!("{mm}.gate.weight"), rand(vec![cfg.num_experts, h]));
t.insert(
format!("{mm}.experts.gate_up_proj"),
rand(vec![cfg.num_experts, cfg.moe_intermediate_size * 2, h]),
);
t.insert(
format!("{mm}.experts.down_proj"),
rand(vec![cfg.num_experts, h, cfg.moe_intermediate_size]),
);
for pj in ["gate_proj", "up_proj"] {
t.insert(
format!("{mm}.shared_expert.{pj}.weight"),
rand(vec![cfg.shared_expert_intermediate_size, h]),
);
}
t.insert(
format!("{mm}.shared_expert.down_proj.weight"),
rand(vec![h, cfg.shared_expert_intermediate_size]),
);
t.insert(format!("{mm}.shared_expert_gate.weight"), rand(vec![1, h]));
}
t
}
@@ -545,7 +757,7 @@ mod tests {
candle_core::safetensors::save(&tensors(&cfg.text_config), &path).unwrap();
let vb =
unsafe { ShardedSafeTensors::var_builder(&[&path], DType::F32, &Device::Cpu).unwrap() };
let model = Qwen4ExpForCausalLM::load(&cfg, DType::F32, &Device::Cpu, &vb).unwrap();
let model = Qwen4ExpForCausalLM::load(&cfg, DType::F32, None, &Device::Cpu, &vb).unwrap();
(dir, model)
}
@@ -603,4 +815,279 @@ mod tests {
assert!(ngram.table.gather(&[(TABLE_ROWS - 1) as i64]).is_ok());
assert!(ngram.table.gather(&[TABLE_ROWS as i64]).is_err());
}
fn tensor_vec(t: &Tensor) -> Vec<f32> {
t.flatten_all()
.unwrap()
.to_dtype(DType::F32)
.unwrap()
.to_vec1()
.unwrap()
}
fn logits_vec(t: &Tensor) -> Vec<f32> {
t.flatten_all().unwrap().to_vec1().unwrap()
}
/// The property prefix caching rests on: a snapshot must *rewind*
/// the live state, not merely coexist with it.
///
/// So the state is deliberately moved past the snapshot before the
/// restore — two further decode steps — and the restored step is
/// then required to reproduce the original exactly. An
/// implementation that captured nothing would not reproduce it; one
/// that restored only the attention K/V and left the indexer cache
/// at seven positions would trip the divergence guard rather than
/// return wrong numbers.
///
/// The control is that a *cleared* model cannot take this step at
/// all: with an empty cache and a query at position 4 the guard
/// fires. That is what distinguishes restore from clear here, and
/// it is asserted rather than assumed.
#[test]
fn a_restored_snapshot_rewinds_rather_than_merely_continuing() {
let (_dir, mut model) = build();
let prompt = Tensor::from_vec(vec![5u32, 9, 3, 12], (1, 4), &Device::Cpu).unwrap();
let next = Tensor::from_vec(vec![7u32], (1, 1), &Device::Cpu).unwrap();
model.forward(&prompt, 0).unwrap();
let snap = model.snapshot_kv_cache().unwrap();
let want = logits_vec(&model.forward(&next, 4).unwrap());
// Move the live state well past the snapshot.
model.forward(&next, 5).unwrap();
model.forward(&next, 6).unwrap();
model.restore_kv_cache(&snap).unwrap();
let got = logits_vec(&model.forward(&next, 4).unwrap());
assert_eq!(got, want, "restore must rewind to the snapshot's boundary");
// Control: a cleared model cannot even take the step, so the
// restore above did something a clear does not.
model.clear_kv_cache().unwrap();
// `{:#}` to walk the context chain: the forward wraps the
// failure in "layer 1", and the guard is what it wraps.
let err = format!("{:#}", model.forward(&next, 4).unwrap_err());
assert!(
err.contains("diverged"),
"expected the cache-divergence guard, got: {err}"
);
}
/// The snapshot carries PLE's state, and PLE is the piece with two
/// distinct kinds of it — a rolling conv context and the carried
/// n-gram ids. Restoring one without the other reads the wrong rows
/// of the table, so the snapshot has to hold both.
#[test]
fn the_snapshot_carries_ple_state() {
let (_dir, mut model) = build();
let prompt = Tensor::from_vec(vec![5u32, 9, 3, 12], (1, 4), &Device::Cpu).unwrap();
model.forward(&prompt, 0).unwrap();
let snap = model.snapshot_kv_cache().unwrap();
let ple = snap.ple.as_ref().expect("layer 1 carries PLE");
assert!(ple.conv_state.is_some(), "the dilated conv context");
assert_eq!(ple.carried_ids.len(), 1, "one batch row");
assert_eq!(
ple.carried_ids[0].len(),
2,
"ngram_size - 1 ids carried across the step"
);
assert!(snap.size_bytes() > 0);
}
/// Round-trip the snapshot itself, not the logits it leads to.
///
/// Taken from llama.cpp's `test-save-load-state` for this
/// architecture (their #27941, which fixed the indexer cache
/// dropping `ext.x`/`ext.y` on restore): comparing *generated
/// output* cannot see a field dropped on the way back in, because a
/// dropped field need not change the next token. Comparing the
/// captured state can. Their fix moved 198 bytes of 335,692 — a
/// logits comparison would never have noticed.
///
/// So: snapshot, restore, snapshot again, and require the two to
/// agree field for field. Anything `restore_kv_cache` silently
/// fails to put back shows up here as a shape or value difference.
#[test]
fn a_snapshot_survives_its_own_round_trip() {
let (_dir, mut model) = build();
let prompt = Tensor::from_vec(vec![5u32, 9, 3, 12, 7], (1, 5), &Device::Cpu).unwrap();
model.forward(&prompt, 0).unwrap();
let first = model.snapshot_kv_cache().unwrap();
model.restore_kv_cache(&first).unwrap();
let second = model.snapshot_kv_cache().unwrap();
assert_eq!(first.layer_count(), second.layer_count());
assert_eq!(
first.size_bytes(),
second.size_bytes(),
"total captured bytes"
);
for (i, (a, b)) in first.layers.iter().zip(second.layers.iter()).enumerate() {
match (a, b) {
(
LayerKvSnapshot::FullSparse {
kv: ka,
indexer_keys: ia,
},
LayerKvSnapshot::FullSparse {
kv: kb,
indexer_keys: ib,
},
) => {
assert_eq!(ka.is_some(), kb.is_some(), "layer {i} kv presence");
if let (Some((k1, v1)), Some((k2, v2))) = (ka, kb) {
assert_eq!(k1.dims(), k2.dims(), "layer {i} k");
assert_eq!(v1.dims(), v2.dims(), "layer {i} v");
assert_eq!(tensor_vec(k1), tensor_vec(k2), "layer {i} k values");
}
// The field their fix was about: dropped silently,
// and invisible to anything that looks at outputs.
assert_eq!(
ia.is_some(),
ib.is_some(),
"layer {i} indexer cache presence"
);
if let (Some(x), Some(y)) = (ia, ib) {
assert_eq!(x.dims(), y.dims(), "layer {i} indexer keys");
assert_eq!(tensor_vec(x), tensor_vec(y), "layer {i} indexer values");
}
}
(
LayerKvSnapshot::Linear {
conv_state: ca,
recurrent_state: ra,
},
LayerKvSnapshot::Linear {
conv_state: cb,
recurrent_state: rb,
},
) => {
assert_eq!(ca.is_some(), cb.is_some(), "layer {i} conv presence");
assert_eq!(ra.is_some(), rb.is_some(), "layer {i} recurrent presence");
if let (Some(x), Some(y)) = (ca, cb) {
assert_eq!(tensor_vec(x), tensor_vec(y), "layer {i} conv values");
}
if let (Some(x), Some(y)) = (ra, rb) {
assert_eq!(tensor_vec(x), tensor_vec(y), "layer {i} recurrent values");
}
}
_ => panic!("layer {i}: snapshot variant changed across a round trip"),
}
}
let (pa, pb) = (first.ple.as_ref().unwrap(), second.ple.as_ref().unwrap());
assert_eq!(pa.carried_ids, pb.carried_ids, "PLE carried ids");
assert_eq!(pa.conv_state.is_some(), pb.conv_state.is_some());
if let (Some(x), Some(y)) = (&pa.conv_state, &pb.conv_state) {
assert_eq!(tensor_vec(x), tensor_vec(y), "PLE conv context");
}
}
/// A snapshot from a differently-shaped model must be refused
/// rather than restored onto whatever lines up.
#[test]
fn a_mismatched_snapshot_is_refused() {
let (_dir, mut model) = build();
let prompt = Tensor::from_vec(vec![5u32, 9], (1, 2), &Device::Cpu).unwrap();
model.forward(&prompt, 0).unwrap();
let mut snap = model.snapshot_kv_cache().unwrap();
snap.layers.pop();
let err = model.restore_kv_cache(&snap).unwrap_err().to_string();
assert!(err.contains("layers"), "got: {err}");
let mut snap = model.snapshot_kv_cache().unwrap();
snap.ple = None;
let err = model.restore_kv_cache(&snap).unwrap_err().to_string();
assert!(err.contains("PLE"), "got: {err}");
}
/// The draft head, end to end on the fixture: the target's
/// pre-mixer stream and the next token's embedding go in, and both
/// of the head's outputs come out — the collapse for `lm_head` and
/// the multi stream that a second draft step would consume.
#[test]
fn the_mtp_head_drafts_from_the_targets_pre_mixer_stream() {
use crate::harness::arch::qwen4_exp::mtp::MtpHead;
let (_dir, mut model) = build();
let cfg = Config::from_config_json(TINY).unwrap();
let dir2 = tempfile::tempdir().unwrap();
let path = dir2.path().join("model.safetensors");
candle_core::safetensors::save(&tensors(&cfg.text_config), &path).unwrap();
let vb =
unsafe { ShardedSafeTensors::var_builder(&[&path], DType::F32, &Device::Cpu).unwrap() };
let rotary = Arc::new(rotary_for(&cfg.text_config, DType::F32, &Device::Cpu).unwrap());
let mut head = MtpHead::load(&cfg.text_config, rotary.clone(), None, &vb).unwrap();
let prompt = Tensor::from_vec(vec![5u32, 9, 3, 12], (1, 4), &Device::Cpu).unwrap();
let (_logits, hidden) = model.forward_with_hidden(&prompt, 0).unwrap();
let wide = cfg.text_config.stream_width();
assert_eq!(hidden.dims(), &[1, 4, wide], "the pre-mixer stream");
// The head owns its own KV and indexer caches, so it has to walk
// the prompt alongside the target before it can draft from
// position 4 — a draft head that only ever sees the newest token
// is decoding against an empty past.
let ids: Vec<u32> = prompt.flatten_all().unwrap().to_vec1().unwrap();
for (i, id) in ids.iter().enumerate() {
let tok = Tensor::from_vec(vec![*id], (1, 1), &Device::Cpu).unwrap();
let e = model.embed(&tok).unwrap();
let h_i = hidden.narrow(1, i, 1).unwrap().contiguous().unwrap();
let (cos, sin) = rotary.plain_cos_sin(i, 1).unwrap();
head.forward(&e, &h_i, None, &cos, &sin, i).unwrap();
}
let next = Tensor::from_vec(vec![7u32], (1, 1), &Device::Cpu).unwrap();
let e = model.embed(&next).unwrap();
let last = hidden.narrow(1, 3, 1).unwrap().contiguous().unwrap();
let (cos, sin) = rotary.plain_cos_sin(4, 1).unwrap();
let step = head.forward(&e, &last, None, &cos, &sin, 4).unwrap();
assert_eq!(
step.sample_hidden.dims(),
&[1, 1, cfg.text_config.hidden_size]
);
assert_eq!(step.multi_stream.dims(), &[1, 1, wide]);
let v: Vec<f32> = step.sample_hidden.flatten_all().unwrap().to_vec1().unwrap();
assert!(v.iter().all(|x| x.is_finite()));
// Chained: the second step reads the head's own multi stream,
// not the target's and not the collapse.
let (cos, sin) = rotary.plain_cos_sin(5, 1).unwrap();
let step2 = head
.forward(&e, &step.multi_stream, None, &cos, &sin, 5)
.unwrap();
assert_eq!(step2.multi_stream.dims(), &[1, 1, wide]);
}
/// Handing the head the collapsed hidden state instead of the
/// pre-mixer stream is the mistake the shapes invite — both are
/// "the hidden state" and only one is right. It is refused by width
/// rather than silently broadcast.
#[test]
fn the_head_refuses_the_collapsed_hidden_state() {
use crate::harness::arch::qwen4_exp::mtp::MtpHead;
let cfg = Config::from_config_json(TINY).unwrap();
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("model.safetensors");
candle_core::safetensors::save(&tensors(&cfg.text_config), &path).unwrap();
let vb =
unsafe { ShardedSafeTensors::var_builder(&[&path], DType::F32, &Device::Cpu).unwrap() };
let rotary = Arc::new(rotary_for(&cfg.text_config, DType::F32, &Device::Cpu).unwrap());
let mut head = MtpHead::load(&cfg.text_config, rotary.clone(), None, &vb).unwrap();
let h = cfg.text_config.hidden_size;
let collapsed = Tensor::zeros((1, 1, h), DType::F32, &Device::Cpu).unwrap();
let e = Tensor::zeros((1, 1, h), DType::F32, &Device::Cpu).unwrap();
let (cos, sin) = rotary.plain_cos_sin(0, 1).unwrap();
let err = head
.forward(&e, &collapsed, None, &cos, &sin, 0)
.unwrap_err()
.to_string();
assert!(err.contains("pre-final-mixer"), "got: {err}");
}
}
+56 -73
View File
@@ -22,26 +22,45 @@
//! of error that produces a model that talks and reasons slightly
//! worse. It is asserted below rather than commented.
//!
//! The fused layout is the better one for a grouped GEMM, and keeping
//! it would be the faster path; that is an algorithm change, and the
//! spec is explicit that this stage is not it. Sparsity is extreme —
//! 10 of 512 with an intermediate of 640 — so the routing, not the
//! GEMM shape, is what dominates today.
//! ## Why the fused tensors stay fused
//!
//! Slicing them into 512 per-expert modules copies **241.6 GB** at
//! BF16, 5.03 GB per layer. That fits in neither beast's 64 GB of VRAM
//! nor its 123 GB of host RAM, and no quantisation rescues it: #309
//! measured 59.4 GB of 63.7 used at 4.25 bpw with only *four* layers'
//! experts offloaded. A model that cannot be held cannot be loaded.
//!
//! So the block takes `Experts::Banked` and views one expert per
//! routed token instead. A slice along dim 0 of a contiguous tensor is
//! contiguous, and `x.matmul(w.t())` is what `Linear` does anyway, so
//! the views feed the same GEMMs with no copy. It is also the layout a
//! host-resident bank with a device-side gather wants (#318).
//!
//! See `doc/qwen4_exp-port-spec.md` §6.
use anyhow::{Context, Result};
use candle_core::{IndexOp, Tensor};
use candle_core::Tensor;
use candle_core::quantized::GgmlDType;
use candle_nn::Linear;
use candle_nn::var_builder::ShardedVarBuilder;
use crate::harness::arch::qwen3_5::mlp::Qwen3_5MLP;
use crate::harness::arch::qwen3_5::moe::Qwen3_5MoeBlock;
use crate::harness::arch::qwen3_5::moe::{Experts, Qwen3_5MoeBlock};
use super::config::TextConfig;
/// `vb` should be `.pp(...)`-ed to the layer's `mlp` prefix.
pub fn load(cfg: &TextConfig, vb: &ShardedVarBuilder) -> Result<Qwen3_5MoeBlock> {
///
/// `quant` quantises the routed experts in situ (#315). Without it the
/// experts stay banked at the checkpoint's precision, which is only
/// useful for a model small enough to hold — 120.8 B routed parameters
/// are 241.6 GB at BF16 and fit nowhere on this hardware, so a real
/// load of `Qwen3.8-Flash-Next` always passes one.
pub fn load(
cfg: &TextConfig,
quant: Option<GgmlDType>,
vb: &ShardedVarBuilder,
) -> Result<Qwen3_5MoeBlock> {
let (h, inter) = (cfg.hidden_size, cfg.moe_intermediate_size);
anyhow::ensure!(
cfg.num_experts > 0 && cfg.num_experts_per_tok > 0 && inter > 0,
@@ -65,7 +84,19 @@ pub fn load(cfg: &TextConfig, vb: &ShardedVarBuilder) -> Result<Qwen3_5MoeBlock>
let down = experts_vb
.get((cfg.num_experts, h, inter), "down_proj")
.with_context(|| format!("load '{}/down_proj'", experts_vb.prefix()))?;
let experts = split_fused_experts(&gate_up, &down, inter)?;
check_fused_experts(&gate_up, &down, inter)?;
// The fused pair is the peak: 5.03 GB per layer at BF16. Quantising
// here and letting it drop at the end of this function is what keeps
// the whole 241.6 GB from ever existing.
let experts = match quant {
Some(dtype) => Experts::quantize_banked(&gate_up, &down, inter, dtype)
.with_context(|| format!("quantize experts to {dtype:?}"))?,
None => Experts::Banked {
gate_up,
down,
intermediate: inter,
},
};
let (shared_expert, shared_expert_gate) = if cfg.shared_expert_intermediate_size > 0 {
let shared = Qwen3_5MLP::load_with_dims(
@@ -93,14 +124,17 @@ pub fn load(cfg: &TextConfig, vb: &ShardedVarBuilder) -> Result<Qwen3_5MoeBlock>
))
}
/// Slice `(E, 2I, H)` and `(E, H, I)` into `E` SwiGLU experts.
/// Validate the fused pair before it is banked.
///
/// Gate first, up second — see the module note.
pub(crate) fn split_fused_experts(
/// The block will take views of these per routed expert, so a
/// disagreement here surfaces as a shape error mid-forward on whichever
/// token first routes to the offending expert — which is a worse place
/// to find it than at load.
pub(crate) fn check_fused_experts(
gate_up: &Tensor,
down: &Tensor,
intermediate: usize,
) -> Result<Vec<Qwen3_5MLP>> {
) -> Result<()> {
let (experts, fused, _) = gate_up.dims3()?;
anyhow::ensure!(
fused == intermediate * 2,
@@ -111,78 +145,27 @@ pub(crate) fn split_fused_experts(
"down_proj holds {} experts, gate_up_proj holds {experts}",
down.dims3()?.0
);
let mut out = Vec::with_capacity(experts);
for e in 0..experts {
let fused_e = gate_up.i(e)?;
out.push(Qwen3_5MLP::from_weights(
Linear::new(fused_e.narrow(0, 0, intermediate)?.contiguous()?, None),
Linear::new(
fused_e
.narrow(0, intermediate, intermediate)?
.contiguous()?,
None,
),
Linear::new(down.i(e)?.contiguous()?, None),
));
}
Ok(out)
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use candle_core::{DType, Device, Module};
use candle_core::{DType, Device};
/// One expert, two hidden dims, one intermediate. The gate row picks
/// the first input channel and the up row picks the second, so
/// swapping them swaps which channel is squashed by the SiLU — and
/// the two answers differ by more than any tolerance.
/// The loader validates the fused pair up front. Without this the
/// first token to route to a bad expert fails mid-forward, which is
/// a worse place to learn the checkpoint disagrees with the config.
#[test]
fn the_gate_is_the_first_half_of_the_fused_tensor() {
let dev = Device::Cpu;
// gate_up: [[gate row], [up row]] = [[1, 0], [0, 1]]
let gate_up = Tensor::from_vec(vec![1.0f32, 0.0, 0.0, 1.0], (1, 2, 2), &dev).unwrap();
// down: (E=1, H=2, I=1)
let down = Tensor::from_vec(vec![2.0f32, 3.0], (1, 2, 1), &dev).unwrap();
let experts = split_fused_experts(&gate_up, &down, 1).unwrap();
assert_eq!(experts.len(), 1);
let x = Tensor::from_vec(vec![1.0f32, 2.0], (1, 2), &dev).unwrap();
let got: Vec<f32> = experts[0]
.forward(&x)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
// gate(x) = 1, up(x) = 2 -> silu(1) * 2 -> down
let silu1 = 1.0f32 / (1.0 + (-1.0f32).exp());
let want = [2.0 * silu1 * 2.0, 3.0 * silu1 * 2.0];
for (g, w) in got.iter().zip(want.iter()) {
assert!((g - w).abs() < 1e-5, "got {got:?} want {want:?}");
}
// The swapped reading: silu(2) * 1, which is a different model.
let silu2 = 2.0f32 / (1.0 + (-2.0f32).exp());
assert!(
(got[0] - 2.0 * silu2).abs() > 1e-3,
"this is silu(up) * gate, not silu(gate) * up: {got:?}"
);
}
#[test]
fn a_fused_tensor_of_the_wrong_width_is_rejected() {
fn a_fused_pair_that_disagrees_with_the_config_is_rejected() {
let dev = Device::Cpu;
let gate_up = Tensor::zeros((2, 6, 4), DType::F32, &dev).unwrap();
let down = Tensor::zeros((2, 4, 3), DType::F32, &dev).unwrap();
assert!(split_fused_experts(&gate_up, &down, 3).is_ok());
assert!(check_fused_experts(&gate_up, &down, 3).is_ok());
// 2 x 4 != 6
assert!(split_fused_experts(&gate_up, &down, 4).is_err());
assert!(check_fused_experts(&gate_up, &down, 4).is_err());
// expert counts must agree
let down_short = Tensor::zeros((1, 4, 3), DType::F32, &dev).unwrap();
assert!(split_fused_experts(&gate_up, &down_short, 3).is_err());
assert!(check_fused_experts(&gate_up, &down_short, 3).is_err());
}
}
@@ -0,0 +1,263 @@
//! The multi-token-prediction head — `qwen4_exp`'s shipped draft model.
//!
//! The checkpoint carries a complete second decoder layer under `mtp.*`:
//! full attention with its own QSA indexer, its own 512-expert MoE, both
//! hyper-connections, and a final mixer. It reuses `embed_tokens` and
//! `lm_head` (`mtp_use_dedicated_embeddings: false`), so what is new
//! here is only how the target model's state and the next token's
//! embedding are fused into an input for it.
//!
//! ```text
//! e = fc_embedding(pre_fc_norm_embedding(embed(t))) [T, 2560]
//! h = pre_fc_norm_hidden(h.flatten(-2)) ONE norm over 10240
//! h = fc_hidden(h.view(T, hc, 2560)) per stream, shared matrix
//! h = e.unsqueeze(-2) + h embedding into every stream
//! h = h.flatten(-2) [T, 10240] -> the layer
//! ```
//!
//! vLLM calls this `residual_linear_shared`, as against the
//! `Linear(2H, H)` + repeat other MTP variants use. Three parts of it
//! cannot be recovered from the tensor shapes, and all three were read
//! from `vllm/models/qwen4_exp/nvidia/mtp.py` rather than inferred:
//!
//! 1. **`pre_fc_norm_hidden` is ungrouped** — one normalisation over the
//! whole 10240, unlike every other 10240-wide norm in this
//! architecture. vLLM has a `GroupedGemmaRMSNorm` and deliberately
//! does not use it here. Grouping it rescales the draft input in four
//! independent pieces, which costs acceptance rate and reports
//! nothing.
//! 2. **The hidden state is the *pre-final-mixer* multi-stream** — `h`
//! after the target's last decoder layer, before its mixer collapses
//! it. Not the 2560 the LM head sees.
//! 3. **The head emits both** — a collapsed stream for `lm_head` and the
//! pre-final-mixer stream for its own next step. A head that returned
//! only the first cannot be chained past one token.
//!
//! PLE is off in this layer while the stream count is kept, which the
//! absence of `mtp.layers.0.ple.*` in the checkpoint agrees with.
//!
//! **The head keeps its own KV and indexer caches, and they are not the
//! target's.** It therefore has to walk the prompt alongside the target
//! before it can draft: a head fed only the newest token is attending
//! over an empty past, and this architecture says so rather than
//! guessing, because the indexer's divergence check fires. That makes
//! prefill part of the speculative loop's cost, not just decode.
//!
//! See `doc/qwen4_exp-port-spec.md` §9 and #313.
use anyhow::{Context, Result};
use candle_core::quantized::GgmlDType;
use candle_core::{D, Module, Tensor};
use candle_nn::Linear;
use candle_nn::var_builder::ShardedVarBuilder;
use std::sync::Arc;
use crate::harness::arch::qwen3_5::rmsnorm::Qwen3_5RmsNorm;
use crate::harness::arch::qwen3_5::rope::RotaryEmbedding;
use super::config::TextConfig;
use super::decoder::DecoderLayer;
use super::hyper::HyperConnection;
/// One draft step's two outputs.
#[derive(Debug)]
pub struct MtpStep {
/// `[B, L, hidden]` — the mixer's collapse, for `lm_head`.
pub sample_hidden: Tensor,
/// `[B, L, hidden * hc_count]` — pre-final-mixer, and the input to
/// the *next* draft step. Chaining reads this, not the collapse.
pub multi_stream: Tensor,
}
pub struct MtpHead {
pre_fc_norm_embedding: Qwen3_5RmsNorm,
/// Ungrouped over `hidden * hc_count` — see the module note.
pre_fc_norm_hidden: Qwen3_5RmsNorm,
fc_embedding: Linear,
/// Shared across the streams: one `hidden -> hidden` matrix applied
/// to each branch, not a `wide -> wide` one.
fc_hidden: Linear,
layer: DecoderLayer,
mixer: HyperConnection,
hidden_size: usize,
hc_count: usize,
}
impl MtpHead {
/// `vb` is the checkpoint root; the head hangs off `mtp.`.
pub fn load(
cfg: &TextConfig,
rotary: Arc<RotaryEmbedding>,
quant: Option<GgmlDType>,
vb: &ShardedVarBuilder,
) -> Result<Self> {
let mtp = vb.pp("mtp");
let (h, hc) = (cfg.hidden_size, cfg.hc_count);
Ok(Self {
pre_fc_norm_embedding: Qwen3_5RmsNorm::load(
&mtp.pp("pre_fc_norm_embedding"),
h,
cfg.rms_norm_eps,
)?,
// `load`, not `load_grouped`: the one 10240-wide norm in
// this architecture that is not per-stream.
pre_fc_norm_hidden: Qwen3_5RmsNorm::load(
&mtp.pp("pre_fc_norm_hidden"),
h * hc,
cfg.rms_norm_eps,
)?,
fc_embedding: linear(&mtp, "fc_embedding", h, h)?,
fc_hidden: linear(&mtp, "fc_hidden", h, h)?,
layer: DecoderLayer::load_typed(
cfg,
rotary,
"full_attention",
false,
quant,
&mtp.pp("layers").pp(0),
)
.context("load mtp.layers.0")?,
mixer: HyperConnection::load(
&mtp.pp("hyper_connection_mixer"),
h,
hc,
cfg.hc_lowrank,
cfg.rms_norm_eps,
false,
)
.context("load mtp.hyper_connection_mixer")?,
hidden_size: h,
hc_count: hc,
})
}
/// One draft step.
///
/// `token_embed` is `embed_tokens(t)` for the token being predicted
/// *from*, `[B, L, hidden]`. `hidden` is the pre-final-mixer stream,
/// `[B, L, hidden * hc_count]` — from the target model on the first
/// step, from this head's own previous [`MtpStep::multi_stream`]
/// afterwards.
pub fn forward(
&mut self,
token_embed: &Tensor,
hidden: &Tensor,
causal_mask: Option<&Tensor>,
cos: &Tensor,
sin: &Tensor,
past_len: usize,
) -> Result<MtpStep> {
let (b, l, wide) = hidden.dims3()?;
anyhow::ensure!(
wide == self.hidden_size * self.hc_count,
"mtp: expected the pre-final-mixer stream ({} wide), got {wide} — \
passing the collapsed hidden state here is the likely mistake",
self.hidden_size * self.hc_count
);
let e = self
.fc_embedding
.forward(&self.pre_fc_norm_embedding.forward(token_embed)?)?;
// Normalise the flattened stream, *then* split it: one
// normalisation over the whole width.
let h = self.pre_fc_norm_hidden.forward(hidden)?.reshape((
b,
l,
self.hc_count,
self.hidden_size,
))?;
let h = self.fc_hidden.forward(&h)?;
// The embedding enters every stream.
let h = h
.broadcast_add(&e.unsqueeze(D::Minus2)?)?
.flatten_from(D::Minus2)?;
let h = self
.layer
.forward(&h, None, causal_mask, cos, sin, past_len)
.context("mtp draft layer")?;
Ok(MtpStep {
sample_hidden: self.mixer.collapse(&h)?,
multi_stream: h,
})
}
/// Drop the draft layer's caches. The head has its own KV and its
/// own indexer cache, independent of the target model's.
pub fn clear_kv_cache(&mut self) -> Result<()> {
self.layer.clear_kv_cache()
}
}
fn linear(vb: &ShardedVarBuilder, name: &str, in_dim: usize, out_dim: usize) -> Result<Linear> {
let weight = vb
.pp(name)
.get((out_dim, in_dim), "weight")
.with_context(|| format!("load '{}/{name}/weight'", vb.prefix()))?;
Ok(Linear::new(weight, None))
}
#[cfg(test)]
mod tests {
use super::*;
use candle_core::{DType, Device};
/// `pre_fc_norm_hidden` normalises the whole 10240 at once, where
/// every other norm this wide in the architecture normalises four
/// streams of 2560 independently. The port spec asserted the
/// opposite until vLLM settled it, and the difference is invisible
/// downstream — it rescales the draft input and costs acceptance
/// rate.
///
/// So it is pinned by value, on an input the two readings cannot
/// agree on: two streams whose magnitudes differ threefold. Grouped,
/// each is normalised to its own scale and both come out all-ones.
/// Ungrouped, the larger stream stays larger.
#[test]
fn the_hidden_pre_norm_is_ungrouped() {
let dev = Device::Cpu;
let (hidden, hc) = (4usize, 2usize);
let wide = hidden * hc;
// Zero weight: (1 + w) = 1, so this is a pure RMS normalisation.
let norm =
Qwen3_5RmsNorm::from_weight(Tensor::zeros(wide, DType::F32, &dev).unwrap(), 1e-6, None);
let h = Tensor::from_vec(
vec![1.0f32, 1.0, 1.0, 1.0, 3.0, 3.0, 3.0, 3.0],
(1, 1, wide),
&dev,
)
.unwrap();
let got: Vec<f32> = norm
.forward(&h)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
// rms over all eight = sqrt((4*1 + 4*9)/8) = sqrt(5)
let rms = 5.0f32.sqrt();
let want = [
1.0 / rms,
1.0 / rms,
1.0 / rms,
1.0 / rms,
3.0 / rms,
3.0 / rms,
3.0 / rms,
3.0 / rms,
];
for (g, w) in got.iter().zip(want.iter()) {
assert!((g - w).abs() < 1e-5, "got {got:?} want {want:?}");
}
// The grouped reading would flatten both streams to all-ones.
assert!(
(got[4] - 1.0).abs() > 1e-3,
"this is the grouped reading, which the spec had and vLLM does not: {got:?}"
);
}
}
@@ -491,6 +491,26 @@ impl PleBlock {
(self.kernel_size - 1) * self.dilation
}
/// Capture the rolling conv context for a prefix snapshot.
///
/// Deep-copied rather than shared. The buffer is replaced by
/// assignment today, so a shallow clone would be sound — but
/// `narrow(..).contiguous()` on an already-contiguous slice is a
/// view, so whether the snapshot shares storage depends on a
/// property of the *data*, not of the code. That is exactly the
/// kind of conditional aliasing the GatedDeltaNet states are
/// deep-copied to avoid, and nine positions of a 10240-wide signal
/// is not worth being clever about.
pub fn snapshot_state(&self) -> candle_core::Result<Option<Tensor>> {
self.conv_state.as_ref().map(Tensor::copy).transpose()
}
/// Replace the rolling conv context from a snapshot.
pub fn restore_state(&mut self, conv_state: Option<&Tensor>) -> candle_core::Result<()> {
self.conv_state = conv_state.map(Tensor::copy).transpose()?;
Ok(())
}
/// Drop the rolling conv context. Call alongside a KV-cache clear;
/// the carried n-gram ids ([`NGramHasher::context_len`]) reset with
/// it.
@@ -293,6 +293,21 @@ impl Indexer {
&self.selector
}
/// Capture the raw key cache for a prefix snapshot.
///
/// A shallow clone, like the attention K/V it travels with: the
/// cache is only ever *replaced* by a `cat` into a fresh
/// allocation, never written in place, so the snapshot stays valid
/// after the live cache moves on.
pub fn snapshot_keys(&self) -> Option<Tensor> {
self.key_cache.clone()
}
/// Replace the key cache from a snapshot.
pub fn restore_keys(&mut self, keys: Option<&Tensor>) {
self.key_cache = keys.cloned();
}
/// Drop the indexer's cache. Must happen with the main KV clear —
/// a stale indexer cache selects blocks of another request's text.
pub fn clear_cache(&mut self) {
@@ -1,5 +1,12 @@
//! Cache-state snapshots for prefix KV caching (#11).
//!
//! Shared across architectures rather than owned by one: `qwen3_5`,
//! its TP mirror `tp_qwen3_5`, and `qwen4_exp` all capture state
//! through these types, and `ModelArch::snapshot_kv_cache` hands one
//! back whatever the architecture. It lived under `arch/qwen3_5/`
//! while that was the only arch with a snapshot, which stopped being
//! true.
//!
//! A snapshot captures everything `clear_kv_cache` would destroy, at
//! one consistent token boundary:
//!
@@ -40,6 +47,17 @@ pub enum LayerKvSnapshot {
conv_state: Option<Tensor>,
recurrent_state: Option<Tensor>,
},
/// A full-attention layer that also keeps a QSA indexer cache
/// (`qwen4_exp`). The indexer's raw keys are a second, separately
/// shaped cache — one head of 128 dims per position — and a
/// snapshot that captured only `kv` would restore a layer whose
/// indexer still holds the *previous* request's keys, selecting
/// blocks of somebody else's text. It fails fluently, so the two
/// travel together or not at all.
FullSparse {
kv: Option<(Tensor, Tensor)>,
indexer_keys: Option<Tensor>,
},
}
/// One consistent cache snapshot of a `Qwen3_5Model` (or its TP
@@ -50,6 +68,9 @@ pub enum LayerKvSnapshot {
pub struct KvCacheSnapshot {
pub(crate) layers: Vec<LayerKvSnapshot>,
pub(crate) rope_delta: i64,
/// `qwen4_exp`'s PLE state. `None` for architectures without a
/// hashed n-gram layer, which is every other one today.
pub(crate) ple: Option<PleSnapshot>,
}
impl KvCacheSnapshot {
@@ -79,8 +100,44 @@ impl KvCacheSnapshot {
conv_state.as_ref().map(t_bytes).unwrap_or(0)
+ recurrent_state.as_ref().map(t_bytes).unwrap_or(0)
}
LayerKvSnapshot::FullSparse { kv, indexer_keys } => {
kv.as_ref()
.map(|(k, v)| t_bytes(k) + t_bytes(v))
.unwrap_or(0)
+ indexer_keys.as_ref().map(t_bytes).unwrap_or(0)
}
})
.sum()
.sum::<u64>()
+ self.ple.as_ref().map(PleSnapshot::size_bytes).unwrap_or(0)
}
}
/// PLE's per-request state (`qwen4_exp`), captured at the same token
/// boundary as the layers.
///
/// Two pieces with different natures. `conv_state` is the dilated
/// short conv's left context — nine positions of a 10240-wide signal,
/// deep-copied like the GatedDeltaNet states because the conv writes
/// its rolling buffer in place. `carried_ids` is the previous
/// `ngram_size - 1` **token ids** per batch row, which upstream keeps
/// as a third "conv state" on the same layer; they are ids, not
/// activations, and cost two integers per sequence.
pub struct PleSnapshot {
pub(crate) conv_state: Option<Tensor>,
pub(crate) carried_ids: Vec<Vec<i64>>,
}
impl PleSnapshot {
fn size_bytes(&self) -> u64 {
self.conv_state
.as_ref()
.map(|t| (t.elem_count() * t.dtype().size_in_bytes()) as u64)
.unwrap_or(0)
+ self
.carried_ids
.iter()
.map(|r| (r.len() * std::mem::size_of::<i64>()) as u64)
.sum::<u64>()
}
}
@@ -145,6 +202,7 @@ pub fn assemble_batch(seqs: &[(&KvCacheSnapshot, usize)]) -> candle_core::Result
}
Ok(BatchedKvState {
snapshot: KvCacheSnapshot {
ple: None,
layers,
rope_delta: 0,
},
@@ -159,6 +217,14 @@ fn assemble_layer(
padded_len: usize,
) -> candle_core::Result<LayerKvSnapshot> {
match &seqs[0].0.layers[li] {
// Batched decode for the sparse-attention layers is not built
// yet. Assembling them as if they were plain full-attention
// would drop the indexer cache and hand every row a selection
// computed from another request's keys, which produces text.
LayerKvSnapshot::FullSparse { .. } => candle_core::bail!(
"assemble_batch: layer {li} carries a QSA indexer cache; batched decode \
for this architecture is not implemented"
),
LayerKvSnapshot::Full(_) => {
let mut ks = Vec::with_capacity(seqs.len());
let mut vs = Vec::with_capacity(seqs.len());
@@ -242,6 +308,10 @@ pub fn extract_row(
let mut layers = Vec::with_capacity(snap.layers.len());
for (li, layer) in snap.layers.iter().enumerate() {
layers.push(match layer {
LayerKvSnapshot::FullSparse { .. } => candle_core::bail!(
"extract_row: layer {li} carries a QSA indexer cache; batched decode \
for this architecture is not implemented"
),
LayerKvSnapshot::Full(Some((k, v))) => {
let (b, _h, s, _d) = k.dims4()?;
if row >= b {
@@ -274,6 +344,7 @@ pub fn extract_row(
});
}
Ok(KvCacheSnapshot {
ple: None,
layers,
rope_delta: 0,
})
@@ -312,7 +383,7 @@ fn pad_seq(t: &Tensor, padded_len: usize) -> candle_core::Result<Tensor> {
#[cfg(test)]
mod tests {
use super::super::{Qwen3_5Model, RopeParameters, TextConfig};
use crate::harness::arch::qwen3_5::{Qwen3_5Model, RopeParameters, TextConfig};
use candle_core::{DType, Device, Tensor};
use std::collections::HashMap;
+108 -26
View File
@@ -555,7 +555,7 @@ impl LoadedHandle {
#[derive(Clone)]
pub enum KvSnapshotRef {
Worker(super::device_worker::KvSnapshotId),
Local(Arc<super::arch::qwen3_5::snapshot::KvCacheSnapshot>),
Local(Arc<super::arch::snapshot::KvCacheSnapshot>),
#[cfg(feature = "cuda")]
Tp(u64),
}
@@ -1116,15 +1116,16 @@ impl ModelArch {
/// candle-transformers archs keep theirs private, so they stay on
/// the clear-every-request path.
pub fn supports_kv_snapshot(&self) -> bool {
matches!(self, ModelArch::Qwen3_5Dense(_))
matches!(self, ModelArch::Qwen3_5Dense(_) | ModelArch::Qwen4Exp(_))
}
/// Capture the live cache state as a prefix snapshot. See
/// `arch/qwen3_5/snapshot.rs` for what a snapshot contains and the
/// copy-semantics constraints.
pub fn snapshot_kv_cache(&self) -> Result<super::arch::qwen3_5::snapshot::KvCacheSnapshot> {
pub fn snapshot_kv_cache(&self) -> Result<super::arch::snapshot::KvCacheSnapshot> {
match self {
ModelArch::Qwen3_5Dense(m) => Ok(m.snapshot_kv_cache()?),
ModelArch::Qwen4Exp(m) => Ok(m.snapshot_kv_cache()?),
_ => anyhow::bail!("snapshot_kv_cache: architecture has no snapshot support"),
}
}
@@ -1133,10 +1134,11 @@ impl ModelArch {
/// restore-instead-of-clear half of prefix caching.
pub fn restore_kv_cache(
&mut self,
snap: &super::arch::qwen3_5::snapshot::KvCacheSnapshot,
snap: &super::arch::snapshot::KvCacheSnapshot,
) -> Result<()> {
match self {
ModelArch::Qwen3_5Dense(m) => Ok(m.restore_kv_cache(snap)?),
ModelArch::Qwen4Exp(m) => Ok(m.restore_kv_cache(snap)?),
_ => anyhow::bail!("restore_kv_cache: architecture has no snapshot support"),
}
}
@@ -1347,7 +1349,7 @@ impl LlamaDense {
/// value. New entries land alongside a new `ModelArch` variant + a
/// dispatch branch in `load_arch_dense` (plus, for TP, a parallel
/// pattern in `tp_qwen3.rs`).
const DENSE_SUPPORTED_MODEL_TYPES: &[&str] = &[
pub(crate) const DENSE_SUPPORTED_MODEL_TYPES: &[&str] = &[
"llama",
"qwen3",
"qwen3_5",
@@ -2733,6 +2735,9 @@ impl CandleHarness {
self.resolve_dense_files(spec, source_id).await?;
let device_for_load = device.clone();
let model_id_for_log = spec.model_id.clone();
// Resolved out here: `spec` is borrowed and does not outlive the
// blocking task.
let isq = super::quant::parse_quant_string(spec.quant.as_deref())?;
let arch = tokio::task::spawn_blocking(move || -> Result<ModelArch> {
let cfg_text = std::fs::read_to_string(&config_path).context("read config.json")?;
@@ -2843,9 +2848,19 @@ impl CandleHarness {
)
.context("build ShardedVarBuilder for qwen4_exp")?
};
// ISQ is not optional for this architecture in
// practice: without it the routed experts are
// 241.6 GB. `quant` comes from the catalogue, and a
// load without one will fail on memory rather than
// on anything this code can say (#315).
// ISQ is not really optional here: without it the
// routed experts are 241.6 GB and the load fails on
// memory rather than on anything this code can say.
// The catalogue's `quant` is the knob (#315, #252).
let model = super::arch::qwen4_exp::model::Qwen4ExpForCausalLM::load(
&cfg,
dtype,
isq,
&device_for_load,
&sharded_vb,
)
@@ -2870,6 +2885,59 @@ impl CandleHarness {
Ok((tokenizer_path, arch))
}
/// Whether this spec's `quant` names a GGUF the repo actually has.
///
/// `quant` is overloaded (#320): the TP path reads it as in-situ
/// quantisation of safetensors, while the single-GPU paths read it
/// as "select the GGUF whose filename contains this string". A repo
/// with no GGUF at all cannot mean the second, so rather than fail
/// with an error about a file that was never going to exist, the
/// same string is read the way the TP path reads it.
///
/// This is deliberately narrow. A repo that *does* carry GGUFs keeps
/// the old behaviour exactly, including the old error when none of
/// them match the requested quant — that case is a typo, not an ISQ
/// request, and turning it into a silent 360 GB dense load would be
/// a worse answer than the error.
async fn quant_selects_a_gguf(
&self,
spec: &ModelSpec,
source_id: &cortex_core::source::ModelSourceId,
) -> Result<bool> {
let Some(quant) = spec.quant.as_deref().filter(|q| !q.is_empty()) else {
return Ok(false);
};
let api = self.hf_api_for(&source_id.scheme)?;
let info = api
.model(source_id.repo_path())
.info()
.await
.with_context(|| format!("fetch HF repo info for {source_id}"))?;
let names: Vec<String> = info
.siblings
.iter()
.map(|s| s.rfilename.to_lowercase())
.collect();
let any_gguf = names.iter().any(|n| n.ends_with(".gguf"));
if any_gguf {
return Ok(true);
}
let any_safetensors = names.iter().any(|n| n.ends_with(".safetensors"));
if any_safetensors {
tracing::info!(
model = %source_id,
quant = %quant,
"no GGUF in this repo; reading `quant` as in-situ quantization of \
the safetensors (#320)"
);
return Ok(false);
}
anyhow::bail!(
"repo {source_id} has neither a GGUF nor safetensors; `quant = {quant:?}` \
cannot be satisfied"
)
}
/// Resolve a model spec to local GGUF and tokenizer file paths via
/// hf-hub. Downloads on first use; subsequent calls are cached.
async fn resolve_files(
@@ -3854,7 +3922,7 @@ impl Harness for CandleHarness {
context_profile,
) = if let Some(w) = &worker {
// CUDA path: resolve, then load in the worker.
if spec.quant.is_some() {
if self.quant_selects_a_gguf(spec, &source_id).await? {
let (gguf_path, tokenizer_path) = self.resolve_files(spec, &source_id).await?;
let handle = w
.load_gguf(gguf_path, spec.model_id.clone())
@@ -3878,36 +3946,38 @@ impl Harness for CandleHarness {
let (config_path, tokenizer_path, safetensors_paths) =
self.resolve_dense_files(spec, &source_id).await?;
let meta = VisionMeta::from_config_path(&config_path);
// Prefix snapshots (#11) exist only for the in-tree
// qwen3_5 arch — which serves BOTH its model_types
// (qwen3_5 and the flat-config qwen3_next MoE family);
// the worker holds the ModelArch so the async side
// decides from config.json instead.
let snapshot_capable = matches!(
config_model_type(&config_path).as_deref(),
Some(super::arch::qwen3_5::MODEL_TYPE)
| Some(super::arch::qwen3_5::MODEL_TYPE_NEXT)
);
// Context-limit physics (#67): single-GPU → world_size 1.
// `None` for non-qwen3_5 dense archs.
// `None` for dense archs with no profile builder — they
// fall back to the static prompt cap.
let context_profile =
super::context_limit::profile_from_qwen3_5_config(&config_path, 1);
let handle = w
.load_dense(config_path, safetensors_paths, spec.model_id.clone())
super::context_limit::profile_from_qwen3_5_config(&config_path, 1).or_else(
|| super::context_limit::profile_from_qwen4_exp_config(&config_path, 1),
);
// In-situ quantisation (#320) reaches the worker load
// too, not just the CPU one — parsed here because
// `spec` does not cross the channel.
let isq = super::quant::parse_quant_string(spec.quant.as_deref())?;
let load = w
.load_dense(config_path, safetensors_paths, spec.model_id.clone(), isq)
.await
.map_err(|e| anyhow::anyhow!("worker load_dense: {e:#}"))?;
// Prefix snapshots (#11): the built model answers for
// itself on the way back, rather than the async side
// re-deriving the answer from `config.json`. See
// `DenseLoad` — the re-derivation is what silently
// dropped batched decode for qwen4_exp.
(
tokenizer_path,
None,
Some(handle),
Some(load.handle),
meta,
snapshot_capable,
load.supports_kv_snapshot,
context_profile,
)
}
} else {
// CPU path: legacy spawn_blocking + Arc<Mutex<ModelArch>>.
let (tokenizer_path, arch) = if spec.quant.is_some() {
let (tokenizer_path, arch) = if self.quant_selects_a_gguf(spec, &source_id).await? {
self.load_arch_gguf(spec, &source_id, &device).await?
} else {
self.load_arch_dense(spec, &source_id, &device).await?
@@ -4725,6 +4795,13 @@ impl CandleHarness {
image_token_id: vision_meta.image_token_id,
image_grid_factor: vision_meta.image_grid_factor,
spec: spec.clone(),
// Snapshot capability re-derived from `config.json`
// rather than asked of the loaded shard. Correct today —
// `TP_SUPPORTED_MODEL_TYPES` is exactly these two — but it
// is the same shape as the bug `DenseLoad` fixed on the
// single-GPU path, so whoever adds an arch here (#319)
// must add it in both places or lose batched decode
// silently.
prefix_cache: self.new_prefix_cache(matches!(
config_model_type(&config_path).as_deref(),
Some(super::arch::qwen3_5::MODEL_TYPE)
@@ -6276,9 +6353,14 @@ struct VisionMeta {
}
/// Peek at `config.json` for its `model_type`. Best-effort — `None`
/// on any read/parse error. The load path uses this to decide prefix-
/// snapshot capability for worker-held models the async side can't
/// inspect directly.
/// on any read/parse error.
///
/// Only the TP load path still asks: the single-GPU path takes the
/// arch's own answer off the load reply (`DenseLoad`) rather than
/// re-deriving it here, which is what stopped the two from drifting.
/// The TP path has the same hazard and not yet the same cure — see
/// the note at its call site.
#[cfg(feature = "cuda")]
fn config_model_type(config_path: &std::path::Path) -> Option<String> {
let text = std::fs::read_to_string(config_path).ok()?;
let v: serde_json::Value = serde_json::from_str(&text).ok()?;
+139 -3
View File
@@ -252,6 +252,85 @@ pub fn profile_from_qwen3_5_config(config_path: &Path, world_size: u32) -> Optio
})
}
/// Bytes of **QSA indexer** cache one token adds, per card (#307).
///
/// `qwen4_exp` keeps a second cache beside the main one: on every
/// full-attention layer the indexer stores the raw index key,
/// `indexer_kv_heads * indexer_head_dim` wide, in the compute dtype
/// (`arch::qwen4_exp::qsa::Indexer::key_cache`). Keys only — the
/// indexer scores blocks, it does not attend them — so there is no
/// `2 *` here.
///
/// It does not shard: `indexer_kv_heads` is 1, so a TP rank holds the
/// whole thing however wide the world is. At the checkpoint's 12
/// full-attention layers x 1 head x 128 dims that is 3 KiB/token on
/// top of the main cache's 24 — the surcharge the epic's original
/// arithmetic omitted, and it grows with context exactly as the main
/// cache does.
pub fn indexer_bytes_per_token(
n_full_attn_layers: usize,
indexer_kv_heads: usize,
indexer_head_dim: usize,
dtype_bytes: usize,
) -> u64 {
(n_full_attn_layers * indexer_kv_heads * indexer_head_dim * dtype_bytes) as u64
}
/// Build a [`ContextProfile`] from a `qwen4_exp` `config.json` — the
/// sibling of [`profile_from_qwen3_5_config`] for the other hybrid
/// arch. Returns `None` for any other `model_type` or an unparseable
/// config.
///
/// Same shape of physics, one term more: the per-token cost is the
/// main KV cache **plus** the QSA indexer's, because both grow per
/// token and both come out of the same card. Advertising against the
/// main cache alone would overstate what is servable by about an
/// eighth, which is the class of error #281 is about.
///
/// `layer_types` is always populated by the time `from_config_json`
/// returns (it fills the array from `full_attention_interval` when the
/// checkpoint omits it), so counting it is authoritative here.
pub fn profile_from_qwen4_exp_config(
config_path: &Path,
world_size: u32,
) -> Option<ContextProfile> {
let text = std::fs::read_to_string(config_path).ok()?;
if !super::arch::qwen4_exp::config::is_qwen4_exp(&text) {
return None;
}
let cfg = super::arch::qwen4_exp::config::Config::from_config_json(&text).ok()?;
let tc = &cfg.text_config;
let n_full_attn_layers = tc
.layer_types
.iter()
.filter(|t| t.as_str() == "full_attention")
.count();
// A model with no full-attention layer has no growing cache, and
// the derivation divides by this. Nothing in the family looks like
// that, so treat it as a config we do not understand rather than
// inventing an unbounded context.
if n_full_attn_layers == 0 {
return None;
}
let kv_bytes_per_token_per_card = kv_bytes_per_token(
n_full_attn_layers,
tc.num_key_value_heads,
tc.head_dim,
KV_CACHE_DTYPE_BYTES,
world_size,
) + indexer_bytes_per_token(
n_full_attn_layers,
tc.indexer_kv_heads,
tc.indexer_head_dim,
KV_CACHE_DTYPE_BYTES,
);
Some(ContextProfile {
max_position_embeddings: tc.max_position_embeddings,
kv_bytes_per_token_per_card,
world_size,
})
}
/// Round a token count down to a clean boundary so the advertised limit
/// doesn't jitter by a handful of tokens as live VRAM / the throughput
/// EMA wobble between polls.
@@ -375,9 +454,15 @@ pub fn derive_limit(
// Observability (#126): every input and intermediate term, so a
// surprising advertised limit is diagnosable from the journal
// instead of re-deriving by hand. DEBUG — this runs on every
// `GET /models` poll (a few lines per poll cycle).
tracing::debug!(
// instead of re-deriving by hand.
//
// TRACE, not debug: this runs on every `GET /models` poll and every
// input is fixed at load, so the line is byte-identical each time —
// about 8,600 a day per model at a 10s poll. It is worth having when
// reasoning about a context ceiling and worth nothing the rest of
// the time, and at debug it buries the lines that are not routine
// (#320).
tracing::trace!(
max_pos = profile.max_position_embeddings,
kv_bytes_per_token_per_card = profile.kv_bytes_per_token_per_card,
free_tightest_mb,
@@ -418,6 +503,57 @@ pub fn derive_limit(
#[cfg(test)]
mod tests {
/// The advertised limit for `qwen4_exp` must price both caches.
///
/// The main KV cache is 12 full-attention layers x 2 kv heads x 256
/// dims x 2 (K and V) x 2 bytes = 24 KiB/token, and the QSA indexer
/// adds 12 x 1 x 128 x 2 bytes = 3 KiB on top. Counting only the
/// first would advertise ~12.5% more context than the card can
/// hold — the failure mode #281 exists to stop, and the one the
/// epic's own arithmetic made before the indexer cache was found.
///
/// Driven from the checkpoint's verbatim `config.json`, so a field
/// that changes shape upstream fails here rather than in
/// production.
#[test]
fn a_qwen4_exp_profile_counts_the_indexer_cache_too() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.json");
std::fs::write(&path, super::super::arch::qwen4_exp::config::SHIPPED).unwrap();
let profile = profile_from_qwen4_exp_config(&path, 1)
.expect("the shipped config.json must yield a profile");
assert_eq!(profile.max_position_embeddings, 262_144);
assert_eq!(
profile.kv_bytes_per_token_per_card,
24 * 1024 + 3 * 1024,
"main KV plus the indexer's second cache"
);
}
/// The two hybrid archs must not answer for each other: whichever
/// builder runs first has to decline a config it does not own, or
/// the load path's `or_else` chain silently prices one model with
/// the other's physics.
#[test]
fn the_profile_builders_decline_each_others_configs() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.json");
std::fs::write(&path, super::super::arch::qwen4_exp::config::SHIPPED).unwrap();
assert!(
profile_from_qwen3_5_config(&path, 1).is_none(),
"the qwen3_5 builder must decline a qwen4_exp config"
);
let path = dir.path().join("other.json");
std::fs::write(&path, r#"{"model_type": "qwen3_next"}"#).unwrap();
assert!(
profile_from_qwen4_exp_config(&path, 1).is_none(),
"the qwen4_exp builder must decline everything else"
);
}
/// The deploy's smoke probe must not set the advertised context.
///
/// 19 tokens in 130 ms reads as ~145 tok/s on a host that measures
@@ -13,11 +13,13 @@
//! ARCH model state in this state slab will gain a companion
//! `tp_models: HashMap<TpHandle, Box<TpLeaderModel>>`.
use crate::harness::arch::qwen3_5::snapshot::KvCacheSnapshot;
use crate::harness::arch::snapshot::KvCacheSnapshot;
use crate::harness::candle::ModelArch;
#[cfg(feature = "cuda")]
use crate::harness::device_worker::jobs::TpHandle;
use crate::harness::device_worker::jobs::{ArchHandle, ImageHandle, ImageInput, Job, KvSnapshotId};
use crate::harness::device_worker::jobs::{
ArchHandle, DenseLoad, ImageHandle, ImageInput, Job, KvSnapshotId,
};
use crate::harness::image::ZImagePipeline;
#[cfg(feature = "cuda")]
use crate::harness::tp::TpLeaderModel;
@@ -131,17 +133,44 @@ pub(crate) fn run(device_index: u32, rx: Receiver<Job>, poisoned: Arc<AtomicBool
} => {
let result = load_gguf_inner(&state.device, &gguf_path, &model_id)
.map(|arch| insert_arch(&mut state, Box::new(arch)));
// Same reasoning as `LoadImage`: release load-time
// transients before serving, and — on the failure
// path — give back whatever the partial load had
// allocated before it gave up. Without this a load
// that OOMs leaves its bytes in the pool, where the
// driver still counts them against the next load.
trim_device_pool(&state);
let _ = reply.send(result);
}
Job::LoadDense {
config_path,
safetensors_paths,
model_id,
quant,
reply,
} => {
let result =
load_dense_inner(&state.device, &config_path, &safetensors_paths, &model_id)
.map(|arch| insert_arch(&mut state, Box::new(arch)));
let result = load_dense_inner(
&state.device,
&config_path,
&safetensors_paths,
&model_id,
quant,
)
.map(|arch| {
// Ask the model, once, while it is still in
// hand. See `DenseLoad`.
let supports_kv_snapshot = arch.supports_kv_snapshot();
let handle = insert_arch(&mut state, Box::new(arch));
DenseLoad {
handle,
supports_kv_snapshot,
}
});
// As for `LoadGguf`. It matters more here: a dense
// load that runs out of VRAM part-way has typically
// allocated tens of gigabytes first, and the evicted
// model waiting to be restored needs them back.
trim_device_pool(&state);
let _ = reply.send(result);
}
Job::DropArch { handle, reply } => {
@@ -158,6 +187,14 @@ pub(crate) fn run(device_index: u32, rx: Receiver<Job>, poisoned: Arc<AtomicBool
// the end of the arm; calling drop() explicitly just
// makes the intent visible.
drop(removed);
// Return the freed blocks to the system rather than
// the stream-ordered pool, so the next load actually
// sees the VRAM — `DropImage` has done this since the
// benjy cold-swap, and an unload is an unload whatever
// modality asked for it.
if was_present {
trim_device_pool(&state);
}
tracing::debug!(
device_index,
handle = handle.0,
@@ -374,6 +411,13 @@ pub(crate) fn run(device_index: u32, rx: Receiver<Job>, poisoned: Arc<AtomicBool
let was_present = removed.is_some();
drop(removed);
state.tp_kv_snapshots.retain(|(h, _), _| *h != handle);
// As `DropArch`: this is only the leader's shard —
// the other ranks free theirs in their own processes
// — but the leader's is the one sharing a card with
// whatever loads next.
if was_present {
trim_device_pool(&state);
}
tracing::debug!(
device_index,
tp_handle = handle.0,
@@ -895,6 +939,7 @@ fn load_dense_inner(
config_path: &std::path::Path,
safetensors_paths: &[std::path::PathBuf],
model_id: &str,
quant: Option<candle_core::quantized::GgmlDType>,
) -> anyhow::Result<ModelArch> {
use anyhow::Context;
use candle_core::DType;
@@ -988,6 +1033,36 @@ fn load_dense_inner(
.context("build Qwen3-Next dense model")?;
Ok(ModelArch::Qwen3_5Dense(model))
}
"qwen4_exp" => {
// Qwen3.8-Flash-Next. Mirrors the CPU path in
// `candle::load_arch_dense` — same config type, same
// sharded backend, same in-situ quantisation.
//
// ISQ is not optional for this architecture: the routed
// experts are 241.6 GB at BF16 (measured from the
// checkpoint's own headers), against 32.6 GB of card. A
// load without a `quant` will fail on memory rather than
// on anything this code can say (#315, #252).
let cfg = crate::harness::arch::qwen4_exp::config::Config::from_config_json(&cfg_text)
.context("parse qwen4_exp config.json")?;
let sharded_vb = unsafe {
candle_nn::var_builder::ShardedSafeTensors::var_builder(
safetensors_paths,
dtype,
device,
)
.context("build ShardedVarBuilder for qwen4_exp")?
};
let model = crate::harness::arch::qwen4_exp::model::Qwen4ExpForCausalLM::load(
&cfg,
dtype,
quant,
device,
&sharded_vb,
)
.context("build qwen4_exp model")?;
Ok(ModelArch::Qwen4Exp(Box::new(model)))
}
other => anyhow::bail!(
"unrouted supported model_type '{other}' — \
DENSE_SUPPORTED_MODEL_TYPES and load_dense_inner \
@@ -1045,7 +1120,7 @@ fn tp_load_shard_inner(
"qwen3_5" | "qwen3_next" => {
let cfg = crate::harness::tp::tp_qwen3_5::Config::from_config_json(config_json)
.context("parse Qwen3-Next Config JSON for leader load")?;
let quant_dtype = crate::harness::tp::worker::parse_quant_string(quant)?;
let quant_dtype = crate::harness::quant::parse_quant_string(quant)?;
TpLeaderModel::Qwen3_5(crate::harness::tp::tp_qwen3_5::TpQwen3_5ForCausalLM::load(
cfg,
&vb,
@@ -1179,7 +1254,7 @@ fn tp_assemble_kv_batch(
})?;
pairs.push((snap, *len));
}
let batch = crate::harness::arch::qwen3_5::snapshot::assemble_batch(&pairs)?;
let batch = crate::harness::arch::snapshot::assemble_batch(&pairs)?;
let model = tp_models
.get_mut(&handle)
.ok_or_else(|| anyhow::anyhow!("TpAssembleKvBatch: no model for handle {}", handle.0))?;
@@ -1212,9 +1287,8 @@ fn tp_extract_kv_rows(
.snapshot_kv_cache()?;
let mut total = 0u64;
for (&(row, prefix_len), &id) in rows.iter().zip(snapshot_ids) {
let snap = crate::harness::arch::qwen3_5::snapshot::extract_row(
&live, row, prefix_len, padded_len, steps,
)?;
let snap =
crate::harness::arch::snapshot::extract_row(&live, row, prefix_len, padded_len, steps)?;
total += snap.size_bytes();
state.tp_kv_snapshots.insert((handle, id), snap);
}
@@ -1337,7 +1411,7 @@ fn assemble_kv_batch(
})?;
pairs.push((snap, *len));
}
let batch = crate::harness::arch::qwen3_5::snapshot::assemble_batch(&pairs)?;
let batch = crate::harness::arch::snapshot::assemble_batch(&pairs)?;
let arch = models
.get_mut(&handle)
.ok_or_else(|| anyhow::anyhow!("AssembleKvBatch: no model for handle {}", handle.0))?;
@@ -1363,9 +1437,8 @@ fn extract_kv_rows(
.snapshot_kv_cache()?;
let mut out = Vec::with_capacity(rows.len());
for &(row, prefix_len) in rows {
let snap = crate::harness::arch::qwen3_5::snapshot::extract_row(
&live, row, prefix_len, padded_len, steps,
)?;
let snap =
crate::harness::arch::snapshot::extract_row(&live, row, prefix_len, padded_len, steps)?;
let id = KvSnapshotId(state.next_kv_snapshot_id);
state.next_kv_snapshot_id = state.next_kv_snapshot_id.wrapping_add(1);
let bytes = snap.size_bytes();
@@ -1671,3 +1744,42 @@ fn drain_poisoned(job: Job, device_index: u32) {
}
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Every architecture the preflight admits must have a loader arm
/// on **this** path, not only on the CPU one in
/// `candle::load_arch_dense`.
///
/// There are two dense loaders — the worker's (every CUDA load)
/// and candle's (the CPU fallback) — and one list gating both.
/// `qwen4_exp` was added to the list and to the CPU loader, so it
/// passed preflight and then hit the "unrouted supported
/// model_type" bail here: on a GPU host the model was
/// unloadable, in about 200 ms, having read nothing. The bail's
/// own comment said the two must stay in sync; nothing checked.
///
/// A bare `{"model_type": ...}` reaches the arm and then fails on
/// the typed parse, which is fine — this asserts only that an arm
/// exists to fail in.
#[test]
fn every_supported_dense_model_type_has_a_loader_arm() {
let dir = tempfile::tempdir().unwrap();
for model_type in crate::harness::candle::DENSE_SUPPORTED_MODEL_TYPES {
let path = dir.path().join(format!("{model_type}.json"));
std::fs::write(&path, format!(r#"{{"model_type": "{model_type}"}}"#)).unwrap();
let msg =
match load_dense_inner(&candle_core::Device::Cpu, &path, &[], model_type, None) {
Ok(_) => panic!("a bare config must not build a model"),
Err(e) => format!("{e:#}"),
};
assert!(
!msg.contains("unrouted"),
"`{model_type}` is in DENSE_SUPPORTED_MODEL_TYPES but load_dense_inner \
has no arm for it: {msg}"
);
}
}
}
@@ -19,6 +19,27 @@ use tokio::sync::oneshot;
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct ArchHandle(pub u64);
/// What a dense load hands back: the handle, plus what the arch it
/// built says about itself.
///
/// The async side cannot ask the model anything — it lives in the
/// worker's slab and never crosses the channel — so anything the
/// serving path needs to know has to ride back on the reply. Prefix
/// snapshot support (#11) is the first such fact, and it is here
/// rather than re-derived from `config.json` because that is exactly
/// how it went wrong: the async side matched a list of `model_type`
/// strings that `ModelArch::supports_kv_snapshot` had since grown
/// past, so `qwen4_exp` loaded with batched decode silently off.
/// A capability the arch answers for itself cannot drift from the
/// arch.
#[derive(Debug, Clone, Copy)]
pub struct DenseLoad {
pub handle: ArchHandle,
/// [`crate::harness::candle::ModelArch::supports_kv_snapshot`], as
/// answered by the model that was actually built.
pub supports_kv_snapshot: bool,
}
/// Opaque handle to a `TpLeaderModel` stored in the worker thread's
/// state slab. Same shape as [`ArchHandle`] but in a separate
/// namespace so the two slabs can coexist without ambiguity. Phase 3
@@ -107,7 +128,11 @@ pub enum Job {
config_path: PathBuf,
safetensors_paths: Vec<PathBuf>,
model_id: String,
reply: oneshot::Sender<Result<ArchHandle>>,
/// In-situ quantisation (#320): the catalogue's `quant`,
/// already parsed. `None` loads at the checkpoint dtype.
/// Archs that cannot quantise in situ ignore it.
quant: Option<candle_core::quantized::GgmlDType>,
reply: oneshot::Sender<Result<DenseLoad>>,
},
/// Remove the model from the slab and drop it. The `Drop` runs on
/// the worker thread so CUDA tensors release their memory on the
@@ -163,7 +188,7 @@ pub enum Job {
/// state and install it as the model's live state (#98). `seqs`
/// pairs each snapshot id with its true token length; attention
/// K/V is right-padded to the batch max and `cat`ed on dim 0 (see
/// `arch::qwen3_5::snapshot::assemble_batch`). Replies with the
/// `arch::snapshot::assemble_batch`). Replies with the
/// padded uniform KV length. The source snapshots remain stored —
/// the caller drops them via `DropKvSnapshot` when the sequences
/// leave the batch.
@@ -51,7 +51,7 @@ use tokio::sync::oneshot;
#[cfg(feature = "cuda")]
pub use jobs::TpHandle;
pub use jobs::{ArchHandle, ImageHandle, Job, KvSnapshotId};
pub use jobs::{ArchHandle, DenseLoad, ImageHandle, Job, KvSnapshotId};
/// Errors returned by `DeviceWorkerHandle` submit methods.
#[derive(Debug, thiserror::Error)]
@@ -221,7 +221,8 @@ impl DeviceWorkerHandle {
config_path: std::path::PathBuf,
safetensors_paths: Vec<std::path::PathBuf>,
model_id: String,
) -> Result<ArchHandle, WorkerError> {
quant: Option<candle_core::quantized::GgmlDType>,
) -> Result<DenseLoad, WorkerError> {
if self.poisoned.load(Ordering::Acquire) {
return Err(WorkerError::Poisoned {
device_index: self.device_index,
@@ -233,6 +234,7 @@ impl DeviceWorkerHandle {
config_path,
safetensors_paths,
model_id,
quant,
reply: reply_tx,
})
.map_err(|_| WorkerError::Gone {
@@ -1243,9 +1245,11 @@ mod tests {
fixture.join("config.json"),
vec![fixture.join("model.safetensors")],
"qwen3_next-tiny".into(),
None,
)
.await
.expect("load tiny fixture");
.expect("load tiny fixture")
.handle;
let prompts: [&[u32]; 3] = [&[1, 2, 3], &[4, 5], &[7, 3, 2, 5, 6]];
let steps: [&[u32]; 3] = [&[11, 12, 13], &[9, 8, 7], &[21, 22, 23]];
+3 -1
View File
@@ -1261,9 +1261,11 @@ mod tests {
fixture.join("config.json"),
vec![fixture.join("model.safetensors")],
"qwen3_next-tiny".into(),
None,
)
.await
.expect("load fixture");
.expect("load fixture")
.handle;
let admission_cfg = AdmissionConfig {
max_in_flight: 3,
+1
View File
@@ -12,6 +12,7 @@ pub mod image;
pub mod prefix_cache;
pub mod preflight;
pub mod preprocess;
pub(crate) mod quant;
pub mod reasoning_budget;
pub mod reasoning_effort;
pub mod sampling;
+49
View File
@@ -0,0 +1,49 @@
//! Catalogue quant names to ggml dtypes.
//!
//! Shared rather than owned by the TP path: `quant = "q4k"` in
//! `models.toml` means the same thing whichever loader reads it. It sat
//! behind `cfg(feature = "cuda")` while only the TP worker used it,
//! which put it out of reach of the single-GPU path — where, for
//! `qwen4_exp`, in-situ quantisation is the difference between a model
//! that loads and 241.6 GB that does not (#315).
/// Parse a `ModelSpec.quant` string into a `GgmlDType`. Accepts the
/// common ggml format names (case-insensitive). `None` and `Some("")`
/// both map to "no quantization".
///
/// Supported: `q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q8_1`,
/// `q2k`/`q2_k`, `q3k`/`q3_k`, `q4k`/`q4_k`, `q5k`/`q5_k`,
/// `q6k`/`q6_k`, `q8k`/`q8_k`, `f16`, `bf16`, `f32`. The underscore
/// is optional and the prefix is case-insensitive.
pub(crate) fn parse_quant_string(
s: Option<&str>,
) -> anyhow::Result<Option<candle_core::quantized::GgmlDType>> {
use candle_core::quantized::GgmlDType;
let s = match s {
Some(s) if !s.is_empty() => s,
_ => return Ok(None),
};
let normalised = s.to_ascii_lowercase().replace('_', "");
let dtype = match normalised.as_str() {
"q40" => GgmlDType::Q4_0,
"q41" => GgmlDType::Q4_1,
"q50" => GgmlDType::Q5_0,
"q51" => GgmlDType::Q5_1,
"q80" => GgmlDType::Q8_0,
"q81" => GgmlDType::Q8_1,
"q2k" => GgmlDType::Q2K,
"q3k" => GgmlDType::Q3K,
"q4k" | "q4km" => GgmlDType::Q4K,
"q5k" | "q5km" => GgmlDType::Q5K,
"q6k" => GgmlDType::Q6K,
"q8k" => GgmlDType::Q8K,
"f16" => GgmlDType::F16,
"bf16" => GgmlDType::BF16,
"f32" => GgmlDType::F32,
other => anyhow::bail!(
"unknown quant '{other}' (expected one of: q4_0, q4_1, q5_0, q5_1, q8_0, \
q8_1, q2k, q3k, q4k, q5k, q6k, q8k, f16, bf16, f32)"
),
};
Ok(Some(dtype))
}
+2 -2
View File
@@ -138,7 +138,7 @@ impl TpLeaderModel {
/// Capture rank 0's cache state for a prefix snapshot (#11).
pub fn snapshot_kv_cache(
&self,
) -> candle_core::Result<crate::harness::arch::qwen3_5::snapshot::KvCacheSnapshot> {
) -> candle_core::Result<crate::harness::arch::snapshot::KvCacheSnapshot> {
match self {
TpLeaderModel::Qwen3(_) => {
candle_core::bail!("snapshot_kv_cache: qwen3 (dense) has no snapshot support")
@@ -150,7 +150,7 @@ impl TpLeaderModel {
/// Replace rank 0's live cache state with a stored snapshot.
pub fn restore_kv_cache(
&mut self,
snap: &crate::harness::arch::qwen3_5::snapshot::KvCacheSnapshot,
snap: &crate::harness::arch::snapshot::KvCacheSnapshot,
) -> candle_core::Result<()> {
match self {
TpLeaderModel::Qwen3(_) => {
+2 -1
View File
@@ -45,10 +45,10 @@ use super::tp_linear::{ColumnParallelLinear, RowParallelLinear};
use crate::harness::arch::qwen3_5::linear_attn::repeat_interleave;
use crate::harness::arch::qwen3_5::rmsnorm::{Qwen3_5RmsNorm, Qwen3_5RmsNormGated, l2norm};
use crate::harness::arch::qwen3_5::rope::RotaryEmbedding;
use crate::harness::arch::qwen3_5::snapshot::{KvCacheSnapshot, LayerKvSnapshot};
use crate::harness::arch::qwen3_5::splice_runs;
use crate::harness::arch::qwen3_5::vision::VisionTower;
pub use crate::harness::arch::qwen3_5::{Config, TextConfig};
use crate::harness::arch::snapshot::{KvCacheSnapshot, LayerKvSnapshot};
// ─── linear-attention (Gated DeltaNet) ──────────────────────────────
@@ -1700,6 +1700,7 @@ impl TpQwen3_5Model {
Ok(KvCacheSnapshot {
layers,
rope_delta: self.rope_delta,
ple: None,
})
}
+13 -51
View File
@@ -16,6 +16,11 @@ use anyhow::Result;
use std::collections::HashMap;
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
// Only the CUDA shard-load path calls this; the CPU build of this
// module has no caller.
#[cfg(feature = "cuda")]
use crate::harness::quant::parse_quant_string;
use super::nccl_state::NcclState;
use super::rpc::{WorkerRequest, WorkerResponse};
@@ -119,7 +124,7 @@ impl WorkerModel {
/// `TpLeaderModel` support first.
fn snapshot_kv_cache(
&self,
) -> candle_core::Result<crate::harness::arch::qwen3_5::snapshot::KvCacheSnapshot> {
) -> candle_core::Result<crate::harness::arch::snapshot::KvCacheSnapshot> {
match self {
WorkerModel::Qwen3(_) => {
candle_core::bail!("snapshot_kv_cache: qwen3 (dense) has no snapshot support")
@@ -130,7 +135,7 @@ impl WorkerModel {
fn restore_kv_cache(
&mut self,
snap: &crate::harness::arch::qwen3_5::snapshot::KvCacheSnapshot,
snap: &crate::harness::arch::snapshot::KvCacheSnapshot,
) -> candle_core::Result<()> {
match self {
WorkerModel::Qwen3(_) => {
@@ -227,7 +232,7 @@ struct WorkerState {
/// pool and shared across all ranks. Dropped with the shard on
/// `UnloadModel`.
#[cfg(feature = "cuda")]
kv_snapshots: HashMap<(String, u64), crate::harness::arch::qwen3_5::snapshot::KvCacheSnapshot>,
kv_snapshots: HashMap<(String, u64), crate::harness::arch::snapshot::KvCacheSnapshot>,
/// Placeholder mirroring `models` on the non-cuda build.
#[cfg(not(feature = "cuda"))]
#[allow(dead_code)]
@@ -689,11 +694,10 @@ impl WorkerState {
};
pairs.push((snap, *len));
}
let result =
crate::harness::arch::qwen3_5::snapshot::assemble_batch(&pairs).and_then(|batch| {
model.restore_kv_cache(&batch.snapshot)?;
Ok(batch.padded_len)
});
let result = crate::harness::arch::snapshot::assemble_batch(&pairs).and_then(|batch| {
model.restore_kv_cache(&batch.snapshot)?;
Ok(batch.padded_len)
});
match result {
Ok(padded_len) => WorkerResponse::KvBatchAssembled {
padded_len: padded_len as u64,
@@ -756,7 +760,7 @@ impl WorkerState {
};
let mut total_bytes = 0u64;
for (&(row, prefix_len), &id) in rows.iter().zip(snapshot_ids) {
match crate::harness::arch::qwen3_5::snapshot::extract_row(
match crate::harness::arch::snapshot::extract_row(
&live, row, prefix_len, padded_len, steps,
) {
Ok(snap) => {
@@ -1036,45 +1040,3 @@ impl WorkerState {
}
}
}
/// Parse a `ModelSpec.quant` string into a `GgmlDType`. Accepts the
/// common ggml format names (case-insensitive). `None` and `Some("")`
/// both map to "no quantization".
///
/// Supported: `q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q8_1`,
/// `q2k`/`q2_k`, `q3k`/`q3_k`, `q4k`/`q4_k`, `q5k`/`q5_k`,
/// `q6k`/`q6_k`, `q8k`/`q8_k`, `f16`, `bf16`, `f32`. The underscore
/// is optional and the prefix is case-insensitive.
#[cfg(feature = "cuda")]
pub(crate) fn parse_quant_string(
s: Option<&str>,
) -> anyhow::Result<Option<candle_core::quantized::GgmlDType>> {
use candle_core::quantized::GgmlDType;
let s = match s {
Some(s) if !s.is_empty() => s,
_ => return Ok(None),
};
let normalised = s.to_ascii_lowercase().replace('_', "");
let dtype = match normalised.as_str() {
"q40" => GgmlDType::Q4_0,
"q41" => GgmlDType::Q4_1,
"q50" => GgmlDType::Q5_0,
"q51" => GgmlDType::Q5_1,
"q80" => GgmlDType::Q8_0,
"q81" => GgmlDType::Q8_1,
"q2k" => GgmlDType::Q2K,
"q3k" => GgmlDType::Q3K,
"q4k" | "q4km" => GgmlDType::Q4K,
"q5k" | "q5km" => GgmlDType::Q5K,
"q6k" => GgmlDType::Q6K,
"q8k" => GgmlDType::Q8K,
"f16" => GgmlDType::F16,
"bf16" => GgmlDType::BF16,
"f32" => GgmlDType::F32,
other => anyhow::bail!(
"unknown quant '{other}' (expected one of: q4_0, q4_1, q5_0, q5_1, q8_0, \
q8_1, q2k, q3k, q4k, q5k, q6k, q8k, f16, bf16, f32)"
),
};
Ok(Some(dtype))
}
+50 -13
View File
@@ -470,11 +470,21 @@ conventions — this is a solved problem in our tree.
**What is new is `group_size`.** `Qwen4ExpTextRMSNorm(dim, group_size)`
reshapes the last axis to `(-1, group_size)`, normalises each group
independently, and flattens back. Every 10240-wide norm in this model
(`hc_norm`, `norm_key`, `norm_query`, `norm_conv`,
`mtp.pre_fc_norm_hidden`) uses `group_size = 2560`, i.e. four
independent RMS normalisations, not one over 10240. A single norm over
the full 10240 vector compiles, runs, and is wrong.
independently, and flattens back. The 10240-wide norms in the decoder
(`hc_norm`, `norm_key`, `norm_query`, `norm_conv`) use
`group_size = 2560`, i.e. four independent RMS normalisations, not one
over 10240. A single norm over the full 10240 vector compiles, runs,
and is wrong.
**`mtp.pre_fc_norm_hidden` is the exception, and this document had it
wrong.** It is 10240 wide and **ungrouped** — one normalisation over
the whole vector. vLLM distinguishes the two cases explicitly: its
hyper-connection uses `GroupedGemmaRMSNorm` with
`group_size = hidden_size`, and the MTP head uses a plain
`GemmaRMSNorm(hidden_size * hc_count)`. Both classes exist in that
tree, so the choice is deliberate rather than an oversight. Grouping it
would rescale the draft head's input in four independent pieces and
show up as a poor acceptance rate, not as an error. See §9.
Variance is computed in f32 and the `(1 + w)` shift is applied in f32
before the multiply — same reasoning as the note already in
@@ -501,14 +511,41 @@ mtp.layers.0.* # a complete full_attention
**Transformers does not implement the MTP head.**
`Qwen4ExpPreTrainedModel._keys_to_ignore_on_load_unexpected = [r"^mtp.*"]`
— the weights are dropped on load. So for #313 the upstream reference is
*not* usable, and the authorities are instead llama.cpp PR #27836
(`--spec-type draft-mtp`) and vLLM's `speculative-config
{"method": "mtp"}` path. The tensor names above and the two `fc_*`
projections make the intended wiring clear (project the next token's
embedding and the accepted hidden state, sum, run one decoder layer,
reuse `lm_head`), but the accept rule and the proposal depth must come
from one of those two implementations, not from this document.
— the weights are dropped on load.
**The wiring, settled against vLLM** (`vllm/models/qwen4_exp/nvidia/
mtp.py`, read 2026-09-02). vLLM calls this shape
`residual_linear_shared`, as against the `Linear(2H, H)` + repeat that
other MTP variants use:
```
e = fc_embedding(pre_fc_norm_embedding(embed(t))) # [T, 2560]
h = pre_fc_norm_hidden(h.flatten(-2)) # ONE norm over 10240
h = fc_hidden(h.view(T, hc, 2560)) # per stream, shared matrix
h = e.unsqueeze(-2) + h # embedding broadcast to all four
h = h.flatten(-2) # [T, 10240] -> the decoder layer
```
Three things here are not guessable from the tensor shapes:
1. **`pre_fc_norm_hidden` is applied to the flattened 10240 and is
ungrouped** — see §8, which this document previously got wrong.
2. **The hidden state is the *pre-final-mixer* multi-stream**, i.e. `h`
after the last decoder layer and *before* `hyper_connection_mixer`
collapses it. Not the 2560 the LM head sees. vLLM calls this
"scheme A"; on later draft steps the head consumes its own previous
multi-stream rather than the target's.
3. **The head emits two things per step**: the collapsed `[T, 2560]`
for `lm_head`, and the pre-final-mixer `[T, 10240]` for the next
draft step.
PLE is forced off in the draft layer while the HC stream count is kept
(`ngram_context=None`), which is consistent with `mtp.layers.0` shipping
no `ple.*` tensors.
The accept rule is standard (greedy prefix match, or the Leviathan
rejection scheme) and is not architecture-specific; this document
previously conflated it with the wiring above.
`mtp.layers.0` carries its own full 512-expert MoE — 2.61 B params,
5.2 GB at BF16. It is not free to keep resident and its placement is