First two stages of the MTP speculative-decoding work on #96. Nothing calls this on the serving path — it loads weights we already ship and runs one draft step, so behaviour is unchanged.
S0 findings (the cheap questions, answered before writing code)
Read from the deployed checkpoint on beast and from vLLM, not inherited from the August notes:
tensor
shape
mtp.fc.weight
[5120, 10240]
100 MB — Linear(2H → H)
mtp.layers.0.self_attn.q_proj
[12288, 5120]
120 MB — output-gated (24×256 doubled)
k_proj / v_proj
[1024, 5120]
GQA 24/4
o_proj
[5120, 6144]
mlp.{gate,up,down}
17408 wide, dense
170 MB each
total
810 MB BF16 (~330 MB at q6k)
So the "849 MB doesn't reconcile with a full-width layer" puzzle resolves: it is 810 MB and the layer is full width. Heads divide for TP-2 (24→12, 4→2).
Qwen/Qwen3.5-0.8B ships the same head — 15 tensors of 488, 39 MB, same structure (fc [1024, 2048], output-gated q_proj, head_dim 256). That is the single-GPU dev target S2–S4 wanted: the whole loop can be built and measured on a small model on any fleet card, or on CPU in cargo test.
From vLLM's qwen3_next_mtp.py, quoted in the module doc:
pre_fc_norm_embedding on the embedding and pre_fc_norm_hidden on the hidden state, each before the concatenation, never one norm over the joined 2H.
the hidden state is the target's post-final-norm output (Qwen3NextModel::forward returns self.norm(...)), i.e. the tensor lm_head consumes — not the pre-norm residual, and not the pre-mixer stream arch/qwen4_exp/mtp.rs takes.
no mtp.lm_head and no mtp.embed_tokens in the checkpoint, and mtp_use_dedicated_embeddings: false, so both come from the target. That is why the resident cost is 810 MB rather than that plus a vocabulary-sized matrix.
S1 changes
arch/qwen3_5/mtp.rs (new): MtpHead::{load, forward, clear_kv_cache, present_in}. Modelled on the sibling head in arch/qwen4_exp/mtp.rs, including its discipline of recording which readings came from vLLM rather than from shapes.
arch/qwen3_5/decoder.rs: Qwen3_5DecoderLayer::load_typed — states the layer flavour instead of looking it up. mtp.layers.0 sits outside the layer_types list that indexes the target's layers, so asking that list about it answers for a different layer that happens to share the index. load now delegates.
arch/qwen3_5/mod.rs: mtp_num_hidden_layers / mtp_use_dedicated_embeddings on TextConfig; forward_with_hidden (logits plus the post-final-norm hidden state, every position); embed_tokens (the embedding the head borrows).
Tests
a_tiny_mtp_head_loads_and_drafts — loads from the mtp.* names the real checkpoints use, on a synthetic tiny head written as safetensors, and drafts.
the_fc_input_is_embedding_then_hidden — pins the concatenation order by value. Both halves are hidden_size wide, so the reversed reading loads, runs and returns finite numbers that are a different model; the probe makes fc read only the first half, so the correct order yields zero and the swap does not. This is the same failure mode arch/qwen4_exp/mtp.rs:229 pins for its own un-inferable norm: silent, and paid for in acceptance rate.
Triad green locally: fmt, clippy -D warnings, 1,104 tests.
What this is not
No draft/verify loop, no rollback, no engine change, no config that turns anything on. Per #96's staging, next are S2 multi-position logits (the forward already computes every position; one slice discards them), S3 the draft-cost probe — which matters more than it sounds, because each draft step reads the replicated q6k lm_head (~1.03 GB) on top of ~330 MB of head weights — and then S4, the acceptance observer that is the go/no-go gate and needs no rollback machinery at all because it commits nothing.
First two stages of the MTP speculative-decoding work on #96. **Nothing calls this on the serving path** — it loads weights we already ship and runs one draft step, so behaviour is unchanged.
## S0 findings (the cheap questions, answered before writing code)
Read from the **deployed** checkpoint on beast and from vLLM, not inherited from the August notes:
| tensor | shape | |
|---|---|---|
| `mtp.fc.weight` | [5120, 10240] | 100 MB — `Linear(2H → H)` |
| `mtp.layers.0.self_attn.q_proj` | [12288, 5120] | 120 MB — output-gated (24×256 doubled) |
| `k_proj` / `v_proj` | [1024, 5120] | GQA 24/4 |
| `o_proj` | [5120, 6144] | |
| `mlp.{gate,up,down}` | 17408 wide, dense | 170 MB each |
| **total** | | **810 MB BF16** (~330 MB at q6k) |
So the "849 MB doesn't reconcile with a full-width layer" puzzle resolves: it is 810 MB and the layer *is* full width. Heads divide for TP-2 (24→12, 4→2).
**`Qwen/Qwen3.5-0.8B` ships the same head** — 15 tensors of 488, 39 MB, same structure (`fc` [1024, 2048], output-gated q_proj, head_dim 256). That is the single-GPU dev target S2–S4 wanted: the whole loop can be built and measured on a small model on any fleet card, or on CPU in `cargo test`.
From vLLM's `qwen3_next_mtp.py`, quoted in the module doc:
- `hidden_states = torch.cat([inputs_embeds, hidden_states], dim=-1)` — **embedding first**.
- `pre_fc_norm_embedding` on the embedding and `pre_fc_norm_hidden` on the hidden state, each before the concatenation, never one norm over the joined 2H.
- the hidden state is the target's **post-final-norm** output (`Qwen3NextModel::forward` returns `self.norm(...)`), i.e. the tensor `lm_head` consumes — *not* the pre-norm residual, and not the pre-mixer stream `arch/qwen4_exp/mtp.rs` takes.
- no `mtp.lm_head` and no `mtp.embed_tokens` in the checkpoint, and `mtp_use_dedicated_embeddings: false`, so both come from the target. That is why the resident cost is 810 MB rather than that plus a vocabulary-sized matrix.
## S1 changes
- **`arch/qwen3_5/mtp.rs`** (new): `MtpHead::{load, forward, clear_kv_cache, present_in}`. Modelled on the sibling head in `arch/qwen4_exp/mtp.rs`, including its discipline of recording which readings came from vLLM rather than from shapes.
- **`arch/qwen3_5/decoder.rs`**: `Qwen3_5DecoderLayer::load_typed` — states the layer flavour instead of looking it up. `mtp.layers.0` sits outside the `layer_types` list that indexes the *target's* layers, so asking that list about it answers for a different layer that happens to share the index. `load` now delegates.
- **`arch/qwen3_5/mod.rs`**: `mtp_num_hidden_layers` / `mtp_use_dedicated_embeddings` on `TextConfig`; `forward_with_hidden` (logits plus the post-final-norm hidden state, every position); `embed_tokens` (the embedding the head borrows).
## Tests
- `a_tiny_mtp_head_loads_and_drafts` — loads from the `mtp.*` names the real checkpoints use, on a synthetic tiny head written as safetensors, and drafts.
- `the_fc_input_is_embedding_then_hidden` — **pins the concatenation order by value**. Both halves are `hidden_size` wide, so the reversed reading loads, runs and returns finite numbers that are a different model; the probe makes `fc` read only the first half, so the correct order yields zero and the swap does not. This is the same failure mode `arch/qwen4_exp/mtp.rs:229` pins for its own un-inferable norm: silent, and paid for in acceptance rate.
Triad green locally: fmt, clippy `-D warnings`, 1,104 tests.
## What this is not
No draft/verify loop, no rollback, no engine change, no config that turns anything on. Per #96's staging, next are **S2** multi-position logits (the forward already computes every position; one slice discards them), **S3** the draft-cost probe — which matters more than it sounds, because each draft step reads the replicated q6k `lm_head` (~1.03 GB) on top of ~330 MB of head weights — and then **S4**, the acceptance observer that is the go/no-go gate and needs no rollback machinery at all because it commits nothing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA
`Qwen/Qwen3.8-27B` ships a complete extra decoder layer under `mtp.*`
that nothing here has ever loaded — 15 tensors of 1199, 810 MB BF16 —
and `Qwen/Qwen3.5-0.8B` ships the same head at 39 MB. It is the drafter
speculative decoding needs, already inside the weights we download,
store and ignore. This loads it and runs one draft step. Nothing calls
it on the serving path yet.
Shapes, read from the deployed checkpoint rather than assumed:
mtp.fc.weight [5120, 10240] Linear(2H -> H)
mtp.layers.0.self_attn.q_proj [12288, 5120] output-gated
mtp.layers.0.self_attn.{k,v}_proj [1024, 5120] GQA 24/4
mtp.layers.0.self_attn.o_proj [5120, 6144]
mtp.layers.0.mlp.{gate,up,down} 17408 wide, dense
+ input/post_attention/q/k norms, mtp.norm, two pre_fc norms
So the head is an ordinary full-attention block plus one projection,
and `Qwen3_5DecoderLayer` loads it as-is. It gains `load_typed`, which
states the flavour instead of looking it up: `mtp.layers.0` sits outside
the `layer_types` list that indexes the target's layers, and asking that
list about it answers for a different layer that happens to share the
index.
Three things the tensor shapes cannot settle were read from vLLM's
`qwen3_next_mtp.py`, not inferred:
- the concatenation is `cat[embedding, hidden]`, and since both halves
are `hidden_size` wide the reversed reading loads, runs and returns
finite logits that are a different model. Pinned by value in a test
that fails on the swap.
- each half is normalised by its own norm before the concatenation,
never once over the joined 2H.
- the hidden state is the target's post-final-norm output — the tensor
its `lm_head` consumes — because `Qwen3NextModel::forward` returns
`self.norm(...)` and the runner hands that to the head. The sibling
head in `arch/qwen4_exp/mtp.rs` takes its pre-mixer stream instead;
the two do not generalise to each other.
`forward_with_hidden` returns logits alongside that hidden state, and
`embed_tokens` exposes the embedding the head borrows (the checkpoint
carries neither an `mtp.lm_head` nor an `mtp.embed_tokens`, and
`mtp_use_dedicated_embeddings` is false).
Next, per the staging on #96: multi-position logits (S2), the draft-cost
probe (S3), then the acceptance observer (S4) — which is the go/no-go,
measures acceptance on replayed agentic sessions rather than GSM8K, and
needs no rollback machinery because it commits nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA
A speculative verify pass forwards the drafted block in one pass and
needs the target's own token at each drafted position. The forward
already computes all of them — chunked prefill depends on it — and one
slice throws them away:
hidden.i((.., l - 1.., ..))?.apply(&self.lm_head)
So this adds the path that declines to, end to end and additive: nothing
existing changes shape or call site.
Qwen3_5ForCausalLM::forward_multi (B, L, vocab)
ModelArch::forward_multi dispatch, no squeeze_to_vocab
Job::ForwardLogitsMulti one CPU row per position
DeviceWorkerHandle::forward_logits_multi
`squeeze_to_vocab` is deliberately not applied: it bails on a
non-singleton leading dim, which is exactly the axis this exists to
keep.
The equivalence test is the point of the stage. `forward_multi`'s last
row must be bit-identical to what `forward` returns for the same tokens
at the same offset — the whole verify scheme rests on those logits being
the ones the model would have produced anyway — and its first row must
differ from its last, so a head that returned one row L times cannot
pass.
Per-position rows cross the channel as `Vec<Vec<f32>>`: at K+1 = 5 and a
248k vocabulary that is ~5 MB a round. Fine for a measurement path,
recorded in the dispatch doc as the first thing to replace with a
device-side argmax if the loop ever runs in production.
TP (`WorkerRequest::GenerateStepMulti`) is not here — single-GPU first,
which is also where S3/S4 will run, since `Qwen/Qwen3.5-0.8B` ships the
same MTP head at 39 MB.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA
grenade
changed title from feat(qwen3_5): load and run the MTP draft head (#96, S0/S1) to feat(neuron): MTP draft head + multi-position logits (#96, S0–S2)2026-09-23 07:33:14 +00:00
Qwen3_5ForCausalLM::forward_multi → ModelArch::forward_multi → Job::ForwardLogitsMulti → DeviceWorkerHandle::forward_logits_multi. All additive; no existing call site changes shape. squeeze_to_vocab is deliberately not applied, since it bails on a non-singleton leading dim — the axis this exists to keep.
The equivalence test is the point of the stage: forward_multi's last row is bit-identical to what forward returns for the same tokens at the same offset, because the verify pass rests on those logits being the ones the model would have produced anyway. Its first row must differ from its last, so a path that returned one row L times cannot pass — the fixture can express the failure it is there to catch.
Per-position rows cross the channel as Vec<Vec<f32>>: at K+1 = 5 against a 248k vocabulary that is ~5 MB a round. Fine while this is a measurement path; recorded in the dispatch doc as the first thing to replace with a device-side argmax if the loop ever runs in production.
TP is not in this PR.WorkerRequest::GenerateStepMulti and the TP slab mirror come next, and single-GPU is where S3/S4 will run anyway — Qwen/Qwen3.5-0.8B ships the same head at 39 MB, so the draft-cost probe and the acceptance observer can be built and measured without beast or a 27B in the loop.
Triad green locally: fmt, clippy -D warnings, 1,105 tests.
Next: S3 (draft-step cost — the denominator; each draft step reads the replicated q6k lm_head, ~1.03 GB, on top of ~330 MB of head weights, so c may be worse than the head's size suggests) and then S4, the acceptance observer that is the go/no-go gate.
## S2 added to this branch: logits at every position
`Qwen3_5ForCausalLM::forward_multi` → `ModelArch::forward_multi` → `Job::ForwardLogitsMulti` → `DeviceWorkerHandle::forward_logits_multi`. All additive; no existing call site changes shape. `squeeze_to_vocab` is deliberately not applied, since it bails on a non-singleton leading dim — the axis this exists to keep.
The equivalence test is the point of the stage: `forward_multi`'s **last row is bit-identical** to what `forward` returns for the same tokens at the same offset, because the verify pass rests on those logits being the ones the model would have produced anyway. Its **first row must differ from its last**, so a path that returned one row L times cannot pass — the fixture can express the failure it is there to catch.
Per-position rows cross the channel as `Vec<Vec<f32>>`: at K+1 = 5 against a 248k vocabulary that is ~5 MB a round. Fine while this is a measurement path; recorded in the dispatch doc as the first thing to replace with a device-side argmax if the loop ever runs in production.
**TP is not in this PR.** `WorkerRequest::GenerateStepMulti` and the TP slab mirror come next, and single-GPU is where S3/S4 will run anyway — `Qwen/Qwen3.5-0.8B` ships the same head at 39 MB, so the draft-cost probe and the acceptance observer can be built and measured without beast or a 27B in the loop.
Triad green locally: fmt, clippy `-D warnings`, 1,105 tests.
Next: **S3** (draft-step cost — the denominator; each draft step reads the replicated q6k `lm_head`, ~1.03 GB, on top of ~330 MB of head weights, so `c` may be worse than the head's size suggests) and then **S4**, the acceptance observer that is the go/no-go gate.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA
CI caught this and local runs did not, which is the useful part.
`forward` slices the last position *before* the LM head and multiplies
`(1, 1, H) x (H, vocab)`; `forward_multi` multiplies `(1, L, H) x (H,
vocab)` and slices after. Different GEMM shapes accumulate in a
different order, so the two differ in the last ULP or two — on this
fixture, 0.0016435327 against 0.0016435329. `assert_eq!` on the raw
f32s therefore asserted something about one CPU's BLAS blocking rather
than about the model: it passed here and failed on the runner.
Now: max absolute difference under 1e-5, plus an argmax assertion,
which is the property the verify pass actually depends on — it compares
tokens, not floats. The row-distinctness check stays.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA
grenade
merged commit f72c1e4c05 into main2026-09-23 08:39:27 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
First two stages of the MTP speculative-decoding work on #96. Nothing calls this on the serving path — it loads weights we already ship and runs one draft step, so behaviour is unchanged.
S0 findings (the cheap questions, answered before writing code)
Read from the deployed checkpoint on beast and from vLLM, not inherited from the August notes:
mtp.fc.weightLinear(2H → H)mtp.layers.0.self_attn.q_projk_proj/v_projo_projmlp.{gate,up,down}So the "849 MB doesn't reconcile with a full-width layer" puzzle resolves: it is 810 MB and the layer is full width. Heads divide for TP-2 (24→12, 4→2).
Qwen/Qwen3.5-0.8Bships the same head — 15 tensors of 488, 39 MB, same structure (fc[1024, 2048], output-gated q_proj, head_dim 256). That is the single-GPU dev target S2–S4 wanted: the whole loop can be built and measured on a small model on any fleet card, or on CPU incargo test.From vLLM's
qwen3_next_mtp.py, quoted in the module doc:hidden_states = torch.cat([inputs_embeds, hidden_states], dim=-1)— embedding first.pre_fc_norm_embeddingon the embedding andpre_fc_norm_hiddenon the hidden state, each before the concatenation, never one norm over the joined 2H.Qwen3NextModel::forwardreturnsself.norm(...)), i.e. the tensorlm_headconsumes — not the pre-norm residual, and not the pre-mixer streamarch/qwen4_exp/mtp.rstakes.mtp.lm_headand nomtp.embed_tokensin the checkpoint, andmtp_use_dedicated_embeddings: false, so both come from the target. That is why the resident cost is 810 MB rather than that plus a vocabulary-sized matrix.S1 changes
arch/qwen3_5/mtp.rs(new):MtpHead::{load, forward, clear_kv_cache, present_in}. Modelled on the sibling head inarch/qwen4_exp/mtp.rs, including its discipline of recording which readings came from vLLM rather than from shapes.arch/qwen3_5/decoder.rs:Qwen3_5DecoderLayer::load_typed— states the layer flavour instead of looking it up.mtp.layers.0sits outside thelayer_typeslist that indexes the target's layers, so asking that list about it answers for a different layer that happens to share the index.loadnow delegates.arch/qwen3_5/mod.rs:mtp_num_hidden_layers/mtp_use_dedicated_embeddingsonTextConfig;forward_with_hidden(logits plus the post-final-norm hidden state, every position);embed_tokens(the embedding the head borrows).Tests
a_tiny_mtp_head_loads_and_drafts— loads from themtp.*names the real checkpoints use, on a synthetic tiny head written as safetensors, and drafts.the_fc_input_is_embedding_then_hidden— pins the concatenation order by value. Both halves arehidden_sizewide, so the reversed reading loads, runs and returns finite numbers that are a different model; the probe makesfcread only the first half, so the correct order yields zero and the swap does not. This is the same failure modearch/qwen4_exp/mtp.rs:229pins for its own un-inferable norm: silent, and paid for in acceptance rate.Triad green locally: fmt, clippy
-D warnings, 1,104 tests.What this is not
No draft/verify loop, no rollback, no engine change, no config that turns anything on. Per #96's staging, next are S2 multi-position logits (the forward already computes every position; one slice discards them), S3 the draft-cost probe — which matters more than it sounds, because each draft step reads the replicated q6k
lm_head(~1.03 GB) on top of ~330 MB of head weights — and then S4, the acceptance observer that is the go/no-go gate and needs no rollback machinery at all because it commits nothing.🤖 Generated with Claude Code
https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA
`Qwen/Qwen3.8-27B` ships a complete extra decoder layer under `mtp.*` that nothing here has ever loaded — 15 tensors of 1199, 810 MB BF16 — and `Qwen/Qwen3.5-0.8B` ships the same head at 39 MB. It is the drafter speculative decoding needs, already inside the weights we download, store and ignore. This loads it and runs one draft step. Nothing calls it on the serving path yet. Shapes, read from the deployed checkpoint rather than assumed: mtp.fc.weight [5120, 10240] Linear(2H -> H) mtp.layers.0.self_attn.q_proj [12288, 5120] output-gated mtp.layers.0.self_attn.{k,v}_proj [1024, 5120] GQA 24/4 mtp.layers.0.self_attn.o_proj [5120, 6144] mtp.layers.0.mlp.{gate,up,down} 17408 wide, dense + input/post_attention/q/k norms, mtp.norm, two pre_fc norms So the head is an ordinary full-attention block plus one projection, and `Qwen3_5DecoderLayer` loads it as-is. It gains `load_typed`, which states the flavour instead of looking it up: `mtp.layers.0` sits outside the `layer_types` list that indexes the target's layers, and asking that list about it answers for a different layer that happens to share the index. Three things the tensor shapes cannot settle were read from vLLM's `qwen3_next_mtp.py`, not inferred: - the concatenation is `cat[embedding, hidden]`, and since both halves are `hidden_size` wide the reversed reading loads, runs and returns finite logits that are a different model. Pinned by value in a test that fails on the swap. - each half is normalised by its own norm before the concatenation, never once over the joined 2H. - the hidden state is the target's post-final-norm output — the tensor its `lm_head` consumes — because `Qwen3NextModel::forward` returns `self.norm(...)` and the runner hands that to the head. The sibling head in `arch/qwen4_exp/mtp.rs` takes its pre-mixer stream instead; the two do not generalise to each other. `forward_with_hidden` returns logits alongside that hidden state, and `embed_tokens` exposes the embedding the head borrows (the checkpoint carries neither an `mtp.lm_head` nor an `mtp.embed_tokens`, and `mtp_use_dedicated_embeddings` is false). Next, per the staging on #96: multi-position logits (S2), the draft-cost probe (S3), then the acceptance observer (S4) — which is the go/no-go, measures acceptance on replayed agentic sessions rather than GSM8K, and needs no rollback machinery because it commits nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MAA speculative verify pass forwards the drafted block in one pass and needs the target's own token at each drafted position. The forward already computes all of them — chunked prefill depends on it — and one slice throws them away: hidden.i((.., l - 1.., ..))?.apply(&self.lm_head) So this adds the path that declines to, end to end and additive: nothing existing changes shape or call site. Qwen3_5ForCausalLM::forward_multi (B, L, vocab) ModelArch::forward_multi dispatch, no squeeze_to_vocab Job::ForwardLogitsMulti one CPU row per position DeviceWorkerHandle::forward_logits_multi `squeeze_to_vocab` is deliberately not applied: it bails on a non-singleton leading dim, which is exactly the axis this exists to keep. The equivalence test is the point of the stage. `forward_multi`'s last row must be bit-identical to what `forward` returns for the same tokens at the same offset — the whole verify scheme rests on those logits being the ones the model would have produced anyway — and its first row must differ from its last, so a head that returned one row L times cannot pass. Per-position rows cross the channel as `Vec<Vec<f32>>`: at K+1 = 5 and a 248k vocabulary that is ~5 MB a round. Fine for a measurement path, recorded in the dispatch doc as the first thing to replace with a device-side argmax if the loop ever runs in production. TP (`WorkerRequest::GenerateStepMulti`) is not here — single-GPU first, which is also where S3/S4 will run, since `Qwen/Qwen3.5-0.8B` ships the same MTP head at 39 MB. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MAfeat(qwen3_5): load and run the MTP draft head (#96, S0/S1)to feat(neuron): MTP draft head + multi-position logits (#96, S0–S2)S2 added to this branch: logits at every position
Qwen3_5ForCausalLM::forward_multi→ModelArch::forward_multi→Job::ForwardLogitsMulti→DeviceWorkerHandle::forward_logits_multi. All additive; no existing call site changes shape.squeeze_to_vocabis deliberately not applied, since it bails on a non-singleton leading dim — the axis this exists to keep.The equivalence test is the point of the stage:
forward_multi's last row is bit-identical to whatforwardreturns for the same tokens at the same offset, because the verify pass rests on those logits being the ones the model would have produced anyway. Its first row must differ from its last, so a path that returned one row L times cannot pass — the fixture can express the failure it is there to catch.Per-position rows cross the channel as
Vec<Vec<f32>>: at K+1 = 5 against a 248k vocabulary that is ~5 MB a round. Fine while this is a measurement path; recorded in the dispatch doc as the first thing to replace with a device-side argmax if the loop ever runs in production.TP is not in this PR.
WorkerRequest::GenerateStepMultiand the TP slab mirror come next, and single-GPU is where S3/S4 will run anyway —Qwen/Qwen3.5-0.8Bships the same head at 39 MB, so the draft-cost probe and the acceptance observer can be built and measured without beast or a 27B in the loop.Triad green locally: fmt, clippy
-D warnings, 1,105 tests.Next: S3 (draft-step cost — the denominator; each draft step reads the replicated q6k
lm_head, ~1.03 GB, on top of ~330 MB of head weights, socmay be worse than the head's size suggests) and then S4, the acceptance observer that is the go/no-go gate.🤖 Generated with Claude Code
https://claude.ai/code/session_01XViS3RVfGTRaRSopLN68MA