Compare commits
25
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
15c664c5c3
|
||
|
|
7593ab3d5d
|
||
|
|
bf1e80ed55
|
||
|
|
a14d1af447
|
||
|
|
f1af926c30
|
||
|
|
430b6be62b
|
||
|
|
af40bac9b8
|
||
|
|
d5738bfb89
|
||
|
|
ecbfe22124
|
||
|
|
3e5d39807d
|
||
|
|
05a5aff25e
|
||
|
|
cd74eeaf1c
|
||
|
|
60bbe88ba7
|
||
|
|
499ed5013e
|
||
|
|
0ab159e144
|
||
|
|
e3a82e412d
|
||
|
|
2c852c05cb
|
||
|
|
1862f0ad2d
|
||
|
|
559ff46f2a
|
||
|
|
bc57e8423e
|
||
|
|
25157ca473
|
||
|
|
b303a453cb
|
||
|
|
da749be5d7
|
||
|
|
c62bfccb3a
|
||
|
|
2eb683441a
|
+85
-15
@@ -317,21 +317,82 @@ jobs:
|
||||
# restart decision below, exactly like the drop-in.
|
||||
neuron_toml_changed="${{ steps.neuronconf.outputs.changed }}"
|
||||
|
||||
# ── Desired per-model systemd drop-in ─────────────────────────
|
||||
# model.conf carries NEURON_MAX_PROMPT_TOKENS so the context cap
|
||||
# is deterministic per host and rolled out (with a restart) by
|
||||
# ── Desired per-host systemd drop-in ──────────────────────────
|
||||
# model.conf carries the environment neuron is started with, so
|
||||
# it is deterministic per host and rolled out (with a restart) by
|
||||
# this workflow, not hand-edited. It sorts after local.conf, so a
|
||||
# deploy-managed value wins over any manual local override of the
|
||||
# same variable. See doc/context-limits.md.
|
||||
# same variable.
|
||||
#
|
||||
# RUST_LOG is fleet-uniform and unconditional. The code default
|
||||
# is `info`, which is too coarse to diagnose a load or a routing
|
||||
# decision after the fact, and before this it was `debug` on
|
||||
# exactly one host because someone set it there by hand. A
|
||||
# verbosity you have to remember to set is a verbosity you find
|
||||
# out you did not have at the moment you needed it. Bumping a
|
||||
# module (`debug,neuron::harness::candle=trace`) is a one-line
|
||||
# change here, applied everywhere, with an audit trail.
|
||||
#
|
||||
# NEURON_MAX_PROMPT_TOKENS is per-host and stays conditional;
|
||||
# see doc/context-limits.md.
|
||||
conf=/etc/systemd/system/neuron.service.d/model.conf
|
||||
config_changed=${neuron_toml_changed:-0}
|
||||
rust_log="debug"
|
||||
desired=$(printf '%s\n%s\n%s' \
|
||||
"# Managed by .gitea/workflows/deploy.yml - do not edit by hand." \
|
||||
"[Service]" \
|
||||
"Environment=RUST_LOG=${rust_log}")
|
||||
if [ -n "${max_prompt_tokens}" ]; then
|
||||
desired=$(printf '%s\n%s\n%s\n%s' \
|
||||
"# Managed by .gitea/workflows/deploy.yml - do not edit by hand." \
|
||||
"# Per-model context cap; see doc/context-limits.md." \
|
||||
"[Service]" \
|
||||
desired=$(printf '%s\n%s' "${desired}" \
|
||||
"Environment=NEURON_MAX_PROMPT_TOKENS=${max_prompt_tokens}")
|
||||
[ "${desired}" = "$(cat "${conf}" 2>/dev/null || true)" ] || config_changed=1
|
||||
fi
|
||||
[ "${desired}" = "$(cat "${conf}" 2>/dev/null || true)" ] || config_changed=1
|
||||
|
||||
# ── Desired registry drop-in ──────────────────────────────────
|
||||
# Every model fetch goes through rustingface rather than
|
||||
# huggingface.co: it holds the upstream credential, and it
|
||||
# caches to the operator's store, so the second host to want a
|
||||
# checkpoint does not re-cross the internet for it.
|
||||
#
|
||||
# This makes rustingface a hard dependency for every cold load
|
||||
# on the fleet. That is the point, not a side effect. An outage
|
||||
# on our own host is one we can end; an outage at huggingface.co
|
||||
# is one we can only wait out, and a serving stack that stops
|
||||
# when someone else's CDN does is not sovereign in any sense
|
||||
# worth the word.
|
||||
#
|
||||
# So do NOT add a fallback to huggingface.co here. It would read
|
||||
# as a robustness improvement and would delete the property this
|
||||
# exists for: the fleet would silently resume depending on a
|
||||
# third party at exactly the moments that dependency matters.
|
||||
#
|
||||
# registry.conf sorts after local.conf, so the deploy-managed
|
||||
# values win over a hand-set HF_TOKEN — which is the point:
|
||||
# this is the mechanism that stops the registry being a
|
||||
# hand-synced per-host decision (#287).
|
||||
#
|
||||
# Mode 0600, and compared by hash under sudo, because it
|
||||
# carries a token. Written ONLY when the secret is non-empty:
|
||||
# pointing HF_ENDPOINT at a registry we cannot authenticate to
|
||||
# turns every cold load into a 401, which is strictly worse
|
||||
# than fetching from upstream.
|
||||
rf_conf=/etc/systemd/system/neuron.service.d/registry.conf
|
||||
rf_token='${{ secrets.RUSTINGFACE_TOKEN }}'
|
||||
rf_endpoint='https://rf.internal'
|
||||
if [ -n "${rf_token}" ]; then
|
||||
rf_desired=$(printf '%s\n%s\n%s\n%s\n%s\n%s' \
|
||||
"# Managed by .gitea/workflows/deploy.yml - do not edit by hand." \
|
||||
"# All model fetches go through rustingface, which holds the" \
|
||||
"# upstream Hugging Face credential and caches to our store." \
|
||||
"[Service]" \
|
||||
"Environment=HF_ENDPOINT=${rf_endpoint}" \
|
||||
"Environment=HF_TOKEN=${rf_token}")
|
||||
rf_have=$(sudo /usr/bin/sha256sum "${rf_conf}" 2>/dev/null | cut -d' ' -f1 || true)
|
||||
rf_want=$(printf '%s\n' "${rf_desired}" | sha256sum | cut -d' ' -f1)
|
||||
[ "${rf_have}" = "${rf_want}" ] || config_changed=1
|
||||
else
|
||||
echo "WARNING: RUSTINGFACE_TOKEN is empty - leaving ${rf_conf} as-is."
|
||||
echo " Fetches use whatever HF_ENDPOINT/HF_TOKEN the host already has."
|
||||
fi
|
||||
|
||||
# ── Package version gate (manifest rationale: see deploy-cortex) ──
|
||||
@@ -364,15 +425,24 @@ jobs:
|
||||
echo "installed=${installed} published=${latest:-unknown} pkg_changed=${pkg_changed} config_changed=${config_changed} — deploying"
|
||||
|
||||
# Write the drop-in (staged in gitea_ci's dir, installed root-owned).
|
||||
# Guarded on max_prompt_tokens as well as config_changed: since
|
||||
# a neuron.toml change also sets config_changed, an entry with
|
||||
# no max_prompt_tokens would otherwise reach this with `desired`
|
||||
# unset.
|
||||
if [ "${config_changed}" -eq 1 ] && [ -n "${max_prompt_tokens}" ]; then
|
||||
# `desired` is always set now that RUST_LOG is unconditional, so
|
||||
# config_changed alone is the guard.
|
||||
if [ "${config_changed}" -eq 1 ]; then
|
||||
printf '%s\n' "${desired}" > /var/lib/gitea_ci/model.conf
|
||||
sudo /usr/bin/install -o root -g root -m 0644 -D /var/lib/gitea_ci/model.conf "${conf}"
|
||||
rm -f /var/lib/gitea_ci/model.conf
|
||||
echo "applied ${conf}: NEURON_MAX_PROMPT_TOKENS=${max_prompt_tokens}"
|
||||
echo "applied ${conf}: RUST_LOG=${rust_log} NEURON_MAX_PROMPT_TOKENS=${max_prompt_tokens:-<unset>}"
|
||||
fi
|
||||
|
||||
# Staged 0600 in gitea_ci's own dir before install, so the token
|
||||
# is never briefly world-readable on the way in.
|
||||
if [ -n "${rf_token}" ] && [ "${rf_have:-}" != "${rf_want:-}" ]; then
|
||||
umask 077
|
||||
printf '%s\n' "${rf_desired}" > /var/lib/gitea_ci/registry.conf
|
||||
sudo /usr/bin/install -o root -g root -m 0600 -D /var/lib/gitea_ci/registry.conf "${rf_conf}"
|
||||
rm -f /var/lib/gitea_ci/registry.conf
|
||||
umask 022
|
||||
echo "applied ${rf_conf}: HF_ENDPOINT=${rf_endpoint} (token not logged)"
|
||||
fi
|
||||
|
||||
if systemctl is-active --quiet neuron.service; then
|
||||
|
||||
@@ -18,6 +18,14 @@ gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/neuron/neuron.toml
|
||||
# NEURON_MAX_PROMPT_TOKENS: gitea_ci stages it in its own dir, then
|
||||
# installs it root-owned. Exact source/dest paths; see doc/context-limits.md.
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/install -o root -g root -m 0644 -D /var/lib/gitea_ci/model.conf /etc/systemd/system/neuron.service.d/model.conf
|
||||
# deploy.yml also writes the registry drop-in, which points every model
|
||||
# fetch at rustingface and carries the client token for it. Mode 0600
|
||||
# rather than model.conf's 0644: this one holds a credential, and a
|
||||
# world-readable systemd drop-in is a world-readable token.
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/install -o root -g root -m 0600 -D /var/lib/gitea_ci/registry.conf /etc/systemd/system/neuron.service.d/registry.conf
|
||||
# Reading it back to decide whether it changed needs root too, since
|
||||
# 0600 excludes gitea_ci. Restricted to the one file.
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/systemd/system/neuron.service.d/registry.conf
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start neuron.service
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop neuron.service
|
||||
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now neuron.service
|
||||
|
||||
@@ -22,3 +22,4 @@
|
||||
|
||||
pub mod qwen3_5;
|
||||
pub mod qwen4_exp;
|
||||
pub mod snapshot;
|
||||
|
||||
@@ -18,6 +18,7 @@ use candle_core::{Module, Tensor};
|
||||
use candle_nn::var_builder::ShardedVarBuilder;
|
||||
use std::sync::Arc;
|
||||
|
||||
use super::super::snapshot::LayerKvSnapshot;
|
||||
use super::TextConfig;
|
||||
use super::full_attn::Qwen3_5Attention;
|
||||
use super::linear_attn::GatedDeltaNet;
|
||||
@@ -25,7 +26,6 @@ use super::mlp::Qwen3_5MLP;
|
||||
use super::moe::Qwen3_5MoeBlock;
|
||||
use super::rmsnorm::Qwen3_5RmsNorm;
|
||||
use super::rope::RotaryEmbedding;
|
||||
use super::snapshot::LayerKvSnapshot;
|
||||
|
||||
/// One of the two attention flavours sitting in a decoder layer's
|
||||
/// attention slot. Full-attention layers need the rotary table and
|
||||
|
||||
@@ -19,9 +19,12 @@ pub struct Qwen3_5MLP {
|
||||
impl Qwen3_5MLP {
|
||||
/// Construct directly from pre-built projections.
|
||||
///
|
||||
/// `qwen4_exp` ships its routed experts as fused 3D tensors rather
|
||||
/// than per-expert modules, so its loader slices them and builds the
|
||||
/// experts here — same SwiGLU, different storage.
|
||||
/// Test-only again: `qwen4_exp` briefly built its 512 experts this
|
||||
/// way before the fused tensors stayed banked (`Experts::Banked`),
|
||||
/// which removed the last caller outside tests. The differential
|
||||
/// test that proves banking neutral still needs it, to express the
|
||||
/// same weights in the layout it is being compared against.
|
||||
#[cfg(test)]
|
||||
pub(crate) fn from_weights(gate_proj: Linear, up_proj: Linear, down_proj: Linear) -> Self {
|
||||
Self {
|
||||
gate_proj,
|
||||
|
||||
@@ -79,7 +79,6 @@ pub mod mlp;
|
||||
pub mod moe;
|
||||
pub mod rmsnorm;
|
||||
pub mod rope;
|
||||
pub mod snapshot;
|
||||
pub mod vision;
|
||||
|
||||
use decoder::Qwen3_5DecoderLayer;
|
||||
@@ -534,15 +533,18 @@ impl Qwen3_5Model {
|
||||
/// counter as one consistent prefix snapshot (#11). Only valid at
|
||||
/// a token boundary — i.e. between forward calls, which is the
|
||||
/// only time the caller can reach this anyway.
|
||||
pub fn snapshot_kv_cache(&self) -> candle_core::Result<snapshot::KvCacheSnapshot> {
|
||||
pub fn snapshot_kv_cache(
|
||||
&self,
|
||||
) -> candle_core::Result<crate::harness::arch::snapshot::KvCacheSnapshot> {
|
||||
let layers = self
|
||||
.layers
|
||||
.iter()
|
||||
.map(|l| l.snapshot_kv())
|
||||
.collect::<candle_core::Result<Vec<_>>>()?;
|
||||
Ok(snapshot::KvCacheSnapshot {
|
||||
Ok(crate::harness::arch::snapshot::KvCacheSnapshot {
|
||||
layers,
|
||||
rope_delta: self.rope_delta,
|
||||
ple: None,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -550,7 +552,7 @@ impl Qwen3_5Model {
|
||||
/// snapshot. The snapshot stays valid for further restores.
|
||||
pub fn restore_kv_cache(
|
||||
&mut self,
|
||||
snap: &snapshot::KvCacheSnapshot,
|
||||
snap: &crate::harness::arch::snapshot::KvCacheSnapshot,
|
||||
) -> candle_core::Result<()> {
|
||||
if snap.layers.len() != self.layers.len() {
|
||||
candle_core::bail!(
|
||||
@@ -582,7 +584,7 @@ impl Qwen3_5Model {
|
||||
/// new token per batch row — with each row at its own sequence
|
||||
/// position `positions[i]` (typically `prefix_lens[i] + step`).
|
||||
/// The cache must hold batched state (see
|
||||
/// `snapshot::assemble_batch`); `attn_mask` is the padding mask
|
||||
/// `crate::harness::arch::snapshot::assemble_batch`); `attn_mask` is the padding mask
|
||||
/// from [`Self::batch_decode_mask`] (or `None` when no row is
|
||||
/// padded). Text-only: `rope_delta` is ignored — positions are
|
||||
/// explicit and vision requests never enter the batch path.
|
||||
@@ -1084,14 +1086,16 @@ impl Qwen3_5ForCausalLM {
|
||||
}
|
||||
|
||||
/// See [`Qwen3_5Model::snapshot_kv_cache`].
|
||||
pub fn snapshot_kv_cache(&self) -> candle_core::Result<snapshot::KvCacheSnapshot> {
|
||||
pub fn snapshot_kv_cache(
|
||||
&self,
|
||||
) -> candle_core::Result<crate::harness::arch::snapshot::KvCacheSnapshot> {
|
||||
self.base.snapshot_kv_cache()
|
||||
}
|
||||
|
||||
/// See [`Qwen3_5Model::restore_kv_cache`].
|
||||
pub fn restore_kv_cache(
|
||||
&mut self,
|
||||
snap: &snapshot::KvCacheSnapshot,
|
||||
snap: &crate::harness::arch::snapshot::KvCacheSnapshot,
|
||||
) -> candle_core::Result<()> {
|
||||
self.base.restore_kv_cache(snap)
|
||||
}
|
||||
|
||||
@@ -29,18 +29,238 @@
|
||||
//! replaces the loop behind the same `forward` signature.
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use candle_core::{DType, Module, Tensor};
|
||||
use candle_core::quantized::{GgmlDType, QMatMul, QTensor};
|
||||
use candle_core::{DType, IndexOp, Module, Tensor};
|
||||
use candle_nn::Linear;
|
||||
use candle_nn::var_builder::ShardedVarBuilder;
|
||||
|
||||
use super::TextConfig;
|
||||
use super::mlp::Qwen3_5MLP;
|
||||
|
||||
/// How a layer's routed experts are stored.
|
||||
///
|
||||
/// Two shapes, because the right one depends on whether the model fits.
|
||||
///
|
||||
/// `PerExpert` is the Qwen3-Next layout: one module per expert, sliced
|
||||
/// out at load. Fine when the weights are device-resident anyway.
|
||||
///
|
||||
/// `Banked` keeps the checkpoint's own fused tensors whole and takes a
|
||||
/// view per routed expert. `qwen4_exp` needs it: slicing its 512
|
||||
/// experts into modules copies **241.6 GB** at BF16 — 5.03 GB per
|
||||
/// layer — which fits in neither 64 GB of VRAM nor 123 GB of host RAM,
|
||||
/// at any quantisation. It is also the layout a host-resident bank with
|
||||
/// a device-side gather wants (#318), so the fused form is where that
|
||||
/// work starts rather than something it would have to undo.
|
||||
///
|
||||
/// Banking costs nothing per call: a slice along dim 0 of a contiguous
|
||||
/// tensor is contiguous, and `x.matmul(w.t())` is what `Linear` does
|
||||
/// anyway, so the views feed the same GEMMs without a copy.
|
||||
pub(crate) enum Experts {
|
||||
PerExpert(Vec<Qwen3_5MLP>),
|
||||
/// Quantised in situ at load (#315). One `(gate, up, down)` triple
|
||||
/// per expert.
|
||||
///
|
||||
/// This is what makes `qwen4_exp` loadable at all: 120.8 B routed
|
||||
/// parameters are 241.6 GB at BF16 and 67.9 GB at q4k, against
|
||||
/// beast's 64 GB of VRAM and ~110 GB of usable host RAM. Nothing
|
||||
/// about the arithmetic works until they shrink.
|
||||
///
|
||||
/// Quantising *from* the banked tensor is what keeps the peak
|
||||
/// bounded: one layer's fused BF16 tensor is 5.03 GB, and it is
|
||||
/// dropped once its 512 experts are quantised, so the full 241.6 GB
|
||||
/// never exists at once.
|
||||
Quantized(Vec<QuantizedExpert>),
|
||||
Banked {
|
||||
/// `(num_experts, 2 * intermediate, hidden)` — gate rows first,
|
||||
/// then up. Reversing them computes `silu(up) * gate`, which is
|
||||
/// a different function of the same weights (#312).
|
||||
gate_up: Tensor,
|
||||
/// `(num_experts, hidden, intermediate)`
|
||||
down: Tensor,
|
||||
intermediate: usize,
|
||||
},
|
||||
}
|
||||
|
||||
/// One expert's three projections, quantised.
|
||||
pub(crate) struct QuantizedExpert {
|
||||
gate: QMatMul,
|
||||
up: QMatMul,
|
||||
down: QMatMul,
|
||||
}
|
||||
|
||||
/// The quantisation type actually usable for rows of `row` elements.
|
||||
///
|
||||
/// ggml quantises along the last dimension, and a row must be a whole
|
||||
/// number of blocks. The k-quants (`Q2K`..`Q8K`) block on 256; the
|
||||
/// older types block on 32. `qwen4_exp`'s `down_proj` reduces over
|
||||
/// `moe_intermediate_size = 640`, which is 2.5 blocks of 256 — so a
|
||||
/// `quant = "q4k"` load fails on the very first layer with
|
||||
/// `quantized tensor must have their last dim divisible by block size
|
||||
/// [2560, 640] 256`, having read 131 shards to get there.
|
||||
///
|
||||
/// **This table is llama.cpp's**, from `tensor_type_fallback` in
|
||||
/// `src/llama-quant.cpp`, and the principle behind it is the part
|
||||
/// worth stating: **every fallback spends bits rather than saving
|
||||
/// them.** A k-quant carries a two-level scale hierarchy that the
|
||||
/// legacy 32-block types do not, so it is better than a legacy type at
|
||||
/// equal width — dropping `Q4K` (4.5 bpw) to `Q4_0` (also 4.5) would
|
||||
/// hold the memory constant and take the quality hit silently.
|
||||
/// Upstream declines that trade and pays a bit instead:
|
||||
///
|
||||
/// | requested | fallback | bpw |
|
||||
/// |---|---|---|
|
||||
/// | `Q2K`, `Q3K` | `Q4_0` | 2.63 / 3.44 -> 4.5 |
|
||||
/// | `Q4K` | `Q5_0` | 4.5 -> 5.5 |
|
||||
/// | `Q5K` | `Q5_1` | 5.5 -> 6.0 |
|
||||
/// | `Q6K`, `Q8K` | `Q8_0` | 6.56 -> 8.5 |
|
||||
///
|
||||
/// For `qwen4_exp` that is not free: `down_proj` is exactly a third of
|
||||
/// the 120.8 B routed parameters, so `Q5_0` rather than `Q4_0` costs
|
||||
/// **+5.0 GB** across the expert bank (67.9 -> 73.0 GB). We take it
|
||||
/// anyway. The cheaper table was an unmeasured divergence in the one
|
||||
/// direction that flatters the constraint we are straining against,
|
||||
/// and this matrix is the one a mixed k-quant scheme normally treats
|
||||
/// as quality-critical. If the 5 GB turns out to buy nothing, #315 is
|
||||
/// where that gets measured and reversed on evidence.
|
||||
///
|
||||
/// A row that divides 32 either — vanishingly rare, and impossible for
|
||||
/// this checkpoint since 640 and 2560 are both multiples of 32 — falls
|
||||
/// back to F16 as upstream does, loudly. It is a 3.5x blowup on that
|
||||
/// tensor, so it must never be quiet.
|
||||
pub(crate) fn quant_for_row(requested: GgmlDType, row: usize) -> candle_core::Result<GgmlDType> {
|
||||
if row.is_multiple_of(requested.block_size()) {
|
||||
return Ok(requested);
|
||||
}
|
||||
let fallback = match requested {
|
||||
GgmlDType::Q2K | GgmlDType::Q3K => GgmlDType::Q4_0,
|
||||
GgmlDType::Q4K => GgmlDType::Q5_0,
|
||||
GgmlDType::Q5K => GgmlDType::Q5_1,
|
||||
GgmlDType::Q6K | GgmlDType::Q8K => GgmlDType::Q8_0,
|
||||
// Already a 32-block type (or not a quant at all): there is no
|
||||
// smaller block to demote to, so the F16 check below decides.
|
||||
other => other,
|
||||
};
|
||||
if row.is_multiple_of(fallback.block_size()) {
|
||||
return Ok(fallback);
|
||||
}
|
||||
tracing::warn!(
|
||||
?requested,
|
||||
?fallback,
|
||||
row,
|
||||
"row divides neither the requested block size nor its 32-wide \
|
||||
fallback; storing this tensor as F16 — 3.5x the bytes"
|
||||
);
|
||||
Ok(GgmlDType::F16)
|
||||
}
|
||||
|
||||
impl Experts {
|
||||
pub(crate) fn len(&self) -> usize {
|
||||
match self {
|
||||
Experts::PerExpert(v) => v.len(),
|
||||
Experts::Banked { gate_up, .. } => gate_up.dims()[0],
|
||||
Experts::Quantized(v) => v.len(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Quantise a banked pair into per-expert triples.
|
||||
///
|
||||
/// Takes the fused tensors by reference and drops nothing itself —
|
||||
/// the caller owns the peak, and should hold one layer at a time.
|
||||
///
|
||||
/// The two halves may not end up the same dtype: see
|
||||
/// [`quant_for_row`]. `qwen4_exp`'s `down_proj` reduces over 640,
|
||||
/// which no k-quant can block.
|
||||
pub(crate) fn quantize_banked(
|
||||
gate_up: &Tensor,
|
||||
down: &Tensor,
|
||||
intermediate: usize,
|
||||
dtype: GgmlDType,
|
||||
) -> candle_core::Result<Self> {
|
||||
let n = gate_up.dims()[0];
|
||||
// Decided once, from the shapes, rather than per expert —
|
||||
// every expert in a layer has the same two row widths, and
|
||||
// 512 identical log lines per layer would bury the fact.
|
||||
let gate_up_dtype = quant_for_row(dtype, gate_up.dims()[2])?;
|
||||
let down_dtype = quant_for_row(dtype, down.dims()[2])?;
|
||||
if gate_up_dtype != dtype || down_dtype != dtype {
|
||||
tracing::info!(
|
||||
requested = ?dtype,
|
||||
gate_up_row = gate_up.dims()[2],
|
||||
gate_up_dtype = ?gate_up_dtype,
|
||||
down_row = down.dims()[2],
|
||||
down_dtype = ?down_dtype,
|
||||
"expert quantisation: a row width does not divide the requested \
|
||||
block size; substituting a 32-block type for that half"
|
||||
);
|
||||
}
|
||||
let mut out = Vec::with_capacity(n);
|
||||
for e in 0..n {
|
||||
let gu = gate_up.i(e)?;
|
||||
// The source must own its storage outright.
|
||||
// `QTensor::quantize` reads `src.storage()` — the whole
|
||||
// backing buffer, ignoring shape and offset — so a view into
|
||||
// the fused tensor hands it every expert's bytes while the
|
||||
// shape claims one, and candle panics on a block-count
|
||||
// mismatch inside `from_float`. Neither `contiguous()` nor
|
||||
// `copy()` saves you here: a slice along dim 0 is already
|
||||
// contiguous, so both are free to hand back something still
|
||||
// sharing the parent's buffer. Round-tripping through a Vec
|
||||
// is the part that actually allocates, and it is also where
|
||||
// the f32 the quantiser wants comes from.
|
||||
let q = |t: Tensor, dtype: GgmlDType| -> candle_core::Result<QMatMul> {
|
||||
let dims = t.dims().to_vec();
|
||||
let vals = t.to_dtype(DType::F32)?.flatten_all()?.to_vec1::<f32>()?;
|
||||
let owned = Tensor::from_vec(vals, dims, t.device())?;
|
||||
QMatMul::from_qtensor(QTensor::quantize(&owned, dtype)?)
|
||||
};
|
||||
out.push(QuantizedExpert {
|
||||
gate: q(gu.narrow(0, 0, intermediate)?, gate_up_dtype)?,
|
||||
up: q(gu.narrow(0, intermediate, intermediate)?, gate_up_dtype)?,
|
||||
down: q(down.i(e)?, down_dtype)?,
|
||||
});
|
||||
}
|
||||
Ok(Experts::Quantized(out))
|
||||
}
|
||||
|
||||
/// Expert `e` applied to the rows routed to it.
|
||||
pub(crate) fn forward_one(&self, e: usize, xs: &Tensor) -> candle_core::Result<Tensor> {
|
||||
match self {
|
||||
Experts::PerExpert(v) => v[e].forward(xs),
|
||||
Experts::Banked {
|
||||
gate_up,
|
||||
down,
|
||||
intermediate,
|
||||
} => {
|
||||
let gu = gate_up.i(e)?;
|
||||
let gate_w = gu.narrow(0, 0, *intermediate)?;
|
||||
let up_w = gu.narrow(0, *intermediate, *intermediate)?;
|
||||
let lhs = candle_nn::ops::silu(&xs.matmul(&gate_w.t()?)?)?;
|
||||
let rhs = xs.matmul(&up_w.t()?)?;
|
||||
(lhs * rhs)?.matmul(&down.i(e)?.t()?)
|
||||
}
|
||||
Experts::Quantized(v) => {
|
||||
let q = &v[e];
|
||||
// The quantised kernels accumulate in f32 whatever the
|
||||
// activation dtype, so the cast is explicit here rather
|
||||
// than implied — and the result comes back in the
|
||||
// caller's dtype so the routing arithmetic upstream
|
||||
// does not have to know which layout it got.
|
||||
let dtype = xs.dtype();
|
||||
let xs32 = xs.to_dtype(DType::F32)?;
|
||||
let lhs = candle_nn::ops::silu(&q.gate.forward(&xs32)?)?;
|
||||
let rhs = q.up.forward(&xs32)?;
|
||||
q.down.forward(&(lhs * rhs)?)?.to_dtype(dtype)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub struct Qwen3_5MoeBlock {
|
||||
/// Router: `(num_experts, hidden)`, checkpoint name `mlp.gate`.
|
||||
gate: Linear,
|
||||
/// Routed experts, checkpoint names `mlp.experts.{i}.{gate,up,down}_proj`.
|
||||
experts: Vec<Qwen3_5MLP>,
|
||||
/// Routed experts. Per-expert modules for `qwen3_5`; the
|
||||
/// checkpoint's fused tensors for `qwen4_exp` — see [`Experts`].
|
||||
experts: Experts,
|
||||
/// Always-on expert, `mlp.shared_expert.*`. `None` when the config
|
||||
/// declares no shared expert (Qwen3-30B-A3B style).
|
||||
shared_expert: Option<Qwen3_5MLP>,
|
||||
@@ -62,7 +282,7 @@ impl Qwen3_5MoeBlock {
|
||||
/// `qwen3_5` config.
|
||||
pub(crate) fn from_parts(
|
||||
gate: Linear,
|
||||
experts: Vec<Qwen3_5MLP>,
|
||||
experts: Experts,
|
||||
shared_expert: Option<Qwen3_5MLP>,
|
||||
shared_expert_gate: Option<Linear>,
|
||||
num_experts_per_tok: usize,
|
||||
@@ -129,7 +349,7 @@ impl Qwen3_5MoeBlock {
|
||||
|
||||
Ok(Self {
|
||||
gate,
|
||||
experts,
|
||||
experts: Experts::PerExpert(experts),
|
||||
shared_expert,
|
||||
shared_expert_gate,
|
||||
num_experts_per_tok: cfg.num_experts_per_tok,
|
||||
@@ -201,13 +421,13 @@ impl Module for Qwen3_5MoeBlock {
|
||||
)?;
|
||||
|
||||
let mut ys = xs_flat.zeros_like()?;
|
||||
for (e, expert) in self.experts.iter().enumerate() {
|
||||
for e in 0..self.experts.len() {
|
||||
if tokens_for[e].is_empty() {
|
||||
continue;
|
||||
}
|
||||
let rows = Tensor::new(tokens_for[e].as_slice(), xs.device())?;
|
||||
let picked = xs_flat.index_select(&rows, 0)?;
|
||||
let out = expert.forward(&picked)?;
|
||||
let out = self.experts.forward_one(e, &picked)?;
|
||||
let w = Tensor::new(weights_for[e].as_slice(), xs.device())?
|
||||
.to_dtype(out.dtype())?
|
||||
.reshape(((), 1))?;
|
||||
@@ -252,7 +472,7 @@ mod tests {
|
||||
|
||||
let block = Qwen3_5MoeBlock {
|
||||
gate: Linear::new(randn(&[n_exp, hidden]), None),
|
||||
experts: (0..n_exp).map(|_| rand_mlp(hidden, inter)).collect(),
|
||||
experts: Experts::PerExpert((0..n_exp).map(|_| rand_mlp(hidden, inter)).collect()),
|
||||
shared_expert: Some(rand_mlp(hidden, inter)),
|
||||
shared_expert_gate: Some(Linear::new(randn(&[1, hidden]), None)),
|
||||
num_experts_per_tok: top_k,
|
||||
@@ -283,8 +503,9 @@ mod tests {
|
||||
let mut expect = vec![0f32; hidden];
|
||||
for &e in selected {
|
||||
let w = probs[e] / denom;
|
||||
let out: Vec<f32> = block.experts[e]
|
||||
.forward(&row)
|
||||
let out: Vec<f32> = block
|
||||
.experts
|
||||
.forward_one(e, &row)
|
||||
.unwrap()
|
||||
.flatten_all()
|
||||
.unwrap()
|
||||
@@ -336,7 +557,7 @@ mod tests {
|
||||
let (hidden, inter, n_exp) = (4, 2, 3);
|
||||
let block = Qwen3_5MoeBlock {
|
||||
gate: Linear::new(randn(&[n_exp, hidden]), None),
|
||||
experts: (0..n_exp).map(|_| rand_mlp(hidden, inter)).collect(),
|
||||
experts: Experts::PerExpert((0..n_exp).map(|_| rand_mlp(hidden, inter)).collect()),
|
||||
shared_expert: None,
|
||||
shared_expert_gate: None,
|
||||
num_experts_per_tok: 1,
|
||||
@@ -368,8 +589,9 @@ mod tests {
|
||||
.max_by(|&a, &b| exps[a].partial_cmp(&exps[b]).unwrap())
|
||||
.unwrap();
|
||||
let w = exps[best] / sum;
|
||||
let out: Vec<f32> = block.experts[best]
|
||||
.forward(&xs.reshape(((), hidden)).unwrap())
|
||||
let out: Vec<f32> = block
|
||||
.experts
|
||||
.forward_one(best, &xs.reshape(((), hidden)).unwrap())
|
||||
.unwrap()
|
||||
.flatten_all()
|
||||
.unwrap()
|
||||
@@ -383,4 +605,216 @@ mod tests {
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Banking is a storage change, so the two layouts must compute the
|
||||
/// same function. Same weights, expressed both ways, asserted
|
||||
/// identical — otherwise "we only changed where the bytes live" is
|
||||
/// a claim rather than a fact.
|
||||
#[test]
|
||||
fn banked_experts_equal_per_expert_modules() {
|
||||
let (n_experts, hidden, inter) = (3usize, 4usize, 2usize);
|
||||
|
||||
// One source of truth for the weights, laid out both ways.
|
||||
let gate_up = randn(&[n_experts, inter * 2, hidden]);
|
||||
let down = randn(&[n_experts, hidden, inter]);
|
||||
|
||||
let per_expert: Vec<Qwen3_5MLP> = (0..n_experts)
|
||||
.map(|e| {
|
||||
let gu = gate_up.i(e).unwrap();
|
||||
Qwen3_5MLP::from_weights(
|
||||
Linear::new(gu.narrow(0, 0, inter).unwrap().contiguous().unwrap(), None),
|
||||
Linear::new(
|
||||
gu.narrow(0, inter, inter).unwrap().contiguous().unwrap(),
|
||||
None,
|
||||
),
|
||||
Linear::new(down.i(e).unwrap().contiguous().unwrap(), None),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
|
||||
let sliced = Experts::PerExpert(per_expert);
|
||||
let banked = Experts::Banked {
|
||||
gate_up,
|
||||
down,
|
||||
intermediate: inter,
|
||||
};
|
||||
assert_eq!(sliced.len(), banked.len());
|
||||
|
||||
let xs = randn(&[5, hidden]);
|
||||
for e in 0..n_experts {
|
||||
let a: Vec<f32> = sliced
|
||||
.forward_one(e, &xs)
|
||||
.unwrap()
|
||||
.flatten_all()
|
||||
.unwrap()
|
||||
.to_vec1()
|
||||
.unwrap();
|
||||
let b: Vec<f32> = banked
|
||||
.forward_one(e, &xs)
|
||||
.unwrap()
|
||||
.flatten_all()
|
||||
.unwrap()
|
||||
.to_vec1()
|
||||
.unwrap();
|
||||
assert_eq!(a.len(), b.len());
|
||||
for (x, y) in a.iter().zip(b.iter()) {
|
||||
assert!(
|
||||
(x - y).abs() < 1e-5,
|
||||
"expert {e} differs between layouts: {a:?} vs {b:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// And the halves are not interchangeable: reading `gate_up` the
|
||||
/// other way round computes silu(up) * gate, which the banked path
|
||||
/// would do just as silently as the sliced one.
|
||||
#[test]
|
||||
fn banked_reads_the_gate_before_the_up() {
|
||||
let dev = Device::Cpu;
|
||||
// gate row picks channel 0, up row picks channel 1.
|
||||
let gate_up = Tensor::from_vec(vec![1.0f32, 0.0, 0.0, 1.0], (1, 2, 2), &dev).unwrap();
|
||||
let down = Tensor::from_vec(vec![1.0f32, 0.0], (1, 2, 1), &dev).unwrap();
|
||||
let banked = Experts::Banked {
|
||||
gate_up,
|
||||
down,
|
||||
intermediate: 1,
|
||||
};
|
||||
let xs = Tensor::from_vec(vec![1.0f32, 2.0], (1, 2), &dev).unwrap();
|
||||
let got: Vec<f32> = banked
|
||||
.forward_one(0, &xs)
|
||||
.unwrap()
|
||||
.flatten_all()
|
||||
.unwrap()
|
||||
.to_vec1()
|
||||
.unwrap();
|
||||
|
||||
// silu(gate.x) * (up.x) = silu(1) * 2
|
||||
let want = (1.0f32 / (1.0 + (-1.0f32).exp())) * 2.0;
|
||||
assert!((got[0] - want).abs() < 1e-5, "got {got:?} want {want}");
|
||||
// the swapped reading would be silu(2) * 1
|
||||
let swapped = (2.0f32 / (1.0 + (-2.0f32).exp())) * 1.0;
|
||||
assert!(
|
||||
(got[0] - swapped).abs() > 1e-3,
|
||||
"this is silu(up) * gate: {got:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// In-situ quantisation has to approximate the dense expert, not
|
||||
/// merely produce numbers of the right shape.
|
||||
///
|
||||
/// Dimensions are 256 because the k-quants block on 256 elements —
|
||||
/// a tiny fixture silently cannot be quantised at all, which is the
|
||||
/// first thing that bites when this is tried on a toy model.
|
||||
///
|
||||
/// `qwen4_exp`'s real expert widths, which no k-quant can block.
|
||||
///
|
||||
/// `moe_intermediate_size` is 640, so `down_proj` is `[2560, 640]`
|
||||
/// and its 640-element rows are 2.5 blocks of 256. The load died
|
||||
/// on beast at decoder layer 0 with `quantized tensor must have
|
||||
/// their last dim divisible by block size [2560, 640] 256`, after
|
||||
/// resolving all 131 shards.
|
||||
///
|
||||
/// The fixture that missed it was 256 wide in both dimensions —
|
||||
/// the smallest width a k-quant accepts — so every row divided
|
||||
/// and the orientation could not matter. This one uses the
|
||||
/// checkpoint's own numbers, scaled down only in expert count.
|
||||
#[test]
|
||||
fn the_real_expert_widths_quantise_despite_a_640_wide_reduction() {
|
||||
assert_eq!(
|
||||
quant_for_row(GgmlDType::Q4K, 2560).unwrap(),
|
||||
GgmlDType::Q4K,
|
||||
"gate_up reduces over hidden_size, which is 10 whole blocks"
|
||||
);
|
||||
assert_eq!(
|
||||
quant_for_row(GgmlDType::Q4K, 640).unwrap(),
|
||||
GgmlDType::Q5_0,
|
||||
"down reduces over moe_intermediate_size, which is not"
|
||||
);
|
||||
// llama.cpp's `tensor_type_fallback` table, which spends bits
|
||||
// rather than saving them: a legacy 32-block type is worse than
|
||||
// a k-quant at equal width, so upstream pays one to avoid the
|
||||
// quality cliff. Q4_0 above would be 4.5 bpw like Q4K and would
|
||||
// save 5.0 GB across the expert bank — that is the trade we
|
||||
// declined, and this is where it is pinned.
|
||||
assert_eq!(quant_for_row(GgmlDType::Q5K, 640).unwrap(), GgmlDType::Q5_1);
|
||||
assert_eq!(quant_for_row(GgmlDType::Q2K, 640).unwrap(), GgmlDType::Q4_0);
|
||||
assert_eq!(quant_for_row(GgmlDType::Q3K, 640).unwrap(), GgmlDType::Q4_0);
|
||||
assert_eq!(quant_for_row(GgmlDType::Q6K, 640).unwrap(), GgmlDType::Q8_0);
|
||||
assert_eq!(
|
||||
quant_for_row(GgmlDType::Q8_0, 640).unwrap(),
|
||||
GgmlDType::Q8_0
|
||||
);
|
||||
// 100 divides neither 256 nor 32: F16, as upstream does.
|
||||
assert_eq!(quant_for_row(GgmlDType::Q4K, 100).unwrap(), GgmlDType::F16);
|
||||
|
||||
// And the whole path runs at those widths.
|
||||
let (n_experts, hidden, inter) = (2usize, 2560usize, 640usize);
|
||||
let gate_up = randn(&[n_experts, inter * 2, hidden]);
|
||||
let down = randn(&[n_experts, hidden, inter]);
|
||||
let quantised = Experts::quantize_banked(&gate_up, &down, inter, GgmlDType::Q4K)
|
||||
.expect("the real widths must quantise");
|
||||
let xs = randn(&[2, hidden]);
|
||||
let got = quantised.forward_one(0, &xs).expect("forward");
|
||||
assert_eq!(got.dims(), &[2, hidden]);
|
||||
assert!(
|
||||
got.flatten_all()
|
||||
.unwrap()
|
||||
.to_vec1::<f32>()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.all(|v| v.is_finite()),
|
||||
"a substituted dtype must still produce finite output"
|
||||
);
|
||||
}
|
||||
|
||||
/// The assertion is relative error against the dense output, with a
|
||||
/// noise baseline: a q8_0 expert should be close, a q4k expert
|
||||
/// looser, and both far nearer the truth than an unrelated expert
|
||||
/// is. Without that last comparison a "quantised" path that
|
||||
/// returned anything smoothly wrong would pass.
|
||||
#[test]
|
||||
fn quantised_experts_approximate_the_dense_ones() {
|
||||
let (n_experts, hidden, inter) = (2usize, 256usize, 256usize);
|
||||
let gate_up = randn(&[n_experts, inter * 2, hidden]);
|
||||
let down = randn(&[n_experts, hidden, inter]);
|
||||
let banked = Experts::Banked {
|
||||
gate_up: gate_up.clone(),
|
||||
down: down.clone(),
|
||||
intermediate: inter,
|
||||
};
|
||||
let xs = randn(&[4, hidden]);
|
||||
|
||||
let rel_err = |a: &[f32], b: &[f32]| -> f32 {
|
||||
let num: f32 = a.iter().zip(b).map(|(x, y)| (x - y) * (x - y)).sum();
|
||||
let den: f32 = a.iter().map(|x| x * x).sum::<f32>().max(1e-12);
|
||||
(num / den).sqrt()
|
||||
};
|
||||
let out = |e: &Experts, i: usize| -> Vec<f32> {
|
||||
e.forward_one(i, &xs)
|
||||
.unwrap()
|
||||
.flatten_all()
|
||||
.unwrap()
|
||||
.to_vec1()
|
||||
.unwrap()
|
||||
};
|
||||
|
||||
let dense0 = out(&banked, 0);
|
||||
// Baseline: a different expert on the same input. Any quantised
|
||||
// reading must beat this by a wide margin.
|
||||
let noise = rel_err(&dense0, &out(&banked, 1));
|
||||
|
||||
for (dtype, tol) in [(GgmlDType::Q8_0, 0.05f32), (GgmlDType::Q4K, 0.20f32)] {
|
||||
let q = Experts::quantize_banked(&gate_up, &down, inter, dtype).unwrap();
|
||||
assert_eq!(q.len(), n_experts);
|
||||
let err = rel_err(&dense0, &out(&q, 0));
|
||||
assert!(err < tol, "{dtype:?} relative error {err:.4} exceeds {tol}");
|
||||
assert!(
|
||||
err < noise / 2.0,
|
||||
"{dtype:?} error {err:.4} is not clearly better than an unrelated \
|
||||
expert ({noise:.4}) — the quantised path may not be reading \
|
||||
this expert at all"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -41,11 +41,14 @@ use crate::harness::arch::qwen3_5::linear_attn::GatedDeltaNet;
|
||||
use crate::harness::arch::qwen3_5::moe::Qwen3_5MoeBlock;
|
||||
use crate::harness::arch::qwen3_5::rope::RotaryEmbedding;
|
||||
|
||||
use candle_core::quantized::GgmlDType;
|
||||
|
||||
use super::config::TextConfig;
|
||||
use super::full_attn::Attention;
|
||||
use super::hyper::HyperConnection;
|
||||
use super::ple::PleBlock;
|
||||
use super::{linear_attn, moe};
|
||||
use crate::harness::arch::snapshot::LayerKvSnapshot;
|
||||
|
||||
/// The attention slot: one layer in four is full attention with the QSA
|
||||
/// indexer, the rest are the recurrent delta rule.
|
||||
@@ -71,6 +74,7 @@ impl DecoderLayer {
|
||||
cfg: &TextConfig,
|
||||
rotary: Arc<RotaryEmbedding>,
|
||||
layer_idx: usize,
|
||||
quant: Option<GgmlDType>,
|
||||
vb: &ShardedVarBuilder,
|
||||
) -> Result<Self> {
|
||||
let layer_type = cfg
|
||||
@@ -83,6 +87,30 @@ impl DecoderLayer {
|
||||
cfg.layer_types.len()
|
||||
)
|
||||
})?;
|
||||
Self::load_typed(
|
||||
cfg,
|
||||
rotary,
|
||||
layer_type,
|
||||
cfg.ple_layers().contains(&layer_idx),
|
||||
quant,
|
||||
vb,
|
||||
)
|
||||
}
|
||||
|
||||
/// Load a layer whose type the caller knows, rather than one looked
|
||||
/// up in `layer_types`.
|
||||
///
|
||||
/// The MTP head's layer (#313) needs this: it lives under
|
||||
/// `mtp.layers.0` with its own single-entry type list, and it
|
||||
/// carries no PLE wherever the main model puts one.
|
||||
pub fn load_typed(
|
||||
cfg: &TextConfig,
|
||||
rotary: Arc<RotaryEmbedding>,
|
||||
layer_type: &str,
|
||||
with_ple: bool,
|
||||
quant: Option<GgmlDType>,
|
||||
vb: &ShardedVarBuilder,
|
||||
) -> Result<Self> {
|
||||
let mixer = match layer_type {
|
||||
"full_attention" => {
|
||||
Mixer::Full(Box::new(Attention::load(cfg, rotary, &vb.pp("self_attn"))?))
|
||||
@@ -91,8 +119,8 @@ impl DecoderLayer {
|
||||
Mixer::Linear(Box::new(linear_attn::load(cfg, &vb.pp("linear_attn"))?))
|
||||
}
|
||||
other => anyhow::bail!(
|
||||
"unknown layer_type '{other}' for layer {layer_idx} (expected \
|
||||
'full_attention' or 'linear_attention')"
|
||||
"unknown layer_type '{other}' (expected 'full_attention' or \
|
||||
'linear_attention')"
|
||||
),
|
||||
};
|
||||
|
||||
@@ -109,7 +137,7 @@ impl DecoderLayer {
|
||||
)
|
||||
};
|
||||
|
||||
let ple = if cfg.ple_layers().contains(&layer_idx) {
|
||||
let ple = if with_ple {
|
||||
Some(PleBlock::load(
|
||||
&vb.pp("ple"),
|
||||
cfg.hidden_size,
|
||||
@@ -127,7 +155,7 @@ impl DecoderLayer {
|
||||
attn_hc: hc("attn_hyper_connection")?,
|
||||
mlp_hc: hc("mlp_hyper_connection")?,
|
||||
mixer,
|
||||
mlp: moe::load(cfg, &vb.pp("mlp"))?,
|
||||
mlp: moe::load(cfg, quant, &vb.pp("mlp"))?,
|
||||
ple,
|
||||
})
|
||||
}
|
||||
@@ -190,6 +218,65 @@ impl DecoderLayer {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Capture this layer's attention-side state.
|
||||
pub fn snapshot_kv(&self) -> candle_core::Result<LayerKvSnapshot> {
|
||||
Ok(match &self.mixer {
|
||||
Mixer::Full(attn) => {
|
||||
let (kv, indexer_keys) = attn.snapshot_kv();
|
||||
LayerKvSnapshot::FullSparse { kv, indexer_keys }
|
||||
}
|
||||
Mixer::Linear(net) => {
|
||||
let (conv_state, recurrent_state) = net.snapshot_state()?;
|
||||
LayerKvSnapshot::Linear {
|
||||
conv_state,
|
||||
recurrent_state,
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
/// Replace this layer's attention-side state. A variant mismatch
|
||||
/// means the snapshot came from a different model, or from the same
|
||||
/// model with a different `layer_types` — either way, restoring it
|
||||
/// would put a recurrent state into an attention layer.
|
||||
pub fn restore_kv(&mut self, snap: &LayerKvSnapshot) -> candle_core::Result<()> {
|
||||
match (&mut self.mixer, snap) {
|
||||
(Mixer::Full(attn), LayerKvSnapshot::FullSparse { kv, indexer_keys }) => {
|
||||
attn.restore_kv(kv.as_ref(), indexer_keys.as_ref())
|
||||
}
|
||||
(
|
||||
Mixer::Linear(net),
|
||||
LayerKvSnapshot::Linear {
|
||||
conv_state,
|
||||
recurrent_state,
|
||||
},
|
||||
) => net.restore_state(conv_state.as_ref(), recurrent_state.as_ref()),
|
||||
_ => candle_core::bail!(
|
||||
"restore_kv: snapshot layer kind does not match this layer's mixer kind"
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// The PLE block's rolling conv context, if this layer carries one.
|
||||
pub fn snapshot_ple(&self) -> candle_core::Result<Option<Tensor>> {
|
||||
match &self.ple {
|
||||
Some(block) => block.snapshot_state(),
|
||||
None => Ok(None),
|
||||
}
|
||||
}
|
||||
|
||||
/// Restore the PLE conv context. Refuses a snapshot that disagrees
|
||||
/// with this layer about whether PLE lives here at all.
|
||||
pub fn restore_ple(&mut self, conv_state: Option<&Tensor>) -> candle_core::Result<()> {
|
||||
match &mut self.ple {
|
||||
Some(block) => block.restore_state(conv_state),
|
||||
None if conv_state.is_none() => Ok(()),
|
||||
None => candle_core::bail!(
|
||||
"restore_ple: snapshot carries PLE state but this layer has no PLE block"
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn has_ple(&self) -> bool {
|
||||
self.ple.is_some()
|
||||
}
|
||||
|
||||
@@ -170,6 +170,34 @@ impl Attention {
|
||||
Ok(self.o_proj.forward(&gated)?)
|
||||
}
|
||||
|
||||
/// Capture both caches at one token boundary.
|
||||
///
|
||||
/// The same reason `clear_kv_cache` drops them together: a
|
||||
/// snapshot holding only the attention K/V restores a layer whose
|
||||
/// indexer still holds the previous request's keys, and the
|
||||
/// selection it computes from them is wrong without being invalid.
|
||||
pub fn snapshot_kv(&self) -> (Option<(Tensor, Tensor)>, Option<Tensor>) {
|
||||
let kv = match (self.kv_cache.k(), self.kv_cache.v()) {
|
||||
(Some(k), Some(v)) => Some((k.clone(), v.clone())),
|
||||
_ => None,
|
||||
};
|
||||
(kv, self.indexer.snapshot_keys())
|
||||
}
|
||||
|
||||
/// Replace both caches from a snapshot.
|
||||
pub fn restore_kv(
|
||||
&mut self,
|
||||
kv: Option<&(Tensor, Tensor)>,
|
||||
indexer_keys: Option<&Tensor>,
|
||||
) -> candle_core::Result<()> {
|
||||
self.kv_cache.reset();
|
||||
if let Some((k, v)) = kv {
|
||||
self.kv_cache.append(k, v)?;
|
||||
}
|
||||
self.indexer.restore_keys(indexer_keys);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Both caches, together. Dropping only one leaves the indexer
|
||||
/// selecting blocks of a previous request's text.
|
||||
pub fn clear_kv_cache(&mut self) {
|
||||
|
||||
@@ -25,6 +25,8 @@
|
||||
//! from `qwen3_5` with two settings corrected.
|
||||
//! - [`model`] — embeddings in, logits out; four streams, one
|
||||
//! n-gram lookup, and no final norm.
|
||||
//! - [`mtp`] — the shipped draft head: one more decoder layer, fed
|
||||
//! the target's pre-mixer stream and the next token's embedding.
|
||||
//! - [`moe`] — the FFN every layer has, loaded from fused expert
|
||||
//! tensors instead of 512 modules.
|
||||
//! - [`ple`] — the hashed n-gram table on layer 1: how its rows
|
||||
@@ -39,5 +41,6 @@ pub mod hyper;
|
||||
pub mod linear_attn;
|
||||
pub mod model;
|
||||
pub mod moe;
|
||||
pub mod mtp;
|
||||
pub mod ple;
|
||||
pub mod qsa;
|
||||
|
||||
@@ -34,6 +34,7 @@
|
||||
//! See `doc/qwen4_exp-port-spec.md`.
|
||||
|
||||
use anyhow::{Context, Result, ensure};
|
||||
use candle_core::quantized::GgmlDType;
|
||||
use candle_core::{DType, Device, IndexOp, Module, Tensor};
|
||||
use candle_nn::var_builder::ShardedVarBuilder;
|
||||
use candle_nn::{Embedding, Linear};
|
||||
@@ -45,6 +46,7 @@ use super::config::{Config, TextConfig};
|
||||
use super::decoder::DecoderLayer;
|
||||
use super::hyper::HyperConnection;
|
||||
use super::ple::{NGramHasher, NGramTable, ShardedNGramTable};
|
||||
use crate::harness::arch::snapshot::{KvCacheSnapshot, PleSnapshot};
|
||||
|
||||
/// The hashed n-gram lookup, hoisted to the model because it is a
|
||||
/// function of the input ids alone.
|
||||
@@ -89,6 +91,18 @@ impl NGramEmbedding {
|
||||
Ok(stacked)
|
||||
}
|
||||
|
||||
/// The carried ids, for a prefix snapshot. Two integers per batch
|
||||
/// row — small, but they decide which n-grams the next token
|
||||
/// hashes, so a restore that forgets them silently addresses the
|
||||
/// wrong rows of a 320-million-row table.
|
||||
fn snapshot(&self) -> Vec<Vec<i64>> {
|
||||
self.carried.clone()
|
||||
}
|
||||
|
||||
fn restore(&mut self, carried: &[Vec<i64>]) {
|
||||
self.carried = carried.to_vec();
|
||||
}
|
||||
|
||||
fn reset(&mut self) {
|
||||
for row in &mut self.carried {
|
||||
row.clear();
|
||||
@@ -111,9 +125,14 @@ pub struct Qwen4ExpForCausalLM {
|
||||
}
|
||||
|
||||
impl Qwen4ExpForCausalLM {
|
||||
/// `quant` quantises the routed experts in situ (#315). `None`
|
||||
/// keeps the checkpoint's precision, which for this architecture is
|
||||
/// useful only on a model small enough to hold: 241.6 GB of BF16
|
||||
/// experts fit neither 64 GB of VRAM nor 123 GB of host RAM.
|
||||
pub fn load(
|
||||
cfg: &Config,
|
||||
dtype: DType,
|
||||
quant: Option<GgmlDType>,
|
||||
device: &Device,
|
||||
vb: &ShardedVarBuilder,
|
||||
) -> Result<Self> {
|
||||
@@ -134,7 +153,7 @@ impl Qwen4ExpForCausalLM {
|
||||
let mut layers = Vec::with_capacity(text.num_hidden_layers);
|
||||
for i in 0..text.num_hidden_layers {
|
||||
layers.push(
|
||||
DecoderLayer::load(text, rotary.clone(), i, &layers_vb.pp(i))
|
||||
DecoderLayer::load(text, rotary.clone(), i, quant, &layers_vb.pp(i))
|
||||
.with_context(|| format!("load decoder layer {i}"))?,
|
||||
);
|
||||
}
|
||||
@@ -177,6 +196,24 @@ impl Qwen4ExpForCausalLM {
|
||||
/// `input_ids` is `(B, T)`; `offset` is the sequence position the
|
||||
/// first of them sits at. Returns logits `(B, T, vocab)`.
|
||||
pub fn forward(&mut self, input_ids: &Tensor, offset: usize) -> Result<Tensor> {
|
||||
Ok(self.forward_inner(input_ids, offset)?.0)
|
||||
}
|
||||
|
||||
/// Logits, plus the four-stream residual as it stood *before* the
|
||||
/// mixer collapsed it.
|
||||
///
|
||||
/// The MTP draft head consumes that pre-mixer stream, not the 2560
|
||||
/// the LM head sees (#313), so it has to leave the model here rather
|
||||
/// than be reconstructed from the logits — which it could not be.
|
||||
pub fn forward_with_hidden(
|
||||
&mut self,
|
||||
input_ids: &Tensor,
|
||||
offset: usize,
|
||||
) -> Result<(Tensor, Tensor)> {
|
||||
self.forward_inner(input_ids, offset)
|
||||
}
|
||||
|
||||
fn forward_inner(&mut self, input_ids: &Tensor, offset: usize) -> Result<(Tensor, Tensor)> {
|
||||
let (batch, seq_len) = input_ids.dims2()?;
|
||||
|
||||
// Four identical streams — the residual this architecture
|
||||
@@ -209,7 +246,92 @@ impl Qwen4ExpForCausalLM {
|
||||
|
||||
// The mixer's hc_norm is the only normalisation before the head.
|
||||
let x = self.mixer.collapse(&h)?;
|
||||
Ok(self.lm_head.forward(&x)?)
|
||||
Ok((self.lm_head.forward(&x)?, h))
|
||||
}
|
||||
|
||||
/// Capture every piece of per-request state at one token boundary:
|
||||
/// the attention K/V and the QSA indexer cache per full-attention
|
||||
/// layer, the GatedDeltaNet conv and recurrent states per linear
|
||||
/// layer, and PLE's conv context and carried ids.
|
||||
///
|
||||
/// There is no `rope_delta` here — this architecture's text path
|
||||
/// takes positions from the caller's offset rather than tracking a
|
||||
/// vision-induced skew, so the field is recorded as zero. When the
|
||||
/// vision tower lands (#314) that stops being true.
|
||||
pub fn snapshot_kv_cache(&self) -> candle_core::Result<KvCacheSnapshot> {
|
||||
let layers = self
|
||||
.layers
|
||||
.iter()
|
||||
.map(|l| l.snapshot_kv())
|
||||
.collect::<candle_core::Result<Vec<_>>>()?;
|
||||
let conv_state = self
|
||||
.layers
|
||||
.iter()
|
||||
.map(|l| l.snapshot_ple())
|
||||
.collect::<candle_core::Result<Vec<_>>>()?
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.next();
|
||||
Ok(KvCacheSnapshot {
|
||||
layers,
|
||||
rope_delta: 0,
|
||||
ple: self.ngram.as_ref().map(|n| PleSnapshot {
|
||||
conv_state,
|
||||
carried_ids: n.snapshot(),
|
||||
}),
|
||||
})
|
||||
}
|
||||
|
||||
/// Replace the live state from a snapshot. The snapshot stays valid
|
||||
/// for further restores.
|
||||
pub fn restore_kv_cache(&mut self, snap: &KvCacheSnapshot) -> candle_core::Result<()> {
|
||||
if snap.layers.len() != self.layers.len() {
|
||||
candle_core::bail!(
|
||||
"restore_kv_cache: snapshot has {} layers, model has {}",
|
||||
snap.layers.len(),
|
||||
self.layers.len()
|
||||
);
|
||||
}
|
||||
// A snapshot from a model whose PLE layer sits elsewhere would
|
||||
// otherwise restore the conv context onto the wrong layer and
|
||||
// leave this one holding the previous request's.
|
||||
if snap.ple.is_some() != self.ngram.is_some() {
|
||||
candle_core::bail!(
|
||||
"restore_kv_cache: snapshot {} PLE state, model {} a PLE layer",
|
||||
if snap.ple.is_some() {
|
||||
"carries"
|
||||
} else {
|
||||
"has no"
|
||||
},
|
||||
if self.ngram.is_some() {
|
||||
"has"
|
||||
} else {
|
||||
"has none"
|
||||
}
|
||||
);
|
||||
}
|
||||
for (layer, layer_snap) in self.layers.iter_mut().zip(snap.layers.iter()) {
|
||||
layer.restore_kv(layer_snap)?;
|
||||
}
|
||||
let conv = snap.ple.as_ref().and_then(|p| p.conv_state.as_ref());
|
||||
for layer in self.layers.iter_mut() {
|
||||
if layer.has_ple() {
|
||||
layer.restore_ple(conv)?;
|
||||
}
|
||||
}
|
||||
if let (Some(ngram), Some(ple)) = (&mut self.ngram, snap.ple.as_ref()) {
|
||||
ngram.restore(&ple.carried_ids);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// `embed_tokens` applied to `input_ids`.
|
||||
///
|
||||
/// The MTP head reuses this embedding
|
||||
/// (`mtp_use_dedicated_embeddings: false`) but does not own it, so
|
||||
/// the speculative loop borrows it from the target model.
|
||||
pub fn embed(&self, input_ids: &Tensor) -> Result<Tensor> {
|
||||
Ok(self.embed_tokens.forward(input_ids)?)
|
||||
}
|
||||
|
||||
pub fn clear_kv_cache(&mut self) -> Result<()> {
|
||||
@@ -320,6 +442,7 @@ fn rotary_for(cfg: &TextConfig, dtype: DType, device: &Device) -> Result<RotaryE
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::harness::arch::snapshot::LayerKvSnapshot;
|
||||
use candle_nn::var_builder::ShardedSafeTensors;
|
||||
use std::collections::HashMap;
|
||||
|
||||
@@ -535,6 +658,95 @@ mod tests {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- the MTP draft head ----
|
||||
// A second full-attention layer plus the two fc_* projections
|
||||
// and their pre-norms. Emitted unconditionally: the head is what
|
||||
// #313 builds on and the fixture is the only place its loader
|
||||
// gets exercised.
|
||||
{
|
||||
let m = "mtp";
|
||||
t.insert(format!("{m}.pre_fc_norm_embedding.weight"), rand(vec![h]));
|
||||
t.insert(format!("{m}.pre_fc_norm_hidden.weight"), rand(vec![wide]));
|
||||
t.insert(format!("{m}.fc_embedding.weight"), rand(vec![h, h]));
|
||||
t.insert(format!("{m}.fc_hidden.weight"), rand(vec![h, h]));
|
||||
t.insert(
|
||||
format!("{m}.hyper_connection_mixer.hc_norm.weight"),
|
||||
rand(vec![wide]),
|
||||
);
|
||||
t.insert(
|
||||
format!("{m}.hyper_connection_mixer.input_mix_weight_down.weight"),
|
||||
rand(vec![cfg.hc_lowrank, wide]),
|
||||
);
|
||||
t.insert(
|
||||
format!("{m}.hyper_connection_mixer.input_mix_weight_up.weight"),
|
||||
rand(vec![wide, cfg.hc_lowrank]),
|
||||
);
|
||||
let l = format!("{m}.layers.0");
|
||||
for which in ["attn_hyper_connection", "mlp_hyper_connection"] {
|
||||
t.insert(format!("{l}.{which}.hc_norm.weight"), rand(vec![wide]));
|
||||
t.insert(
|
||||
format!("{l}.{which}.input_mix_weight_down.weight"),
|
||||
rand(vec![cfg.hc_lowrank, wide]),
|
||||
);
|
||||
t.insert(
|
||||
format!("{l}.{which}.input_mix_weight_up.weight"),
|
||||
rand(vec![wide, cfg.hc_lowrank]),
|
||||
);
|
||||
t.insert(
|
||||
format!("{l}.{which}.block_inject_weight.weight"),
|
||||
rand(vec![hc, wide]),
|
||||
);
|
||||
}
|
||||
let a = format!("{l}.self_attn");
|
||||
let heads = cfg.num_attention_heads * cfg.head_dim;
|
||||
t.insert(format!("{a}.q_proj.weight"), rand(vec![heads * 2, h]));
|
||||
t.insert(
|
||||
format!("{a}.k_proj.weight"),
|
||||
rand(vec![cfg.num_key_value_heads * cfg.head_dim, h]),
|
||||
);
|
||||
t.insert(
|
||||
format!("{a}.v_proj.weight"),
|
||||
rand(vec![cfg.num_key_value_heads * cfg.head_dim, h]),
|
||||
);
|
||||
t.insert(format!("{a}.o_proj.weight"), rand(vec![h, heads]));
|
||||
t.insert(format!("{a}.q_norm.weight"), rand(vec![cfg.head_dim]));
|
||||
t.insert(format!("{a}.k_norm.weight"), rand(vec![cfg.head_dim]));
|
||||
let fused = (cfg.indexer_n_heads + cfg.indexer_kv_heads) * cfg.indexer_head_dim;
|
||||
t.insert(
|
||||
format!("{a}.indexer.index_qk_proj.weight"),
|
||||
rand(vec![fused, h]),
|
||||
);
|
||||
t.insert(
|
||||
format!("{a}.indexer.q_layernorm.weight"),
|
||||
rand(vec![cfg.indexer_head_dim]),
|
||||
);
|
||||
t.insert(
|
||||
format!("{a}.indexer.k_layernorm.weight"),
|
||||
rand(vec![cfg.indexer_head_dim]),
|
||||
);
|
||||
let mm = format!("{l}.mlp");
|
||||
t.insert(format!("{mm}.gate.weight"), rand(vec![cfg.num_experts, h]));
|
||||
t.insert(
|
||||
format!("{mm}.experts.gate_up_proj"),
|
||||
rand(vec![cfg.num_experts, cfg.moe_intermediate_size * 2, h]),
|
||||
);
|
||||
t.insert(
|
||||
format!("{mm}.experts.down_proj"),
|
||||
rand(vec![cfg.num_experts, h, cfg.moe_intermediate_size]),
|
||||
);
|
||||
for pj in ["gate_proj", "up_proj"] {
|
||||
t.insert(
|
||||
format!("{mm}.shared_expert.{pj}.weight"),
|
||||
rand(vec![cfg.shared_expert_intermediate_size, h]),
|
||||
);
|
||||
}
|
||||
t.insert(
|
||||
format!("{mm}.shared_expert.down_proj.weight"),
|
||||
rand(vec![h, cfg.shared_expert_intermediate_size]),
|
||||
);
|
||||
t.insert(format!("{mm}.shared_expert_gate.weight"), rand(vec![1, h]));
|
||||
}
|
||||
t
|
||||
}
|
||||
|
||||
@@ -545,7 +757,7 @@ mod tests {
|
||||
candle_core::safetensors::save(&tensors(&cfg.text_config), &path).unwrap();
|
||||
let vb =
|
||||
unsafe { ShardedSafeTensors::var_builder(&[&path], DType::F32, &Device::Cpu).unwrap() };
|
||||
let model = Qwen4ExpForCausalLM::load(&cfg, DType::F32, &Device::Cpu, &vb).unwrap();
|
||||
let model = Qwen4ExpForCausalLM::load(&cfg, DType::F32, None, &Device::Cpu, &vb).unwrap();
|
||||
(dir, model)
|
||||
}
|
||||
|
||||
@@ -603,4 +815,279 @@ mod tests {
|
||||
assert!(ngram.table.gather(&[(TABLE_ROWS - 1) as i64]).is_ok());
|
||||
assert!(ngram.table.gather(&[TABLE_ROWS as i64]).is_err());
|
||||
}
|
||||
|
||||
fn tensor_vec(t: &Tensor) -> Vec<f32> {
|
||||
t.flatten_all()
|
||||
.unwrap()
|
||||
.to_dtype(DType::F32)
|
||||
.unwrap()
|
||||
.to_vec1()
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
fn logits_vec(t: &Tensor) -> Vec<f32> {
|
||||
t.flatten_all().unwrap().to_vec1().unwrap()
|
||||
}
|
||||
|
||||
/// The property prefix caching rests on: a snapshot must *rewind*
|
||||
/// the live state, not merely coexist with it.
|
||||
///
|
||||
/// So the state is deliberately moved past the snapshot before the
|
||||
/// restore — two further decode steps — and the restored step is
|
||||
/// then required to reproduce the original exactly. An
|
||||
/// implementation that captured nothing would not reproduce it; one
|
||||
/// that restored only the attention K/V and left the indexer cache
|
||||
/// at seven positions would trip the divergence guard rather than
|
||||
/// return wrong numbers.
|
||||
///
|
||||
/// The control is that a *cleared* model cannot take this step at
|
||||
/// all: with an empty cache and a query at position 4 the guard
|
||||
/// fires. That is what distinguishes restore from clear here, and
|
||||
/// it is asserted rather than assumed.
|
||||
#[test]
|
||||
fn a_restored_snapshot_rewinds_rather_than_merely_continuing() {
|
||||
let (_dir, mut model) = build();
|
||||
let prompt = Tensor::from_vec(vec![5u32, 9, 3, 12], (1, 4), &Device::Cpu).unwrap();
|
||||
let next = Tensor::from_vec(vec![7u32], (1, 1), &Device::Cpu).unwrap();
|
||||
|
||||
model.forward(&prompt, 0).unwrap();
|
||||
let snap = model.snapshot_kv_cache().unwrap();
|
||||
let want = logits_vec(&model.forward(&next, 4).unwrap());
|
||||
|
||||
// Move the live state well past the snapshot.
|
||||
model.forward(&next, 5).unwrap();
|
||||
model.forward(&next, 6).unwrap();
|
||||
|
||||
model.restore_kv_cache(&snap).unwrap();
|
||||
let got = logits_vec(&model.forward(&next, 4).unwrap());
|
||||
assert_eq!(got, want, "restore must rewind to the snapshot's boundary");
|
||||
|
||||
// Control: a cleared model cannot even take the step, so the
|
||||
// restore above did something a clear does not.
|
||||
model.clear_kv_cache().unwrap();
|
||||
// `{:#}` to walk the context chain: the forward wraps the
|
||||
// failure in "layer 1", and the guard is what it wraps.
|
||||
let err = format!("{:#}", model.forward(&next, 4).unwrap_err());
|
||||
assert!(
|
||||
err.contains("diverged"),
|
||||
"expected the cache-divergence guard, got: {err}"
|
||||
);
|
||||
}
|
||||
|
||||
/// The snapshot carries PLE's state, and PLE is the piece with two
|
||||
/// distinct kinds of it — a rolling conv context and the carried
|
||||
/// n-gram ids. Restoring one without the other reads the wrong rows
|
||||
/// of the table, so the snapshot has to hold both.
|
||||
#[test]
|
||||
fn the_snapshot_carries_ple_state() {
|
||||
let (_dir, mut model) = build();
|
||||
let prompt = Tensor::from_vec(vec![5u32, 9, 3, 12], (1, 4), &Device::Cpu).unwrap();
|
||||
model.forward(&prompt, 0).unwrap();
|
||||
|
||||
let snap = model.snapshot_kv_cache().unwrap();
|
||||
let ple = snap.ple.as_ref().expect("layer 1 carries PLE");
|
||||
assert!(ple.conv_state.is_some(), "the dilated conv context");
|
||||
assert_eq!(ple.carried_ids.len(), 1, "one batch row");
|
||||
assert_eq!(
|
||||
ple.carried_ids[0].len(),
|
||||
2,
|
||||
"ngram_size - 1 ids carried across the step"
|
||||
);
|
||||
assert!(snap.size_bytes() > 0);
|
||||
}
|
||||
|
||||
/// Round-trip the snapshot itself, not the logits it leads to.
|
||||
///
|
||||
/// Taken from llama.cpp's `test-save-load-state` for this
|
||||
/// architecture (their #27941, which fixed the indexer cache
|
||||
/// dropping `ext.x`/`ext.y` on restore): comparing *generated
|
||||
/// output* cannot see a field dropped on the way back in, because a
|
||||
/// dropped field need not change the next token. Comparing the
|
||||
/// captured state can. Their fix moved 198 bytes of 335,692 — a
|
||||
/// logits comparison would never have noticed.
|
||||
///
|
||||
/// So: snapshot, restore, snapshot again, and require the two to
|
||||
/// agree field for field. Anything `restore_kv_cache` silently
|
||||
/// fails to put back shows up here as a shape or value difference.
|
||||
#[test]
|
||||
fn a_snapshot_survives_its_own_round_trip() {
|
||||
let (_dir, mut model) = build();
|
||||
let prompt = Tensor::from_vec(vec![5u32, 9, 3, 12, 7], (1, 5), &Device::Cpu).unwrap();
|
||||
model.forward(&prompt, 0).unwrap();
|
||||
|
||||
let first = model.snapshot_kv_cache().unwrap();
|
||||
model.restore_kv_cache(&first).unwrap();
|
||||
let second = model.snapshot_kv_cache().unwrap();
|
||||
|
||||
assert_eq!(first.layer_count(), second.layer_count());
|
||||
assert_eq!(
|
||||
first.size_bytes(),
|
||||
second.size_bytes(),
|
||||
"total captured bytes"
|
||||
);
|
||||
|
||||
for (i, (a, b)) in first.layers.iter().zip(second.layers.iter()).enumerate() {
|
||||
match (a, b) {
|
||||
(
|
||||
LayerKvSnapshot::FullSparse {
|
||||
kv: ka,
|
||||
indexer_keys: ia,
|
||||
},
|
||||
LayerKvSnapshot::FullSparse {
|
||||
kv: kb,
|
||||
indexer_keys: ib,
|
||||
},
|
||||
) => {
|
||||
assert_eq!(ka.is_some(), kb.is_some(), "layer {i} kv presence");
|
||||
if let (Some((k1, v1)), Some((k2, v2))) = (ka, kb) {
|
||||
assert_eq!(k1.dims(), k2.dims(), "layer {i} k");
|
||||
assert_eq!(v1.dims(), v2.dims(), "layer {i} v");
|
||||
assert_eq!(tensor_vec(k1), tensor_vec(k2), "layer {i} k values");
|
||||
}
|
||||
// The field their fix was about: dropped silently,
|
||||
// and invisible to anything that looks at outputs.
|
||||
assert_eq!(
|
||||
ia.is_some(),
|
||||
ib.is_some(),
|
||||
"layer {i} indexer cache presence"
|
||||
);
|
||||
if let (Some(x), Some(y)) = (ia, ib) {
|
||||
assert_eq!(x.dims(), y.dims(), "layer {i} indexer keys");
|
||||
assert_eq!(tensor_vec(x), tensor_vec(y), "layer {i} indexer values");
|
||||
}
|
||||
}
|
||||
(
|
||||
LayerKvSnapshot::Linear {
|
||||
conv_state: ca,
|
||||
recurrent_state: ra,
|
||||
},
|
||||
LayerKvSnapshot::Linear {
|
||||
conv_state: cb,
|
||||
recurrent_state: rb,
|
||||
},
|
||||
) => {
|
||||
assert_eq!(ca.is_some(), cb.is_some(), "layer {i} conv presence");
|
||||
assert_eq!(ra.is_some(), rb.is_some(), "layer {i} recurrent presence");
|
||||
if let (Some(x), Some(y)) = (ca, cb) {
|
||||
assert_eq!(tensor_vec(x), tensor_vec(y), "layer {i} conv values");
|
||||
}
|
||||
if let (Some(x), Some(y)) = (ra, rb) {
|
||||
assert_eq!(tensor_vec(x), tensor_vec(y), "layer {i} recurrent values");
|
||||
}
|
||||
}
|
||||
_ => panic!("layer {i}: snapshot variant changed across a round trip"),
|
||||
}
|
||||
}
|
||||
|
||||
let (pa, pb) = (first.ple.as_ref().unwrap(), second.ple.as_ref().unwrap());
|
||||
assert_eq!(pa.carried_ids, pb.carried_ids, "PLE carried ids");
|
||||
assert_eq!(pa.conv_state.is_some(), pb.conv_state.is_some());
|
||||
if let (Some(x), Some(y)) = (&pa.conv_state, &pb.conv_state) {
|
||||
assert_eq!(tensor_vec(x), tensor_vec(y), "PLE conv context");
|
||||
}
|
||||
}
|
||||
|
||||
/// A snapshot from a differently-shaped model must be refused
|
||||
/// rather than restored onto whatever lines up.
|
||||
#[test]
|
||||
fn a_mismatched_snapshot_is_refused() {
|
||||
let (_dir, mut model) = build();
|
||||
let prompt = Tensor::from_vec(vec![5u32, 9], (1, 2), &Device::Cpu).unwrap();
|
||||
model.forward(&prompt, 0).unwrap();
|
||||
let mut snap = model.snapshot_kv_cache().unwrap();
|
||||
|
||||
snap.layers.pop();
|
||||
let err = model.restore_kv_cache(&snap).unwrap_err().to_string();
|
||||
assert!(err.contains("layers"), "got: {err}");
|
||||
|
||||
let mut snap = model.snapshot_kv_cache().unwrap();
|
||||
snap.ple = None;
|
||||
let err = model.restore_kv_cache(&snap).unwrap_err().to_string();
|
||||
assert!(err.contains("PLE"), "got: {err}");
|
||||
}
|
||||
|
||||
/// The draft head, end to end on the fixture: the target's
|
||||
/// pre-mixer stream and the next token's embedding go in, and both
|
||||
/// of the head's outputs come out — the collapse for `lm_head` and
|
||||
/// the multi stream that a second draft step would consume.
|
||||
#[test]
|
||||
fn the_mtp_head_drafts_from_the_targets_pre_mixer_stream() {
|
||||
use crate::harness::arch::qwen4_exp::mtp::MtpHead;
|
||||
let (_dir, mut model) = build();
|
||||
let cfg = Config::from_config_json(TINY).unwrap();
|
||||
let dir2 = tempfile::tempdir().unwrap();
|
||||
let path = dir2.path().join("model.safetensors");
|
||||
candle_core::safetensors::save(&tensors(&cfg.text_config), &path).unwrap();
|
||||
let vb =
|
||||
unsafe { ShardedSafeTensors::var_builder(&[&path], DType::F32, &Device::Cpu).unwrap() };
|
||||
let rotary = Arc::new(rotary_for(&cfg.text_config, DType::F32, &Device::Cpu).unwrap());
|
||||
let mut head = MtpHead::load(&cfg.text_config, rotary.clone(), None, &vb).unwrap();
|
||||
|
||||
let prompt = Tensor::from_vec(vec![5u32, 9, 3, 12], (1, 4), &Device::Cpu).unwrap();
|
||||
let (_logits, hidden) = model.forward_with_hidden(&prompt, 0).unwrap();
|
||||
let wide = cfg.text_config.stream_width();
|
||||
assert_eq!(hidden.dims(), &[1, 4, wide], "the pre-mixer stream");
|
||||
|
||||
// The head owns its own KV and indexer caches, so it has to walk
|
||||
// the prompt alongside the target before it can draft from
|
||||
// position 4 — a draft head that only ever sees the newest token
|
||||
// is decoding against an empty past.
|
||||
let ids: Vec<u32> = prompt.flatten_all().unwrap().to_vec1().unwrap();
|
||||
for (i, id) in ids.iter().enumerate() {
|
||||
let tok = Tensor::from_vec(vec![*id], (1, 1), &Device::Cpu).unwrap();
|
||||
let e = model.embed(&tok).unwrap();
|
||||
let h_i = hidden.narrow(1, i, 1).unwrap().contiguous().unwrap();
|
||||
let (cos, sin) = rotary.plain_cos_sin(i, 1).unwrap();
|
||||
head.forward(&e, &h_i, None, &cos, &sin, i).unwrap();
|
||||
}
|
||||
|
||||
let next = Tensor::from_vec(vec![7u32], (1, 1), &Device::Cpu).unwrap();
|
||||
let e = model.embed(&next).unwrap();
|
||||
let last = hidden.narrow(1, 3, 1).unwrap().contiguous().unwrap();
|
||||
let (cos, sin) = rotary.plain_cos_sin(4, 1).unwrap();
|
||||
|
||||
let step = head.forward(&e, &last, None, &cos, &sin, 4).unwrap();
|
||||
assert_eq!(
|
||||
step.sample_hidden.dims(),
|
||||
&[1, 1, cfg.text_config.hidden_size]
|
||||
);
|
||||
assert_eq!(step.multi_stream.dims(), &[1, 1, wide]);
|
||||
let v: Vec<f32> = step.sample_hidden.flatten_all().unwrap().to_vec1().unwrap();
|
||||
assert!(v.iter().all(|x| x.is_finite()));
|
||||
|
||||
// Chained: the second step reads the head's own multi stream,
|
||||
// not the target's and not the collapse.
|
||||
let (cos, sin) = rotary.plain_cos_sin(5, 1).unwrap();
|
||||
let step2 = head
|
||||
.forward(&e, &step.multi_stream, None, &cos, &sin, 5)
|
||||
.unwrap();
|
||||
assert_eq!(step2.multi_stream.dims(), &[1, 1, wide]);
|
||||
}
|
||||
|
||||
/// Handing the head the collapsed hidden state instead of the
|
||||
/// pre-mixer stream is the mistake the shapes invite — both are
|
||||
/// "the hidden state" and only one is right. It is refused by width
|
||||
/// rather than silently broadcast.
|
||||
#[test]
|
||||
fn the_head_refuses_the_collapsed_hidden_state() {
|
||||
use crate::harness::arch::qwen4_exp::mtp::MtpHead;
|
||||
let cfg = Config::from_config_json(TINY).unwrap();
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("model.safetensors");
|
||||
candle_core::safetensors::save(&tensors(&cfg.text_config), &path).unwrap();
|
||||
let vb =
|
||||
unsafe { ShardedSafeTensors::var_builder(&[&path], DType::F32, &Device::Cpu).unwrap() };
|
||||
let rotary = Arc::new(rotary_for(&cfg.text_config, DType::F32, &Device::Cpu).unwrap());
|
||||
let mut head = MtpHead::load(&cfg.text_config, rotary.clone(), None, &vb).unwrap();
|
||||
|
||||
let h = cfg.text_config.hidden_size;
|
||||
let collapsed = Tensor::zeros((1, 1, h), DType::F32, &Device::Cpu).unwrap();
|
||||
let e = Tensor::zeros((1, 1, h), DType::F32, &Device::Cpu).unwrap();
|
||||
let (cos, sin) = rotary.plain_cos_sin(0, 1).unwrap();
|
||||
let err = head
|
||||
.forward(&e, &collapsed, None, &cos, &sin, 0)
|
||||
.unwrap_err()
|
||||
.to_string();
|
||||
assert!(err.contains("pre-final-mixer"), "got: {err}");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -22,26 +22,45 @@
|
||||
//! of error that produces a model that talks and reasons slightly
|
||||
//! worse. It is asserted below rather than commented.
|
||||
//!
|
||||
//! The fused layout is the better one for a grouped GEMM, and keeping
|
||||
//! it would be the faster path; that is an algorithm change, and the
|
||||
//! spec is explicit that this stage is not it. Sparsity is extreme —
|
||||
//! 10 of 512 with an intermediate of 640 — so the routing, not the
|
||||
//! GEMM shape, is what dominates today.
|
||||
//! ## Why the fused tensors stay fused
|
||||
//!
|
||||
//! Slicing them into 512 per-expert modules copies **241.6 GB** at
|
||||
//! BF16, 5.03 GB per layer. That fits in neither beast's 64 GB of VRAM
|
||||
//! nor its 123 GB of host RAM, and no quantisation rescues it: #309
|
||||
//! measured 59.4 GB of 63.7 used at 4.25 bpw with only *four* layers'
|
||||
//! experts offloaded. A model that cannot be held cannot be loaded.
|
||||
//!
|
||||
//! So the block takes `Experts::Banked` and views one expert per
|
||||
//! routed token instead. A slice along dim 0 of a contiguous tensor is
|
||||
//! contiguous, and `x.matmul(w.t())` is what `Linear` does anyway, so
|
||||
//! the views feed the same GEMMs with no copy. It is also the layout a
|
||||
//! host-resident bank with a device-side gather wants (#318).
|
||||
//!
|
||||
//! See `doc/qwen4_exp-port-spec.md` §6.
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use candle_core::{IndexOp, Tensor};
|
||||
use candle_core::Tensor;
|
||||
use candle_core::quantized::GgmlDType;
|
||||
use candle_nn::Linear;
|
||||
use candle_nn::var_builder::ShardedVarBuilder;
|
||||
|
||||
use crate::harness::arch::qwen3_5::mlp::Qwen3_5MLP;
|
||||
use crate::harness::arch::qwen3_5::moe::Qwen3_5MoeBlock;
|
||||
use crate::harness::arch::qwen3_5::moe::{Experts, Qwen3_5MoeBlock};
|
||||
|
||||
use super::config::TextConfig;
|
||||
|
||||
/// `vb` should be `.pp(...)`-ed to the layer's `mlp` prefix.
|
||||
pub fn load(cfg: &TextConfig, vb: &ShardedVarBuilder) -> Result<Qwen3_5MoeBlock> {
|
||||
///
|
||||
/// `quant` quantises the routed experts in situ (#315). Without it the
|
||||
/// experts stay banked at the checkpoint's precision, which is only
|
||||
/// useful for a model small enough to hold — 120.8 B routed parameters
|
||||
/// are 241.6 GB at BF16 and fit nowhere on this hardware, so a real
|
||||
/// load of `Qwen3.8-Flash-Next` always passes one.
|
||||
pub fn load(
|
||||
cfg: &TextConfig,
|
||||
quant: Option<GgmlDType>,
|
||||
vb: &ShardedVarBuilder,
|
||||
) -> Result<Qwen3_5MoeBlock> {
|
||||
let (h, inter) = (cfg.hidden_size, cfg.moe_intermediate_size);
|
||||
anyhow::ensure!(
|
||||
cfg.num_experts > 0 && cfg.num_experts_per_tok > 0 && inter > 0,
|
||||
@@ -65,7 +84,19 @@ pub fn load(cfg: &TextConfig, vb: &ShardedVarBuilder) -> Result<Qwen3_5MoeBlock>
|
||||
let down = experts_vb
|
||||
.get((cfg.num_experts, h, inter), "down_proj")
|
||||
.with_context(|| format!("load '{}/down_proj'", experts_vb.prefix()))?;
|
||||
let experts = split_fused_experts(&gate_up, &down, inter)?;
|
||||
check_fused_experts(&gate_up, &down, inter)?;
|
||||
// The fused pair is the peak: 5.03 GB per layer at BF16. Quantising
|
||||
// here and letting it drop at the end of this function is what keeps
|
||||
// the whole 241.6 GB from ever existing.
|
||||
let experts = match quant {
|
||||
Some(dtype) => Experts::quantize_banked(&gate_up, &down, inter, dtype)
|
||||
.with_context(|| format!("quantize experts to {dtype:?}"))?,
|
||||
None => Experts::Banked {
|
||||
gate_up,
|
||||
down,
|
||||
intermediate: inter,
|
||||
},
|
||||
};
|
||||
|
||||
let (shared_expert, shared_expert_gate) = if cfg.shared_expert_intermediate_size > 0 {
|
||||
let shared = Qwen3_5MLP::load_with_dims(
|
||||
@@ -93,14 +124,17 @@ pub fn load(cfg: &TextConfig, vb: &ShardedVarBuilder) -> Result<Qwen3_5MoeBlock>
|
||||
))
|
||||
}
|
||||
|
||||
/// Slice `(E, 2I, H)` and `(E, H, I)` into `E` SwiGLU experts.
|
||||
/// Validate the fused pair before it is banked.
|
||||
///
|
||||
/// Gate first, up second — see the module note.
|
||||
pub(crate) fn split_fused_experts(
|
||||
/// The block will take views of these per routed expert, so a
|
||||
/// disagreement here surfaces as a shape error mid-forward on whichever
|
||||
/// token first routes to the offending expert — which is a worse place
|
||||
/// to find it than at load.
|
||||
pub(crate) fn check_fused_experts(
|
||||
gate_up: &Tensor,
|
||||
down: &Tensor,
|
||||
intermediate: usize,
|
||||
) -> Result<Vec<Qwen3_5MLP>> {
|
||||
) -> Result<()> {
|
||||
let (experts, fused, _) = gate_up.dims3()?;
|
||||
anyhow::ensure!(
|
||||
fused == intermediate * 2,
|
||||
@@ -111,78 +145,27 @@ pub(crate) fn split_fused_experts(
|
||||
"down_proj holds {} experts, gate_up_proj holds {experts}",
|
||||
down.dims3()?.0
|
||||
);
|
||||
|
||||
let mut out = Vec::with_capacity(experts);
|
||||
for e in 0..experts {
|
||||
let fused_e = gate_up.i(e)?;
|
||||
out.push(Qwen3_5MLP::from_weights(
|
||||
Linear::new(fused_e.narrow(0, 0, intermediate)?.contiguous()?, None),
|
||||
Linear::new(
|
||||
fused_e
|
||||
.narrow(0, intermediate, intermediate)?
|
||||
.contiguous()?,
|
||||
None,
|
||||
),
|
||||
Linear::new(down.i(e)?.contiguous()?, None),
|
||||
));
|
||||
}
|
||||
Ok(out)
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use candle_core::{DType, Device, Module};
|
||||
use candle_core::{DType, Device};
|
||||
|
||||
/// One expert, two hidden dims, one intermediate. The gate row picks
|
||||
/// the first input channel and the up row picks the second, so
|
||||
/// swapping them swaps which channel is squashed by the SiLU — and
|
||||
/// the two answers differ by more than any tolerance.
|
||||
/// The loader validates the fused pair up front. Without this the
|
||||
/// first token to route to a bad expert fails mid-forward, which is
|
||||
/// a worse place to learn the checkpoint disagrees with the config.
|
||||
#[test]
|
||||
fn the_gate_is_the_first_half_of_the_fused_tensor() {
|
||||
let dev = Device::Cpu;
|
||||
// gate_up: [[gate row], [up row]] = [[1, 0], [0, 1]]
|
||||
let gate_up = Tensor::from_vec(vec![1.0f32, 0.0, 0.0, 1.0], (1, 2, 2), &dev).unwrap();
|
||||
// down: (E=1, H=2, I=1)
|
||||
let down = Tensor::from_vec(vec![2.0f32, 3.0], (1, 2, 1), &dev).unwrap();
|
||||
|
||||
let experts = split_fused_experts(&gate_up, &down, 1).unwrap();
|
||||
assert_eq!(experts.len(), 1);
|
||||
|
||||
let x = Tensor::from_vec(vec![1.0f32, 2.0], (1, 2), &dev).unwrap();
|
||||
let got: Vec<f32> = experts[0]
|
||||
.forward(&x)
|
||||
.unwrap()
|
||||
.flatten_all()
|
||||
.unwrap()
|
||||
.to_vec1()
|
||||
.unwrap();
|
||||
|
||||
// gate(x) = 1, up(x) = 2 -> silu(1) * 2 -> down
|
||||
let silu1 = 1.0f32 / (1.0 + (-1.0f32).exp());
|
||||
let want = [2.0 * silu1 * 2.0, 3.0 * silu1 * 2.0];
|
||||
for (g, w) in got.iter().zip(want.iter()) {
|
||||
assert!((g - w).abs() < 1e-5, "got {got:?} want {want:?}");
|
||||
}
|
||||
|
||||
// The swapped reading: silu(2) * 1, which is a different model.
|
||||
let silu2 = 2.0f32 / (1.0 + (-2.0f32).exp());
|
||||
assert!(
|
||||
(got[0] - 2.0 * silu2).abs() > 1e-3,
|
||||
"this is silu(up) * gate, not silu(gate) * up: {got:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_fused_tensor_of_the_wrong_width_is_rejected() {
|
||||
fn a_fused_pair_that_disagrees_with_the_config_is_rejected() {
|
||||
let dev = Device::Cpu;
|
||||
let gate_up = Tensor::zeros((2, 6, 4), DType::F32, &dev).unwrap();
|
||||
let down = Tensor::zeros((2, 4, 3), DType::F32, &dev).unwrap();
|
||||
assert!(split_fused_experts(&gate_up, &down, 3).is_ok());
|
||||
assert!(check_fused_experts(&gate_up, &down, 3).is_ok());
|
||||
// 2 x 4 != 6
|
||||
assert!(split_fused_experts(&gate_up, &down, 4).is_err());
|
||||
assert!(check_fused_experts(&gate_up, &down, 4).is_err());
|
||||
// expert counts must agree
|
||||
let down_short = Tensor::zeros((1, 4, 3), DType::F32, &dev).unwrap();
|
||||
assert!(split_fused_experts(&gate_up, &down_short, 3).is_err());
|
||||
assert!(check_fused_experts(&gate_up, &down_short, 3).is_err());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,263 @@
|
||||
//! The multi-token-prediction head — `qwen4_exp`'s shipped draft model.
|
||||
//!
|
||||
//! The checkpoint carries a complete second decoder layer under `mtp.*`:
|
||||
//! full attention with its own QSA indexer, its own 512-expert MoE, both
|
||||
//! hyper-connections, and a final mixer. It reuses `embed_tokens` and
|
||||
//! `lm_head` (`mtp_use_dedicated_embeddings: false`), so what is new
|
||||
//! here is only how the target model's state and the next token's
|
||||
//! embedding are fused into an input for it.
|
||||
//!
|
||||
//! ```text
|
||||
//! e = fc_embedding(pre_fc_norm_embedding(embed(t))) [T, 2560]
|
||||
//! h = pre_fc_norm_hidden(h.flatten(-2)) ONE norm over 10240
|
||||
//! h = fc_hidden(h.view(T, hc, 2560)) per stream, shared matrix
|
||||
//! h = e.unsqueeze(-2) + h embedding into every stream
|
||||
//! h = h.flatten(-2) [T, 10240] -> the layer
|
||||
//! ```
|
||||
//!
|
||||
//! vLLM calls this `residual_linear_shared`, as against the
|
||||
//! `Linear(2H, H)` + repeat other MTP variants use. Three parts of it
|
||||
//! cannot be recovered from the tensor shapes, and all three were read
|
||||
//! from `vllm/models/qwen4_exp/nvidia/mtp.py` rather than inferred:
|
||||
//!
|
||||
//! 1. **`pre_fc_norm_hidden` is ungrouped** — one normalisation over the
|
||||
//! whole 10240, unlike every other 10240-wide norm in this
|
||||
//! architecture. vLLM has a `GroupedGemmaRMSNorm` and deliberately
|
||||
//! does not use it here. Grouping it rescales the draft input in four
|
||||
//! independent pieces, which costs acceptance rate and reports
|
||||
//! nothing.
|
||||
//! 2. **The hidden state is the *pre-final-mixer* multi-stream** — `h`
|
||||
//! after the target's last decoder layer, before its mixer collapses
|
||||
//! it. Not the 2560 the LM head sees.
|
||||
//! 3. **The head emits both** — a collapsed stream for `lm_head` and the
|
||||
//! pre-final-mixer stream for its own next step. A head that returned
|
||||
//! only the first cannot be chained past one token.
|
||||
//!
|
||||
//! PLE is off in this layer while the stream count is kept, which the
|
||||
//! absence of `mtp.layers.0.ple.*` in the checkpoint agrees with.
|
||||
//!
|
||||
//! **The head keeps its own KV and indexer caches, and they are not the
|
||||
//! target's.** It therefore has to walk the prompt alongside the target
|
||||
//! before it can draft: a head fed only the newest token is attending
|
||||
//! over an empty past, and this architecture says so rather than
|
||||
//! guessing, because the indexer's divergence check fires. That makes
|
||||
//! prefill part of the speculative loop's cost, not just decode.
|
||||
//!
|
||||
//! See `doc/qwen4_exp-port-spec.md` §9 and #313.
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use candle_core::quantized::GgmlDType;
|
||||
use candle_core::{D, Module, Tensor};
|
||||
use candle_nn::Linear;
|
||||
use candle_nn::var_builder::ShardedVarBuilder;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::harness::arch::qwen3_5::rmsnorm::Qwen3_5RmsNorm;
|
||||
use crate::harness::arch::qwen3_5::rope::RotaryEmbedding;
|
||||
|
||||
use super::config::TextConfig;
|
||||
use super::decoder::DecoderLayer;
|
||||
use super::hyper::HyperConnection;
|
||||
|
||||
/// One draft step's two outputs.
|
||||
#[derive(Debug)]
|
||||
pub struct MtpStep {
|
||||
/// `[B, L, hidden]` — the mixer's collapse, for `lm_head`.
|
||||
pub sample_hidden: Tensor,
|
||||
/// `[B, L, hidden * hc_count]` — pre-final-mixer, and the input to
|
||||
/// the *next* draft step. Chaining reads this, not the collapse.
|
||||
pub multi_stream: Tensor,
|
||||
}
|
||||
|
||||
pub struct MtpHead {
|
||||
pre_fc_norm_embedding: Qwen3_5RmsNorm,
|
||||
/// Ungrouped over `hidden * hc_count` — see the module note.
|
||||
pre_fc_norm_hidden: Qwen3_5RmsNorm,
|
||||
fc_embedding: Linear,
|
||||
/// Shared across the streams: one `hidden -> hidden` matrix applied
|
||||
/// to each branch, not a `wide -> wide` one.
|
||||
fc_hidden: Linear,
|
||||
layer: DecoderLayer,
|
||||
mixer: HyperConnection,
|
||||
hidden_size: usize,
|
||||
hc_count: usize,
|
||||
}
|
||||
|
||||
impl MtpHead {
|
||||
/// `vb` is the checkpoint root; the head hangs off `mtp.`.
|
||||
pub fn load(
|
||||
cfg: &TextConfig,
|
||||
rotary: Arc<RotaryEmbedding>,
|
||||
quant: Option<GgmlDType>,
|
||||
vb: &ShardedVarBuilder,
|
||||
) -> Result<Self> {
|
||||
let mtp = vb.pp("mtp");
|
||||
let (h, hc) = (cfg.hidden_size, cfg.hc_count);
|
||||
Ok(Self {
|
||||
pre_fc_norm_embedding: Qwen3_5RmsNorm::load(
|
||||
&mtp.pp("pre_fc_norm_embedding"),
|
||||
h,
|
||||
cfg.rms_norm_eps,
|
||||
)?,
|
||||
// `load`, not `load_grouped`: the one 10240-wide norm in
|
||||
// this architecture that is not per-stream.
|
||||
pre_fc_norm_hidden: Qwen3_5RmsNorm::load(
|
||||
&mtp.pp("pre_fc_norm_hidden"),
|
||||
h * hc,
|
||||
cfg.rms_norm_eps,
|
||||
)?,
|
||||
fc_embedding: linear(&mtp, "fc_embedding", h, h)?,
|
||||
fc_hidden: linear(&mtp, "fc_hidden", h, h)?,
|
||||
layer: DecoderLayer::load_typed(
|
||||
cfg,
|
||||
rotary,
|
||||
"full_attention",
|
||||
false,
|
||||
quant,
|
||||
&mtp.pp("layers").pp(0),
|
||||
)
|
||||
.context("load mtp.layers.0")?,
|
||||
mixer: HyperConnection::load(
|
||||
&mtp.pp("hyper_connection_mixer"),
|
||||
h,
|
||||
hc,
|
||||
cfg.hc_lowrank,
|
||||
cfg.rms_norm_eps,
|
||||
false,
|
||||
)
|
||||
.context("load mtp.hyper_connection_mixer")?,
|
||||
hidden_size: h,
|
||||
hc_count: hc,
|
||||
})
|
||||
}
|
||||
|
||||
/// One draft step.
|
||||
///
|
||||
/// `token_embed` is `embed_tokens(t)` for the token being predicted
|
||||
/// *from*, `[B, L, hidden]`. `hidden` is the pre-final-mixer stream,
|
||||
/// `[B, L, hidden * hc_count]` — from the target model on the first
|
||||
/// step, from this head's own previous [`MtpStep::multi_stream`]
|
||||
/// afterwards.
|
||||
pub fn forward(
|
||||
&mut self,
|
||||
token_embed: &Tensor,
|
||||
hidden: &Tensor,
|
||||
causal_mask: Option<&Tensor>,
|
||||
cos: &Tensor,
|
||||
sin: &Tensor,
|
||||
past_len: usize,
|
||||
) -> Result<MtpStep> {
|
||||
let (b, l, wide) = hidden.dims3()?;
|
||||
anyhow::ensure!(
|
||||
wide == self.hidden_size * self.hc_count,
|
||||
"mtp: expected the pre-final-mixer stream ({} wide), got {wide} — \
|
||||
passing the collapsed hidden state here is the likely mistake",
|
||||
self.hidden_size * self.hc_count
|
||||
);
|
||||
|
||||
let e = self
|
||||
.fc_embedding
|
||||
.forward(&self.pre_fc_norm_embedding.forward(token_embed)?)?;
|
||||
|
||||
// Normalise the flattened stream, *then* split it: one
|
||||
// normalisation over the whole width.
|
||||
let h = self.pre_fc_norm_hidden.forward(hidden)?.reshape((
|
||||
b,
|
||||
l,
|
||||
self.hc_count,
|
||||
self.hidden_size,
|
||||
))?;
|
||||
let h = self.fc_hidden.forward(&h)?;
|
||||
|
||||
// The embedding enters every stream.
|
||||
let h = h
|
||||
.broadcast_add(&e.unsqueeze(D::Minus2)?)?
|
||||
.flatten_from(D::Minus2)?;
|
||||
|
||||
let h = self
|
||||
.layer
|
||||
.forward(&h, None, causal_mask, cos, sin, past_len)
|
||||
.context("mtp draft layer")?;
|
||||
|
||||
Ok(MtpStep {
|
||||
sample_hidden: self.mixer.collapse(&h)?,
|
||||
multi_stream: h,
|
||||
})
|
||||
}
|
||||
|
||||
/// Drop the draft layer's caches. The head has its own KV and its
|
||||
/// own indexer cache, independent of the target model's.
|
||||
pub fn clear_kv_cache(&mut self) -> Result<()> {
|
||||
self.layer.clear_kv_cache()
|
||||
}
|
||||
}
|
||||
|
||||
fn linear(vb: &ShardedVarBuilder, name: &str, in_dim: usize, out_dim: usize) -> Result<Linear> {
|
||||
let weight = vb
|
||||
.pp(name)
|
||||
.get((out_dim, in_dim), "weight")
|
||||
.with_context(|| format!("load '{}/{name}/weight'", vb.prefix()))?;
|
||||
Ok(Linear::new(weight, None))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use candle_core::{DType, Device};
|
||||
|
||||
/// `pre_fc_norm_hidden` normalises the whole 10240 at once, where
|
||||
/// every other norm this wide in the architecture normalises four
|
||||
/// streams of 2560 independently. The port spec asserted the
|
||||
/// opposite until vLLM settled it, and the difference is invisible
|
||||
/// downstream — it rescales the draft input and costs acceptance
|
||||
/// rate.
|
||||
///
|
||||
/// So it is pinned by value, on an input the two readings cannot
|
||||
/// agree on: two streams whose magnitudes differ threefold. Grouped,
|
||||
/// each is normalised to its own scale and both come out all-ones.
|
||||
/// Ungrouped, the larger stream stays larger.
|
||||
#[test]
|
||||
fn the_hidden_pre_norm_is_ungrouped() {
|
||||
let dev = Device::Cpu;
|
||||
let (hidden, hc) = (4usize, 2usize);
|
||||
let wide = hidden * hc;
|
||||
// Zero weight: (1 + w) = 1, so this is a pure RMS normalisation.
|
||||
let norm =
|
||||
Qwen3_5RmsNorm::from_weight(Tensor::zeros(wide, DType::F32, &dev).unwrap(), 1e-6, None);
|
||||
let h = Tensor::from_vec(
|
||||
vec![1.0f32, 1.0, 1.0, 1.0, 3.0, 3.0, 3.0, 3.0],
|
||||
(1, 1, wide),
|
||||
&dev,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let got: Vec<f32> = norm
|
||||
.forward(&h)
|
||||
.unwrap()
|
||||
.flatten_all()
|
||||
.unwrap()
|
||||
.to_vec1()
|
||||
.unwrap();
|
||||
|
||||
// rms over all eight = sqrt((4*1 + 4*9)/8) = sqrt(5)
|
||||
let rms = 5.0f32.sqrt();
|
||||
let want = [
|
||||
1.0 / rms,
|
||||
1.0 / rms,
|
||||
1.0 / rms,
|
||||
1.0 / rms,
|
||||
3.0 / rms,
|
||||
3.0 / rms,
|
||||
3.0 / rms,
|
||||
3.0 / rms,
|
||||
];
|
||||
for (g, w) in got.iter().zip(want.iter()) {
|
||||
assert!((g - w).abs() < 1e-5, "got {got:?} want {want:?}");
|
||||
}
|
||||
// The grouped reading would flatten both streams to all-ones.
|
||||
assert!(
|
||||
(got[4] - 1.0).abs() > 1e-3,
|
||||
"this is the grouped reading, which the spec had and vLLM does not: {got:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -491,6 +491,26 @@ impl PleBlock {
|
||||
(self.kernel_size - 1) * self.dilation
|
||||
}
|
||||
|
||||
/// Capture the rolling conv context for a prefix snapshot.
|
||||
///
|
||||
/// Deep-copied rather than shared. The buffer is replaced by
|
||||
/// assignment today, so a shallow clone would be sound — but
|
||||
/// `narrow(..).contiguous()` on an already-contiguous slice is a
|
||||
/// view, so whether the snapshot shares storage depends on a
|
||||
/// property of the *data*, not of the code. That is exactly the
|
||||
/// kind of conditional aliasing the GatedDeltaNet states are
|
||||
/// deep-copied to avoid, and nine positions of a 10240-wide signal
|
||||
/// is not worth being clever about.
|
||||
pub fn snapshot_state(&self) -> candle_core::Result<Option<Tensor>> {
|
||||
self.conv_state.as_ref().map(Tensor::copy).transpose()
|
||||
}
|
||||
|
||||
/// Replace the rolling conv context from a snapshot.
|
||||
pub fn restore_state(&mut self, conv_state: Option<&Tensor>) -> candle_core::Result<()> {
|
||||
self.conv_state = conv_state.map(Tensor::copy).transpose()?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Drop the rolling conv context. Call alongside a KV-cache clear;
|
||||
/// the carried n-gram ids ([`NGramHasher::context_len`]) reset with
|
||||
/// it.
|
||||
|
||||
@@ -293,6 +293,21 @@ impl Indexer {
|
||||
&self.selector
|
||||
}
|
||||
|
||||
/// Capture the raw key cache for a prefix snapshot.
|
||||
///
|
||||
/// A shallow clone, like the attention K/V it travels with: the
|
||||
/// cache is only ever *replaced* by a `cat` into a fresh
|
||||
/// allocation, never written in place, so the snapshot stays valid
|
||||
/// after the live cache moves on.
|
||||
pub fn snapshot_keys(&self) -> Option<Tensor> {
|
||||
self.key_cache.clone()
|
||||
}
|
||||
|
||||
/// Replace the key cache from a snapshot.
|
||||
pub fn restore_keys(&mut self, keys: Option<&Tensor>) {
|
||||
self.key_cache = keys.cloned();
|
||||
}
|
||||
|
||||
/// Drop the indexer's cache. Must happen with the main KV clear —
|
||||
/// a stale indexer cache selects blocks of another request's text.
|
||||
pub fn clear_cache(&mut self) {
|
||||
|
||||
+73
-2
@@ -1,5 +1,12 @@
|
||||
//! Cache-state snapshots for prefix KV caching (#11).
|
||||
//!
|
||||
//! Shared across architectures rather than owned by one: `qwen3_5`,
|
||||
//! its TP mirror `tp_qwen3_5`, and `qwen4_exp` all capture state
|
||||
//! through these types, and `ModelArch::snapshot_kv_cache` hands one
|
||||
//! back whatever the architecture. It lived under `arch/qwen3_5/`
|
||||
//! while that was the only arch with a snapshot, which stopped being
|
||||
//! true.
|
||||
//!
|
||||
//! A snapshot captures everything `clear_kv_cache` would destroy, at
|
||||
//! one consistent token boundary:
|
||||
//!
|
||||
@@ -40,6 +47,17 @@ pub enum LayerKvSnapshot {
|
||||
conv_state: Option<Tensor>,
|
||||
recurrent_state: Option<Tensor>,
|
||||
},
|
||||
/// A full-attention layer that also keeps a QSA indexer cache
|
||||
/// (`qwen4_exp`). The indexer's raw keys are a second, separately
|
||||
/// shaped cache — one head of 128 dims per position — and a
|
||||
/// snapshot that captured only `kv` would restore a layer whose
|
||||
/// indexer still holds the *previous* request's keys, selecting
|
||||
/// blocks of somebody else's text. It fails fluently, so the two
|
||||
/// travel together or not at all.
|
||||
FullSparse {
|
||||
kv: Option<(Tensor, Tensor)>,
|
||||
indexer_keys: Option<Tensor>,
|
||||
},
|
||||
}
|
||||
|
||||
/// One consistent cache snapshot of a `Qwen3_5Model` (or its TP
|
||||
@@ -50,6 +68,9 @@ pub enum LayerKvSnapshot {
|
||||
pub struct KvCacheSnapshot {
|
||||
pub(crate) layers: Vec<LayerKvSnapshot>,
|
||||
pub(crate) rope_delta: i64,
|
||||
/// `qwen4_exp`'s PLE state. `None` for architectures without a
|
||||
/// hashed n-gram layer, which is every other one today.
|
||||
pub(crate) ple: Option<PleSnapshot>,
|
||||
}
|
||||
|
||||
impl KvCacheSnapshot {
|
||||
@@ -79,8 +100,44 @@ impl KvCacheSnapshot {
|
||||
conv_state.as_ref().map(t_bytes).unwrap_or(0)
|
||||
+ recurrent_state.as_ref().map(t_bytes).unwrap_or(0)
|
||||
}
|
||||
LayerKvSnapshot::FullSparse { kv, indexer_keys } => {
|
||||
kv.as_ref()
|
||||
.map(|(k, v)| t_bytes(k) + t_bytes(v))
|
||||
.unwrap_or(0)
|
||||
+ indexer_keys.as_ref().map(t_bytes).unwrap_or(0)
|
||||
}
|
||||
})
|
||||
.sum()
|
||||
.sum::<u64>()
|
||||
+ self.ple.as_ref().map(PleSnapshot::size_bytes).unwrap_or(0)
|
||||
}
|
||||
}
|
||||
|
||||
/// PLE's per-request state (`qwen4_exp`), captured at the same token
|
||||
/// boundary as the layers.
|
||||
///
|
||||
/// Two pieces with different natures. `conv_state` is the dilated
|
||||
/// short conv's left context — nine positions of a 10240-wide signal,
|
||||
/// deep-copied like the GatedDeltaNet states because the conv writes
|
||||
/// its rolling buffer in place. `carried_ids` is the previous
|
||||
/// `ngram_size - 1` **token ids** per batch row, which upstream keeps
|
||||
/// as a third "conv state" on the same layer; they are ids, not
|
||||
/// activations, and cost two integers per sequence.
|
||||
pub struct PleSnapshot {
|
||||
pub(crate) conv_state: Option<Tensor>,
|
||||
pub(crate) carried_ids: Vec<Vec<i64>>,
|
||||
}
|
||||
|
||||
impl PleSnapshot {
|
||||
fn size_bytes(&self) -> u64 {
|
||||
self.conv_state
|
||||
.as_ref()
|
||||
.map(|t| (t.elem_count() * t.dtype().size_in_bytes()) as u64)
|
||||
.unwrap_or(0)
|
||||
+ self
|
||||
.carried_ids
|
||||
.iter()
|
||||
.map(|r| (r.len() * std::mem::size_of::<i64>()) as u64)
|
||||
.sum::<u64>()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -145,6 +202,7 @@ pub fn assemble_batch(seqs: &[(&KvCacheSnapshot, usize)]) -> candle_core::Result
|
||||
}
|
||||
Ok(BatchedKvState {
|
||||
snapshot: KvCacheSnapshot {
|
||||
ple: None,
|
||||
layers,
|
||||
rope_delta: 0,
|
||||
},
|
||||
@@ -159,6 +217,14 @@ fn assemble_layer(
|
||||
padded_len: usize,
|
||||
) -> candle_core::Result<LayerKvSnapshot> {
|
||||
match &seqs[0].0.layers[li] {
|
||||
// Batched decode for the sparse-attention layers is not built
|
||||
// yet. Assembling them as if they were plain full-attention
|
||||
// would drop the indexer cache and hand every row a selection
|
||||
// computed from another request's keys, which produces text.
|
||||
LayerKvSnapshot::FullSparse { .. } => candle_core::bail!(
|
||||
"assemble_batch: layer {li} carries a QSA indexer cache; batched decode \
|
||||
for this architecture is not implemented"
|
||||
),
|
||||
LayerKvSnapshot::Full(_) => {
|
||||
let mut ks = Vec::with_capacity(seqs.len());
|
||||
let mut vs = Vec::with_capacity(seqs.len());
|
||||
@@ -242,6 +308,10 @@ pub fn extract_row(
|
||||
let mut layers = Vec::with_capacity(snap.layers.len());
|
||||
for (li, layer) in snap.layers.iter().enumerate() {
|
||||
layers.push(match layer {
|
||||
LayerKvSnapshot::FullSparse { .. } => candle_core::bail!(
|
||||
"extract_row: layer {li} carries a QSA indexer cache; batched decode \
|
||||
for this architecture is not implemented"
|
||||
),
|
||||
LayerKvSnapshot::Full(Some((k, v))) => {
|
||||
let (b, _h, s, _d) = k.dims4()?;
|
||||
if row >= b {
|
||||
@@ -274,6 +344,7 @@ pub fn extract_row(
|
||||
});
|
||||
}
|
||||
Ok(KvCacheSnapshot {
|
||||
ple: None,
|
||||
layers,
|
||||
rope_delta: 0,
|
||||
})
|
||||
@@ -312,7 +383,7 @@ fn pad_seq(t: &Tensor, padded_len: usize) -> candle_core::Result<Tensor> {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::super::{Qwen3_5Model, RopeParameters, TextConfig};
|
||||
use crate::harness::arch::qwen3_5::{Qwen3_5Model, RopeParameters, TextConfig};
|
||||
use candle_core::{DType, Device, Tensor};
|
||||
use std::collections::HashMap;
|
||||
|
||||
@@ -555,7 +555,7 @@ impl LoadedHandle {
|
||||
#[derive(Clone)]
|
||||
pub enum KvSnapshotRef {
|
||||
Worker(super::device_worker::KvSnapshotId),
|
||||
Local(Arc<super::arch::qwen3_5::snapshot::KvCacheSnapshot>),
|
||||
Local(Arc<super::arch::snapshot::KvCacheSnapshot>),
|
||||
#[cfg(feature = "cuda")]
|
||||
Tp(u64),
|
||||
}
|
||||
@@ -1116,15 +1116,16 @@ impl ModelArch {
|
||||
/// candle-transformers archs keep theirs private, so they stay on
|
||||
/// the clear-every-request path.
|
||||
pub fn supports_kv_snapshot(&self) -> bool {
|
||||
matches!(self, ModelArch::Qwen3_5Dense(_))
|
||||
matches!(self, ModelArch::Qwen3_5Dense(_) | ModelArch::Qwen4Exp(_))
|
||||
}
|
||||
|
||||
/// Capture the live cache state as a prefix snapshot. See
|
||||
/// `arch/qwen3_5/snapshot.rs` for what a snapshot contains and the
|
||||
/// copy-semantics constraints.
|
||||
pub fn snapshot_kv_cache(&self) -> Result<super::arch::qwen3_5::snapshot::KvCacheSnapshot> {
|
||||
pub fn snapshot_kv_cache(&self) -> Result<super::arch::snapshot::KvCacheSnapshot> {
|
||||
match self {
|
||||
ModelArch::Qwen3_5Dense(m) => Ok(m.snapshot_kv_cache()?),
|
||||
ModelArch::Qwen4Exp(m) => Ok(m.snapshot_kv_cache()?),
|
||||
_ => anyhow::bail!("snapshot_kv_cache: architecture has no snapshot support"),
|
||||
}
|
||||
}
|
||||
@@ -1133,10 +1134,11 @@ impl ModelArch {
|
||||
/// restore-instead-of-clear half of prefix caching.
|
||||
pub fn restore_kv_cache(
|
||||
&mut self,
|
||||
snap: &super::arch::qwen3_5::snapshot::KvCacheSnapshot,
|
||||
snap: &super::arch::snapshot::KvCacheSnapshot,
|
||||
) -> Result<()> {
|
||||
match self {
|
||||
ModelArch::Qwen3_5Dense(m) => Ok(m.restore_kv_cache(snap)?),
|
||||
ModelArch::Qwen4Exp(m) => Ok(m.restore_kv_cache(snap)?),
|
||||
_ => anyhow::bail!("restore_kv_cache: architecture has no snapshot support"),
|
||||
}
|
||||
}
|
||||
@@ -1347,7 +1349,7 @@ impl LlamaDense {
|
||||
/// value. New entries land alongside a new `ModelArch` variant + a
|
||||
/// dispatch branch in `load_arch_dense` (plus, for TP, a parallel
|
||||
/// pattern in `tp_qwen3.rs`).
|
||||
const DENSE_SUPPORTED_MODEL_TYPES: &[&str] = &[
|
||||
pub(crate) const DENSE_SUPPORTED_MODEL_TYPES: &[&str] = &[
|
||||
"llama",
|
||||
"qwen3",
|
||||
"qwen3_5",
|
||||
@@ -2733,6 +2735,9 @@ impl CandleHarness {
|
||||
self.resolve_dense_files(spec, source_id).await?;
|
||||
let device_for_load = device.clone();
|
||||
let model_id_for_log = spec.model_id.clone();
|
||||
// Resolved out here: `spec` is borrowed and does not outlive the
|
||||
// blocking task.
|
||||
let isq = super::quant::parse_quant_string(spec.quant.as_deref())?;
|
||||
|
||||
let arch = tokio::task::spawn_blocking(move || -> Result<ModelArch> {
|
||||
let cfg_text = std::fs::read_to_string(&config_path).context("read config.json")?;
|
||||
@@ -2843,9 +2848,19 @@ impl CandleHarness {
|
||||
)
|
||||
.context("build ShardedVarBuilder for qwen4_exp")?
|
||||
};
|
||||
// ISQ is not optional for this architecture in
|
||||
// practice: without it the routed experts are
|
||||
// 241.6 GB. `quant` comes from the catalogue, and a
|
||||
// load without one will fail on memory rather than
|
||||
// on anything this code can say (#315).
|
||||
// ISQ is not really optional here: without it the
|
||||
// routed experts are 241.6 GB and the load fails on
|
||||
// memory rather than on anything this code can say.
|
||||
// The catalogue's `quant` is the knob (#315, #252).
|
||||
let model = super::arch::qwen4_exp::model::Qwen4ExpForCausalLM::load(
|
||||
&cfg,
|
||||
dtype,
|
||||
isq,
|
||||
&device_for_load,
|
||||
&sharded_vb,
|
||||
)
|
||||
@@ -2870,6 +2885,59 @@ impl CandleHarness {
|
||||
Ok((tokenizer_path, arch))
|
||||
}
|
||||
|
||||
/// Whether this spec's `quant` names a GGUF the repo actually has.
|
||||
///
|
||||
/// `quant` is overloaded (#320): the TP path reads it as in-situ
|
||||
/// quantisation of safetensors, while the single-GPU paths read it
|
||||
/// as "select the GGUF whose filename contains this string". A repo
|
||||
/// with no GGUF at all cannot mean the second, so rather than fail
|
||||
/// with an error about a file that was never going to exist, the
|
||||
/// same string is read the way the TP path reads it.
|
||||
///
|
||||
/// This is deliberately narrow. A repo that *does* carry GGUFs keeps
|
||||
/// the old behaviour exactly, including the old error when none of
|
||||
/// them match the requested quant — that case is a typo, not an ISQ
|
||||
/// request, and turning it into a silent 360 GB dense load would be
|
||||
/// a worse answer than the error.
|
||||
async fn quant_selects_a_gguf(
|
||||
&self,
|
||||
spec: &ModelSpec,
|
||||
source_id: &cortex_core::source::ModelSourceId,
|
||||
) -> Result<bool> {
|
||||
let Some(quant) = spec.quant.as_deref().filter(|q| !q.is_empty()) else {
|
||||
return Ok(false);
|
||||
};
|
||||
let api = self.hf_api_for(&source_id.scheme)?;
|
||||
let info = api
|
||||
.model(source_id.repo_path())
|
||||
.info()
|
||||
.await
|
||||
.with_context(|| format!("fetch HF repo info for {source_id}"))?;
|
||||
let names: Vec<String> = info
|
||||
.siblings
|
||||
.iter()
|
||||
.map(|s| s.rfilename.to_lowercase())
|
||||
.collect();
|
||||
let any_gguf = names.iter().any(|n| n.ends_with(".gguf"));
|
||||
if any_gguf {
|
||||
return Ok(true);
|
||||
}
|
||||
let any_safetensors = names.iter().any(|n| n.ends_with(".safetensors"));
|
||||
if any_safetensors {
|
||||
tracing::info!(
|
||||
model = %source_id,
|
||||
quant = %quant,
|
||||
"no GGUF in this repo; reading `quant` as in-situ quantization of \
|
||||
the safetensors (#320)"
|
||||
);
|
||||
return Ok(false);
|
||||
}
|
||||
anyhow::bail!(
|
||||
"repo {source_id} has neither a GGUF nor safetensors; `quant = {quant:?}` \
|
||||
cannot be satisfied"
|
||||
)
|
||||
}
|
||||
|
||||
/// Resolve a model spec to local GGUF and tokenizer file paths via
|
||||
/// hf-hub. Downloads on first use; subsequent calls are cached.
|
||||
async fn resolve_files(
|
||||
@@ -3854,7 +3922,7 @@ impl Harness for CandleHarness {
|
||||
context_profile,
|
||||
) = if let Some(w) = &worker {
|
||||
// CUDA path: resolve, then load in the worker.
|
||||
if spec.quant.is_some() {
|
||||
if self.quant_selects_a_gguf(spec, &source_id).await? {
|
||||
let (gguf_path, tokenizer_path) = self.resolve_files(spec, &source_id).await?;
|
||||
let handle = w
|
||||
.load_gguf(gguf_path, spec.model_id.clone())
|
||||
@@ -3878,36 +3946,38 @@ impl Harness for CandleHarness {
|
||||
let (config_path, tokenizer_path, safetensors_paths) =
|
||||
self.resolve_dense_files(spec, &source_id).await?;
|
||||
let meta = VisionMeta::from_config_path(&config_path);
|
||||
// Prefix snapshots (#11) exist only for the in-tree
|
||||
// qwen3_5 arch — which serves BOTH its model_types
|
||||
// (qwen3_5 and the flat-config qwen3_next MoE family);
|
||||
// the worker holds the ModelArch so the async side
|
||||
// decides from config.json instead.
|
||||
let snapshot_capable = matches!(
|
||||
config_model_type(&config_path).as_deref(),
|
||||
Some(super::arch::qwen3_5::MODEL_TYPE)
|
||||
| Some(super::arch::qwen3_5::MODEL_TYPE_NEXT)
|
||||
);
|
||||
// Context-limit physics (#67): single-GPU → world_size 1.
|
||||
// `None` for non-qwen3_5 dense archs.
|
||||
// `None` for dense archs with no profile builder — they
|
||||
// fall back to the static prompt cap.
|
||||
let context_profile =
|
||||
super::context_limit::profile_from_qwen3_5_config(&config_path, 1);
|
||||
let handle = w
|
||||
.load_dense(config_path, safetensors_paths, spec.model_id.clone())
|
||||
super::context_limit::profile_from_qwen3_5_config(&config_path, 1).or_else(
|
||||
|| super::context_limit::profile_from_qwen4_exp_config(&config_path, 1),
|
||||
);
|
||||
// In-situ quantisation (#320) reaches the worker load
|
||||
// too, not just the CPU one — parsed here because
|
||||
// `spec` does not cross the channel.
|
||||
let isq = super::quant::parse_quant_string(spec.quant.as_deref())?;
|
||||
let load = w
|
||||
.load_dense(config_path, safetensors_paths, spec.model_id.clone(), isq)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("worker load_dense: {e:#}"))?;
|
||||
// Prefix snapshots (#11): the built model answers for
|
||||
// itself on the way back, rather than the async side
|
||||
// re-deriving the answer from `config.json`. See
|
||||
// `DenseLoad` — the re-derivation is what silently
|
||||
// dropped batched decode for qwen4_exp.
|
||||
(
|
||||
tokenizer_path,
|
||||
None,
|
||||
Some(handle),
|
||||
Some(load.handle),
|
||||
meta,
|
||||
snapshot_capable,
|
||||
load.supports_kv_snapshot,
|
||||
context_profile,
|
||||
)
|
||||
}
|
||||
} else {
|
||||
// CPU path: legacy spawn_blocking + Arc<Mutex<ModelArch>>.
|
||||
let (tokenizer_path, arch) = if spec.quant.is_some() {
|
||||
let (tokenizer_path, arch) = if self.quant_selects_a_gguf(spec, &source_id).await? {
|
||||
self.load_arch_gguf(spec, &source_id, &device).await?
|
||||
} else {
|
||||
self.load_arch_dense(spec, &source_id, &device).await?
|
||||
@@ -4725,6 +4795,13 @@ impl CandleHarness {
|
||||
image_token_id: vision_meta.image_token_id,
|
||||
image_grid_factor: vision_meta.image_grid_factor,
|
||||
spec: spec.clone(),
|
||||
// Snapshot capability re-derived from `config.json`
|
||||
// rather than asked of the loaded shard. Correct today —
|
||||
// `TP_SUPPORTED_MODEL_TYPES` is exactly these two — but it
|
||||
// is the same shape as the bug `DenseLoad` fixed on the
|
||||
// single-GPU path, so whoever adds an arch here (#319)
|
||||
// must add it in both places or lose batched decode
|
||||
// silently.
|
||||
prefix_cache: self.new_prefix_cache(matches!(
|
||||
config_model_type(&config_path).as_deref(),
|
||||
Some(super::arch::qwen3_5::MODEL_TYPE)
|
||||
@@ -6276,9 +6353,14 @@ struct VisionMeta {
|
||||
}
|
||||
|
||||
/// Peek at `config.json` for its `model_type`. Best-effort — `None`
|
||||
/// on any read/parse error. The load path uses this to decide prefix-
|
||||
/// snapshot capability for worker-held models the async side can't
|
||||
/// inspect directly.
|
||||
/// on any read/parse error.
|
||||
///
|
||||
/// Only the TP load path still asks: the single-GPU path takes the
|
||||
/// arch's own answer off the load reply (`DenseLoad`) rather than
|
||||
/// re-deriving it here, which is what stopped the two from drifting.
|
||||
/// The TP path has the same hazard and not yet the same cure — see
|
||||
/// the note at its call site.
|
||||
#[cfg(feature = "cuda")]
|
||||
fn config_model_type(config_path: &std::path::Path) -> Option<String> {
|
||||
let text = std::fs::read_to_string(config_path).ok()?;
|
||||
let v: serde_json::Value = serde_json::from_str(&text).ok()?;
|
||||
|
||||
@@ -252,6 +252,85 @@ pub fn profile_from_qwen3_5_config(config_path: &Path, world_size: u32) -> Optio
|
||||
})
|
||||
}
|
||||
|
||||
/// Bytes of **QSA indexer** cache one token adds, per card (#307).
|
||||
///
|
||||
/// `qwen4_exp` keeps a second cache beside the main one: on every
|
||||
/// full-attention layer the indexer stores the raw index key,
|
||||
/// `indexer_kv_heads * indexer_head_dim` wide, in the compute dtype
|
||||
/// (`arch::qwen4_exp::qsa::Indexer::key_cache`). Keys only — the
|
||||
/// indexer scores blocks, it does not attend them — so there is no
|
||||
/// `2 *` here.
|
||||
///
|
||||
/// It does not shard: `indexer_kv_heads` is 1, so a TP rank holds the
|
||||
/// whole thing however wide the world is. At the checkpoint's 12
|
||||
/// full-attention layers x 1 head x 128 dims that is 3 KiB/token on
|
||||
/// top of the main cache's 24 — the surcharge the epic's original
|
||||
/// arithmetic omitted, and it grows with context exactly as the main
|
||||
/// cache does.
|
||||
pub fn indexer_bytes_per_token(
|
||||
n_full_attn_layers: usize,
|
||||
indexer_kv_heads: usize,
|
||||
indexer_head_dim: usize,
|
||||
dtype_bytes: usize,
|
||||
) -> u64 {
|
||||
(n_full_attn_layers * indexer_kv_heads * indexer_head_dim * dtype_bytes) as u64
|
||||
}
|
||||
|
||||
/// Build a [`ContextProfile`] from a `qwen4_exp` `config.json` — the
|
||||
/// sibling of [`profile_from_qwen3_5_config`] for the other hybrid
|
||||
/// arch. Returns `None` for any other `model_type` or an unparseable
|
||||
/// config.
|
||||
///
|
||||
/// Same shape of physics, one term more: the per-token cost is the
|
||||
/// main KV cache **plus** the QSA indexer's, because both grow per
|
||||
/// token and both come out of the same card. Advertising against the
|
||||
/// main cache alone would overstate what is servable by about an
|
||||
/// eighth, which is the class of error #281 is about.
|
||||
///
|
||||
/// `layer_types` is always populated by the time `from_config_json`
|
||||
/// returns (it fills the array from `full_attention_interval` when the
|
||||
/// checkpoint omits it), so counting it is authoritative here.
|
||||
pub fn profile_from_qwen4_exp_config(
|
||||
config_path: &Path,
|
||||
world_size: u32,
|
||||
) -> Option<ContextProfile> {
|
||||
let text = std::fs::read_to_string(config_path).ok()?;
|
||||
if !super::arch::qwen4_exp::config::is_qwen4_exp(&text) {
|
||||
return None;
|
||||
}
|
||||
let cfg = super::arch::qwen4_exp::config::Config::from_config_json(&text).ok()?;
|
||||
let tc = &cfg.text_config;
|
||||
let n_full_attn_layers = tc
|
||||
.layer_types
|
||||
.iter()
|
||||
.filter(|t| t.as_str() == "full_attention")
|
||||
.count();
|
||||
// A model with no full-attention layer has no growing cache, and
|
||||
// the derivation divides by this. Nothing in the family looks like
|
||||
// that, so treat it as a config we do not understand rather than
|
||||
// inventing an unbounded context.
|
||||
if n_full_attn_layers == 0 {
|
||||
return None;
|
||||
}
|
||||
let kv_bytes_per_token_per_card = kv_bytes_per_token(
|
||||
n_full_attn_layers,
|
||||
tc.num_key_value_heads,
|
||||
tc.head_dim,
|
||||
KV_CACHE_DTYPE_BYTES,
|
||||
world_size,
|
||||
) + indexer_bytes_per_token(
|
||||
n_full_attn_layers,
|
||||
tc.indexer_kv_heads,
|
||||
tc.indexer_head_dim,
|
||||
KV_CACHE_DTYPE_BYTES,
|
||||
);
|
||||
Some(ContextProfile {
|
||||
max_position_embeddings: tc.max_position_embeddings,
|
||||
kv_bytes_per_token_per_card,
|
||||
world_size,
|
||||
})
|
||||
}
|
||||
|
||||
/// Round a token count down to a clean boundary so the advertised limit
|
||||
/// doesn't jitter by a handful of tokens as live VRAM / the throughput
|
||||
/// EMA wobble between polls.
|
||||
@@ -375,9 +454,15 @@ pub fn derive_limit(
|
||||
|
||||
// Observability (#126): every input and intermediate term, so a
|
||||
// surprising advertised limit is diagnosable from the journal
|
||||
// instead of re-deriving by hand. DEBUG — this runs on every
|
||||
// `GET /models` poll (a few lines per poll cycle).
|
||||
tracing::debug!(
|
||||
// instead of re-deriving by hand.
|
||||
//
|
||||
// TRACE, not debug: this runs on every `GET /models` poll and every
|
||||
// input is fixed at load, so the line is byte-identical each time —
|
||||
// about 8,600 a day per model at a 10s poll. It is worth having when
|
||||
// reasoning about a context ceiling and worth nothing the rest of
|
||||
// the time, and at debug it buries the lines that are not routine
|
||||
// (#320).
|
||||
tracing::trace!(
|
||||
max_pos = profile.max_position_embeddings,
|
||||
kv_bytes_per_token_per_card = profile.kv_bytes_per_token_per_card,
|
||||
free_tightest_mb,
|
||||
@@ -418,6 +503,57 @@ pub fn derive_limit(
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
|
||||
/// The advertised limit for `qwen4_exp` must price both caches.
|
||||
///
|
||||
/// The main KV cache is 12 full-attention layers x 2 kv heads x 256
|
||||
/// dims x 2 (K and V) x 2 bytes = 24 KiB/token, and the QSA indexer
|
||||
/// adds 12 x 1 x 128 x 2 bytes = 3 KiB on top. Counting only the
|
||||
/// first would advertise ~12.5% more context than the card can
|
||||
/// hold — the failure mode #281 exists to stop, and the one the
|
||||
/// epic's own arithmetic made before the indexer cache was found.
|
||||
///
|
||||
/// Driven from the checkpoint's verbatim `config.json`, so a field
|
||||
/// that changes shape upstream fails here rather than in
|
||||
/// production.
|
||||
#[test]
|
||||
fn a_qwen4_exp_profile_counts_the_indexer_cache_too() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("config.json");
|
||||
std::fs::write(&path, super::super::arch::qwen4_exp::config::SHIPPED).unwrap();
|
||||
|
||||
let profile = profile_from_qwen4_exp_config(&path, 1)
|
||||
.expect("the shipped config.json must yield a profile");
|
||||
|
||||
assert_eq!(profile.max_position_embeddings, 262_144);
|
||||
assert_eq!(
|
||||
profile.kv_bytes_per_token_per_card,
|
||||
24 * 1024 + 3 * 1024,
|
||||
"main KV plus the indexer's second cache"
|
||||
);
|
||||
}
|
||||
|
||||
/// The two hybrid archs must not answer for each other: whichever
|
||||
/// builder runs first has to decline a config it does not own, or
|
||||
/// the load path's `or_else` chain silently prices one model with
|
||||
/// the other's physics.
|
||||
#[test]
|
||||
fn the_profile_builders_decline_each_others_configs() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("config.json");
|
||||
std::fs::write(&path, super::super::arch::qwen4_exp::config::SHIPPED).unwrap();
|
||||
assert!(
|
||||
profile_from_qwen3_5_config(&path, 1).is_none(),
|
||||
"the qwen3_5 builder must decline a qwen4_exp config"
|
||||
);
|
||||
|
||||
let path = dir.path().join("other.json");
|
||||
std::fs::write(&path, r#"{"model_type": "qwen3_next"}"#).unwrap();
|
||||
assert!(
|
||||
profile_from_qwen4_exp_config(&path, 1).is_none(),
|
||||
"the qwen4_exp builder must decline everything else"
|
||||
);
|
||||
}
|
||||
|
||||
/// The deploy's smoke probe must not set the advertised context.
|
||||
///
|
||||
/// 19 tokens in 130 ms reads as ~145 tok/s on a host that measures
|
||||
|
||||
@@ -13,11 +13,13 @@
|
||||
//! ARCH model state in this state slab will gain a companion
|
||||
//! `tp_models: HashMap<TpHandle, Box<TpLeaderModel>>`.
|
||||
|
||||
use crate::harness::arch::qwen3_5::snapshot::KvCacheSnapshot;
|
||||
use crate::harness::arch::snapshot::KvCacheSnapshot;
|
||||
use crate::harness::candle::ModelArch;
|
||||
#[cfg(feature = "cuda")]
|
||||
use crate::harness::device_worker::jobs::TpHandle;
|
||||
use crate::harness::device_worker::jobs::{ArchHandle, ImageHandle, ImageInput, Job, KvSnapshotId};
|
||||
use crate::harness::device_worker::jobs::{
|
||||
ArchHandle, DenseLoad, ImageHandle, ImageInput, Job, KvSnapshotId,
|
||||
};
|
||||
use crate::harness::image::ZImagePipeline;
|
||||
#[cfg(feature = "cuda")]
|
||||
use crate::harness::tp::TpLeaderModel;
|
||||
@@ -131,17 +133,44 @@ pub(crate) fn run(device_index: u32, rx: Receiver<Job>, poisoned: Arc<AtomicBool
|
||||
} => {
|
||||
let result = load_gguf_inner(&state.device, &gguf_path, &model_id)
|
||||
.map(|arch| insert_arch(&mut state, Box::new(arch)));
|
||||
// Same reasoning as `LoadImage`: release load-time
|
||||
// transients before serving, and — on the failure
|
||||
// path — give back whatever the partial load had
|
||||
// allocated before it gave up. Without this a load
|
||||
// that OOMs leaves its bytes in the pool, where the
|
||||
// driver still counts them against the next load.
|
||||
trim_device_pool(&state);
|
||||
let _ = reply.send(result);
|
||||
}
|
||||
Job::LoadDense {
|
||||
config_path,
|
||||
safetensors_paths,
|
||||
model_id,
|
||||
quant,
|
||||
reply,
|
||||
} => {
|
||||
let result =
|
||||
load_dense_inner(&state.device, &config_path, &safetensors_paths, &model_id)
|
||||
.map(|arch| insert_arch(&mut state, Box::new(arch)));
|
||||
let result = load_dense_inner(
|
||||
&state.device,
|
||||
&config_path,
|
||||
&safetensors_paths,
|
||||
&model_id,
|
||||
quant,
|
||||
)
|
||||
.map(|arch| {
|
||||
// Ask the model, once, while it is still in
|
||||
// hand. See `DenseLoad`.
|
||||
let supports_kv_snapshot = arch.supports_kv_snapshot();
|
||||
let handle = insert_arch(&mut state, Box::new(arch));
|
||||
DenseLoad {
|
||||
handle,
|
||||
supports_kv_snapshot,
|
||||
}
|
||||
});
|
||||
// As for `LoadGguf`. It matters more here: a dense
|
||||
// load that runs out of VRAM part-way has typically
|
||||
// allocated tens of gigabytes first, and the evicted
|
||||
// model waiting to be restored needs them back.
|
||||
trim_device_pool(&state);
|
||||
let _ = reply.send(result);
|
||||
}
|
||||
Job::DropArch { handle, reply } => {
|
||||
@@ -158,6 +187,14 @@ pub(crate) fn run(device_index: u32, rx: Receiver<Job>, poisoned: Arc<AtomicBool
|
||||
// the end of the arm; calling drop() explicitly just
|
||||
// makes the intent visible.
|
||||
drop(removed);
|
||||
// Return the freed blocks to the system rather than
|
||||
// the stream-ordered pool, so the next load actually
|
||||
// sees the VRAM — `DropImage` has done this since the
|
||||
// benjy cold-swap, and an unload is an unload whatever
|
||||
// modality asked for it.
|
||||
if was_present {
|
||||
trim_device_pool(&state);
|
||||
}
|
||||
tracing::debug!(
|
||||
device_index,
|
||||
handle = handle.0,
|
||||
@@ -374,6 +411,13 @@ pub(crate) fn run(device_index: u32, rx: Receiver<Job>, poisoned: Arc<AtomicBool
|
||||
let was_present = removed.is_some();
|
||||
drop(removed);
|
||||
state.tp_kv_snapshots.retain(|(h, _), _| *h != handle);
|
||||
// As `DropArch`: this is only the leader's shard —
|
||||
// the other ranks free theirs in their own processes
|
||||
// — but the leader's is the one sharing a card with
|
||||
// whatever loads next.
|
||||
if was_present {
|
||||
trim_device_pool(&state);
|
||||
}
|
||||
tracing::debug!(
|
||||
device_index,
|
||||
tp_handle = handle.0,
|
||||
@@ -895,6 +939,7 @@ fn load_dense_inner(
|
||||
config_path: &std::path::Path,
|
||||
safetensors_paths: &[std::path::PathBuf],
|
||||
model_id: &str,
|
||||
quant: Option<candle_core::quantized::GgmlDType>,
|
||||
) -> anyhow::Result<ModelArch> {
|
||||
use anyhow::Context;
|
||||
use candle_core::DType;
|
||||
@@ -988,6 +1033,36 @@ fn load_dense_inner(
|
||||
.context("build Qwen3-Next dense model")?;
|
||||
Ok(ModelArch::Qwen3_5Dense(model))
|
||||
}
|
||||
"qwen4_exp" => {
|
||||
// Qwen3.8-Flash-Next. Mirrors the CPU path in
|
||||
// `candle::load_arch_dense` — same config type, same
|
||||
// sharded backend, same in-situ quantisation.
|
||||
//
|
||||
// ISQ is not optional for this architecture: the routed
|
||||
// experts are 241.6 GB at BF16 (measured from the
|
||||
// checkpoint's own headers), against 32.6 GB of card. A
|
||||
// load without a `quant` will fail on memory rather than
|
||||
// on anything this code can say (#315, #252).
|
||||
let cfg = crate::harness::arch::qwen4_exp::config::Config::from_config_json(&cfg_text)
|
||||
.context("parse qwen4_exp config.json")?;
|
||||
let sharded_vb = unsafe {
|
||||
candle_nn::var_builder::ShardedSafeTensors::var_builder(
|
||||
safetensors_paths,
|
||||
dtype,
|
||||
device,
|
||||
)
|
||||
.context("build ShardedVarBuilder for qwen4_exp")?
|
||||
};
|
||||
let model = crate::harness::arch::qwen4_exp::model::Qwen4ExpForCausalLM::load(
|
||||
&cfg,
|
||||
dtype,
|
||||
quant,
|
||||
device,
|
||||
&sharded_vb,
|
||||
)
|
||||
.context("build qwen4_exp model")?;
|
||||
Ok(ModelArch::Qwen4Exp(Box::new(model)))
|
||||
}
|
||||
other => anyhow::bail!(
|
||||
"unrouted supported model_type '{other}' — \
|
||||
DENSE_SUPPORTED_MODEL_TYPES and load_dense_inner \
|
||||
@@ -1045,7 +1120,7 @@ fn tp_load_shard_inner(
|
||||
"qwen3_5" | "qwen3_next" => {
|
||||
let cfg = crate::harness::tp::tp_qwen3_5::Config::from_config_json(config_json)
|
||||
.context("parse Qwen3-Next Config JSON for leader load")?;
|
||||
let quant_dtype = crate::harness::tp::worker::parse_quant_string(quant)?;
|
||||
let quant_dtype = crate::harness::quant::parse_quant_string(quant)?;
|
||||
TpLeaderModel::Qwen3_5(crate::harness::tp::tp_qwen3_5::TpQwen3_5ForCausalLM::load(
|
||||
cfg,
|
||||
&vb,
|
||||
@@ -1179,7 +1254,7 @@ fn tp_assemble_kv_batch(
|
||||
})?;
|
||||
pairs.push((snap, *len));
|
||||
}
|
||||
let batch = crate::harness::arch::qwen3_5::snapshot::assemble_batch(&pairs)?;
|
||||
let batch = crate::harness::arch::snapshot::assemble_batch(&pairs)?;
|
||||
let model = tp_models
|
||||
.get_mut(&handle)
|
||||
.ok_or_else(|| anyhow::anyhow!("TpAssembleKvBatch: no model for handle {}", handle.0))?;
|
||||
@@ -1212,9 +1287,8 @@ fn tp_extract_kv_rows(
|
||||
.snapshot_kv_cache()?;
|
||||
let mut total = 0u64;
|
||||
for (&(row, prefix_len), &id) in rows.iter().zip(snapshot_ids) {
|
||||
let snap = crate::harness::arch::qwen3_5::snapshot::extract_row(
|
||||
&live, row, prefix_len, padded_len, steps,
|
||||
)?;
|
||||
let snap =
|
||||
crate::harness::arch::snapshot::extract_row(&live, row, prefix_len, padded_len, steps)?;
|
||||
total += snap.size_bytes();
|
||||
state.tp_kv_snapshots.insert((handle, id), snap);
|
||||
}
|
||||
@@ -1337,7 +1411,7 @@ fn assemble_kv_batch(
|
||||
})?;
|
||||
pairs.push((snap, *len));
|
||||
}
|
||||
let batch = crate::harness::arch::qwen3_5::snapshot::assemble_batch(&pairs)?;
|
||||
let batch = crate::harness::arch::snapshot::assemble_batch(&pairs)?;
|
||||
let arch = models
|
||||
.get_mut(&handle)
|
||||
.ok_or_else(|| anyhow::anyhow!("AssembleKvBatch: no model for handle {}", handle.0))?;
|
||||
@@ -1363,9 +1437,8 @@ fn extract_kv_rows(
|
||||
.snapshot_kv_cache()?;
|
||||
let mut out = Vec::with_capacity(rows.len());
|
||||
for &(row, prefix_len) in rows {
|
||||
let snap = crate::harness::arch::qwen3_5::snapshot::extract_row(
|
||||
&live, row, prefix_len, padded_len, steps,
|
||||
)?;
|
||||
let snap =
|
||||
crate::harness::arch::snapshot::extract_row(&live, row, prefix_len, padded_len, steps)?;
|
||||
let id = KvSnapshotId(state.next_kv_snapshot_id);
|
||||
state.next_kv_snapshot_id = state.next_kv_snapshot_id.wrapping_add(1);
|
||||
let bytes = snap.size_bytes();
|
||||
@@ -1671,3 +1744,42 @@ fn drain_poisoned(job: Job, device_index: u32) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Every architecture the preflight admits must have a loader arm
|
||||
/// on **this** path, not only on the CPU one in
|
||||
/// `candle::load_arch_dense`.
|
||||
///
|
||||
/// There are two dense loaders — the worker's (every CUDA load)
|
||||
/// and candle's (the CPU fallback) — and one list gating both.
|
||||
/// `qwen4_exp` was added to the list and to the CPU loader, so it
|
||||
/// passed preflight and then hit the "unrouted supported
|
||||
/// model_type" bail here: on a GPU host the model was
|
||||
/// unloadable, in about 200 ms, having read nothing. The bail's
|
||||
/// own comment said the two must stay in sync; nothing checked.
|
||||
///
|
||||
/// A bare `{"model_type": ...}` reaches the arm and then fails on
|
||||
/// the typed parse, which is fine — this asserts only that an arm
|
||||
/// exists to fail in.
|
||||
#[test]
|
||||
fn every_supported_dense_model_type_has_a_loader_arm() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
for model_type in crate::harness::candle::DENSE_SUPPORTED_MODEL_TYPES {
|
||||
let path = dir.path().join(format!("{model_type}.json"));
|
||||
std::fs::write(&path, format!(r#"{{"model_type": "{model_type}"}}"#)).unwrap();
|
||||
let msg =
|
||||
match load_dense_inner(&candle_core::Device::Cpu, &path, &[], model_type, None) {
|
||||
Ok(_) => panic!("a bare config must not build a model"),
|
||||
Err(e) => format!("{e:#}"),
|
||||
};
|
||||
assert!(
|
||||
!msg.contains("unrouted"),
|
||||
"`{model_type}` is in DENSE_SUPPORTED_MODEL_TYPES but load_dense_inner \
|
||||
has no arm for it: {msg}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -19,6 +19,27 @@ use tokio::sync::oneshot;
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||
pub struct ArchHandle(pub u64);
|
||||
|
||||
/// What a dense load hands back: the handle, plus what the arch it
|
||||
/// built says about itself.
|
||||
///
|
||||
/// The async side cannot ask the model anything — it lives in the
|
||||
/// worker's slab and never crosses the channel — so anything the
|
||||
/// serving path needs to know has to ride back on the reply. Prefix
|
||||
/// snapshot support (#11) is the first such fact, and it is here
|
||||
/// rather than re-derived from `config.json` because that is exactly
|
||||
/// how it went wrong: the async side matched a list of `model_type`
|
||||
/// strings that `ModelArch::supports_kv_snapshot` had since grown
|
||||
/// past, so `qwen4_exp` loaded with batched decode silently off.
|
||||
/// A capability the arch answers for itself cannot drift from the
|
||||
/// arch.
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct DenseLoad {
|
||||
pub handle: ArchHandle,
|
||||
/// [`crate::harness::candle::ModelArch::supports_kv_snapshot`], as
|
||||
/// answered by the model that was actually built.
|
||||
pub supports_kv_snapshot: bool,
|
||||
}
|
||||
|
||||
/// Opaque handle to a `TpLeaderModel` stored in the worker thread's
|
||||
/// state slab. Same shape as [`ArchHandle`] but in a separate
|
||||
/// namespace so the two slabs can coexist without ambiguity. Phase 3
|
||||
@@ -107,7 +128,11 @@ pub enum Job {
|
||||
config_path: PathBuf,
|
||||
safetensors_paths: Vec<PathBuf>,
|
||||
model_id: String,
|
||||
reply: oneshot::Sender<Result<ArchHandle>>,
|
||||
/// In-situ quantisation (#320): the catalogue's `quant`,
|
||||
/// already parsed. `None` loads at the checkpoint dtype.
|
||||
/// Archs that cannot quantise in situ ignore it.
|
||||
quant: Option<candle_core::quantized::GgmlDType>,
|
||||
reply: oneshot::Sender<Result<DenseLoad>>,
|
||||
},
|
||||
/// Remove the model from the slab and drop it. The `Drop` runs on
|
||||
/// the worker thread so CUDA tensors release their memory on the
|
||||
@@ -163,7 +188,7 @@ pub enum Job {
|
||||
/// state and install it as the model's live state (#98). `seqs`
|
||||
/// pairs each snapshot id with its true token length; attention
|
||||
/// K/V is right-padded to the batch max and `cat`ed on dim 0 (see
|
||||
/// `arch::qwen3_5::snapshot::assemble_batch`). Replies with the
|
||||
/// `arch::snapshot::assemble_batch`). Replies with the
|
||||
/// padded uniform KV length. The source snapshots remain stored —
|
||||
/// the caller drops them via `DropKvSnapshot` when the sequences
|
||||
/// leave the batch.
|
||||
|
||||
@@ -51,7 +51,7 @@ use tokio::sync::oneshot;
|
||||
|
||||
#[cfg(feature = "cuda")]
|
||||
pub use jobs::TpHandle;
|
||||
pub use jobs::{ArchHandle, ImageHandle, Job, KvSnapshotId};
|
||||
pub use jobs::{ArchHandle, DenseLoad, ImageHandle, Job, KvSnapshotId};
|
||||
|
||||
/// Errors returned by `DeviceWorkerHandle` submit methods.
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
@@ -221,7 +221,8 @@ impl DeviceWorkerHandle {
|
||||
config_path: std::path::PathBuf,
|
||||
safetensors_paths: Vec<std::path::PathBuf>,
|
||||
model_id: String,
|
||||
) -> Result<ArchHandle, WorkerError> {
|
||||
quant: Option<candle_core::quantized::GgmlDType>,
|
||||
) -> Result<DenseLoad, WorkerError> {
|
||||
if self.poisoned.load(Ordering::Acquire) {
|
||||
return Err(WorkerError::Poisoned {
|
||||
device_index: self.device_index,
|
||||
@@ -233,6 +234,7 @@ impl DeviceWorkerHandle {
|
||||
config_path,
|
||||
safetensors_paths,
|
||||
model_id,
|
||||
quant,
|
||||
reply: reply_tx,
|
||||
})
|
||||
.map_err(|_| WorkerError::Gone {
|
||||
@@ -1243,9 +1245,11 @@ mod tests {
|
||||
fixture.join("config.json"),
|
||||
vec![fixture.join("model.safetensors")],
|
||||
"qwen3_next-tiny".into(),
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.expect("load tiny fixture");
|
||||
.expect("load tiny fixture")
|
||||
.handle;
|
||||
|
||||
let prompts: [&[u32]; 3] = [&[1, 2, 3], &[4, 5], &[7, 3, 2, 5, 6]];
|
||||
let steps: [&[u32]; 3] = [&[11, 12, 13], &[9, 8, 7], &[21, 22, 23]];
|
||||
|
||||
@@ -1261,9 +1261,11 @@ mod tests {
|
||||
fixture.join("config.json"),
|
||||
vec![fixture.join("model.safetensors")],
|
||||
"qwen3_next-tiny".into(),
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.expect("load fixture");
|
||||
.expect("load fixture")
|
||||
.handle;
|
||||
|
||||
let admission_cfg = AdmissionConfig {
|
||||
max_in_flight: 3,
|
||||
|
||||
@@ -12,6 +12,7 @@ pub mod image;
|
||||
pub mod prefix_cache;
|
||||
pub mod preflight;
|
||||
pub mod preprocess;
|
||||
pub(crate) mod quant;
|
||||
pub mod reasoning_budget;
|
||||
pub mod reasoning_effort;
|
||||
pub mod sampling;
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
//! Catalogue quant names to ggml dtypes.
|
||||
//!
|
||||
//! Shared rather than owned by the TP path: `quant = "q4k"` in
|
||||
//! `models.toml` means the same thing whichever loader reads it. It sat
|
||||
//! behind `cfg(feature = "cuda")` while only the TP worker used it,
|
||||
//! which put it out of reach of the single-GPU path — where, for
|
||||
//! `qwen4_exp`, in-situ quantisation is the difference between a model
|
||||
//! that loads and 241.6 GB that does not (#315).
|
||||
|
||||
/// Parse a `ModelSpec.quant` string into a `GgmlDType`. Accepts the
|
||||
/// common ggml format names (case-insensitive). `None` and `Some("")`
|
||||
/// both map to "no quantization".
|
||||
///
|
||||
/// Supported: `q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q8_1`,
|
||||
/// `q2k`/`q2_k`, `q3k`/`q3_k`, `q4k`/`q4_k`, `q5k`/`q5_k`,
|
||||
/// `q6k`/`q6_k`, `q8k`/`q8_k`, `f16`, `bf16`, `f32`. The underscore
|
||||
/// is optional and the prefix is case-insensitive.
|
||||
pub(crate) fn parse_quant_string(
|
||||
s: Option<&str>,
|
||||
) -> anyhow::Result<Option<candle_core::quantized::GgmlDType>> {
|
||||
use candle_core::quantized::GgmlDType;
|
||||
let s = match s {
|
||||
Some(s) if !s.is_empty() => s,
|
||||
_ => return Ok(None),
|
||||
};
|
||||
let normalised = s.to_ascii_lowercase().replace('_', "");
|
||||
let dtype = match normalised.as_str() {
|
||||
"q40" => GgmlDType::Q4_0,
|
||||
"q41" => GgmlDType::Q4_1,
|
||||
"q50" => GgmlDType::Q5_0,
|
||||
"q51" => GgmlDType::Q5_1,
|
||||
"q80" => GgmlDType::Q8_0,
|
||||
"q81" => GgmlDType::Q8_1,
|
||||
"q2k" => GgmlDType::Q2K,
|
||||
"q3k" => GgmlDType::Q3K,
|
||||
"q4k" | "q4km" => GgmlDType::Q4K,
|
||||
"q5k" | "q5km" => GgmlDType::Q5K,
|
||||
"q6k" => GgmlDType::Q6K,
|
||||
"q8k" => GgmlDType::Q8K,
|
||||
"f16" => GgmlDType::F16,
|
||||
"bf16" => GgmlDType::BF16,
|
||||
"f32" => GgmlDType::F32,
|
||||
other => anyhow::bail!(
|
||||
"unknown quant '{other}' (expected one of: q4_0, q4_1, q5_0, q5_1, q8_0, \
|
||||
q8_1, q2k, q3k, q4k, q5k, q6k, q8k, f16, bf16, f32)"
|
||||
),
|
||||
};
|
||||
Ok(Some(dtype))
|
||||
}
|
||||
@@ -138,7 +138,7 @@ impl TpLeaderModel {
|
||||
/// Capture rank 0's cache state for a prefix snapshot (#11).
|
||||
pub fn snapshot_kv_cache(
|
||||
&self,
|
||||
) -> candle_core::Result<crate::harness::arch::qwen3_5::snapshot::KvCacheSnapshot> {
|
||||
) -> candle_core::Result<crate::harness::arch::snapshot::KvCacheSnapshot> {
|
||||
match self {
|
||||
TpLeaderModel::Qwen3(_) => {
|
||||
candle_core::bail!("snapshot_kv_cache: qwen3 (dense) has no snapshot support")
|
||||
@@ -150,7 +150,7 @@ impl TpLeaderModel {
|
||||
/// Replace rank 0's live cache state with a stored snapshot.
|
||||
pub fn restore_kv_cache(
|
||||
&mut self,
|
||||
snap: &crate::harness::arch::qwen3_5::snapshot::KvCacheSnapshot,
|
||||
snap: &crate::harness::arch::snapshot::KvCacheSnapshot,
|
||||
) -> candle_core::Result<()> {
|
||||
match self {
|
||||
TpLeaderModel::Qwen3(_) => {
|
||||
|
||||
@@ -45,10 +45,10 @@ use super::tp_linear::{ColumnParallelLinear, RowParallelLinear};
|
||||
use crate::harness::arch::qwen3_5::linear_attn::repeat_interleave;
|
||||
use crate::harness::arch::qwen3_5::rmsnorm::{Qwen3_5RmsNorm, Qwen3_5RmsNormGated, l2norm};
|
||||
use crate::harness::arch::qwen3_5::rope::RotaryEmbedding;
|
||||
use crate::harness::arch::qwen3_5::snapshot::{KvCacheSnapshot, LayerKvSnapshot};
|
||||
use crate::harness::arch::qwen3_5::splice_runs;
|
||||
use crate::harness::arch::qwen3_5::vision::VisionTower;
|
||||
pub use crate::harness::arch::qwen3_5::{Config, TextConfig};
|
||||
use crate::harness::arch::snapshot::{KvCacheSnapshot, LayerKvSnapshot};
|
||||
|
||||
// ─── linear-attention (Gated DeltaNet) ──────────────────────────────
|
||||
|
||||
@@ -1700,6 +1700,7 @@ impl TpQwen3_5Model {
|
||||
Ok(KvCacheSnapshot {
|
||||
layers,
|
||||
rope_delta: self.rope_delta,
|
||||
ple: None,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -16,6 +16,11 @@ use anyhow::Result;
|
||||
use std::collections::HashMap;
|
||||
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
|
||||
|
||||
// Only the CUDA shard-load path calls this; the CPU build of this
|
||||
// module has no caller.
|
||||
#[cfg(feature = "cuda")]
|
||||
use crate::harness::quant::parse_quant_string;
|
||||
|
||||
use super::nccl_state::NcclState;
|
||||
use super::rpc::{WorkerRequest, WorkerResponse};
|
||||
|
||||
@@ -119,7 +124,7 @@ impl WorkerModel {
|
||||
/// `TpLeaderModel` support first.
|
||||
fn snapshot_kv_cache(
|
||||
&self,
|
||||
) -> candle_core::Result<crate::harness::arch::qwen3_5::snapshot::KvCacheSnapshot> {
|
||||
) -> candle_core::Result<crate::harness::arch::snapshot::KvCacheSnapshot> {
|
||||
match self {
|
||||
WorkerModel::Qwen3(_) => {
|
||||
candle_core::bail!("snapshot_kv_cache: qwen3 (dense) has no snapshot support")
|
||||
@@ -130,7 +135,7 @@ impl WorkerModel {
|
||||
|
||||
fn restore_kv_cache(
|
||||
&mut self,
|
||||
snap: &crate::harness::arch::qwen3_5::snapshot::KvCacheSnapshot,
|
||||
snap: &crate::harness::arch::snapshot::KvCacheSnapshot,
|
||||
) -> candle_core::Result<()> {
|
||||
match self {
|
||||
WorkerModel::Qwen3(_) => {
|
||||
@@ -227,7 +232,7 @@ struct WorkerState {
|
||||
/// pool and shared across all ranks. Dropped with the shard on
|
||||
/// `UnloadModel`.
|
||||
#[cfg(feature = "cuda")]
|
||||
kv_snapshots: HashMap<(String, u64), crate::harness::arch::qwen3_5::snapshot::KvCacheSnapshot>,
|
||||
kv_snapshots: HashMap<(String, u64), crate::harness::arch::snapshot::KvCacheSnapshot>,
|
||||
/// Placeholder mirroring `models` on the non-cuda build.
|
||||
#[cfg(not(feature = "cuda"))]
|
||||
#[allow(dead_code)]
|
||||
@@ -689,11 +694,10 @@ impl WorkerState {
|
||||
};
|
||||
pairs.push((snap, *len));
|
||||
}
|
||||
let result =
|
||||
crate::harness::arch::qwen3_5::snapshot::assemble_batch(&pairs).and_then(|batch| {
|
||||
model.restore_kv_cache(&batch.snapshot)?;
|
||||
Ok(batch.padded_len)
|
||||
});
|
||||
let result = crate::harness::arch::snapshot::assemble_batch(&pairs).and_then(|batch| {
|
||||
model.restore_kv_cache(&batch.snapshot)?;
|
||||
Ok(batch.padded_len)
|
||||
});
|
||||
match result {
|
||||
Ok(padded_len) => WorkerResponse::KvBatchAssembled {
|
||||
padded_len: padded_len as u64,
|
||||
@@ -756,7 +760,7 @@ impl WorkerState {
|
||||
};
|
||||
let mut total_bytes = 0u64;
|
||||
for (&(row, prefix_len), &id) in rows.iter().zip(snapshot_ids) {
|
||||
match crate::harness::arch::qwen3_5::snapshot::extract_row(
|
||||
match crate::harness::arch::snapshot::extract_row(
|
||||
&live, row, prefix_len, padded_len, steps,
|
||||
) {
|
||||
Ok(snap) => {
|
||||
@@ -1036,45 +1040,3 @@ impl WorkerState {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse a `ModelSpec.quant` string into a `GgmlDType`. Accepts the
|
||||
/// common ggml format names (case-insensitive). `None` and `Some("")`
|
||||
/// both map to "no quantization".
|
||||
///
|
||||
/// Supported: `q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q8_1`,
|
||||
/// `q2k`/`q2_k`, `q3k`/`q3_k`, `q4k`/`q4_k`, `q5k`/`q5_k`,
|
||||
/// `q6k`/`q6_k`, `q8k`/`q8_k`, `f16`, `bf16`, `f32`. The underscore
|
||||
/// is optional and the prefix is case-insensitive.
|
||||
#[cfg(feature = "cuda")]
|
||||
pub(crate) fn parse_quant_string(
|
||||
s: Option<&str>,
|
||||
) -> anyhow::Result<Option<candle_core::quantized::GgmlDType>> {
|
||||
use candle_core::quantized::GgmlDType;
|
||||
let s = match s {
|
||||
Some(s) if !s.is_empty() => s,
|
||||
_ => return Ok(None),
|
||||
};
|
||||
let normalised = s.to_ascii_lowercase().replace('_', "");
|
||||
let dtype = match normalised.as_str() {
|
||||
"q40" => GgmlDType::Q4_0,
|
||||
"q41" => GgmlDType::Q4_1,
|
||||
"q50" => GgmlDType::Q5_0,
|
||||
"q51" => GgmlDType::Q5_1,
|
||||
"q80" => GgmlDType::Q8_0,
|
||||
"q81" => GgmlDType::Q8_1,
|
||||
"q2k" => GgmlDType::Q2K,
|
||||
"q3k" => GgmlDType::Q3K,
|
||||
"q4k" | "q4km" => GgmlDType::Q4K,
|
||||
"q5k" | "q5km" => GgmlDType::Q5K,
|
||||
"q6k" => GgmlDType::Q6K,
|
||||
"q8k" => GgmlDType::Q8K,
|
||||
"f16" => GgmlDType::F16,
|
||||
"bf16" => GgmlDType::BF16,
|
||||
"f32" => GgmlDType::F32,
|
||||
other => anyhow::bail!(
|
||||
"unknown quant '{other}' (expected one of: q4_0, q4_1, q5_0, q5_1, q8_0, \
|
||||
q8_1, q2k, q3k, q4k, q5k, q6k, q8k, f16, bf16, f32)"
|
||||
),
|
||||
};
|
||||
Ok(Some(dtype))
|
||||
}
|
||||
|
||||
+50
-13
@@ -470,11 +470,21 @@ conventions — this is a solved problem in our tree.
|
||||
|
||||
**What is new is `group_size`.** `Qwen4ExpTextRMSNorm(dim, group_size)`
|
||||
reshapes the last axis to `(-1, group_size)`, normalises each group
|
||||
independently, and flattens back. Every 10240-wide norm in this model
|
||||
(`hc_norm`, `norm_key`, `norm_query`, `norm_conv`,
|
||||
`mtp.pre_fc_norm_hidden`) uses `group_size = 2560`, i.e. four
|
||||
independent RMS normalisations, not one over 10240. A single norm over
|
||||
the full 10240 vector compiles, runs, and is wrong.
|
||||
independently, and flattens back. The 10240-wide norms in the decoder
|
||||
(`hc_norm`, `norm_key`, `norm_query`, `norm_conv`) use
|
||||
`group_size = 2560`, i.e. four independent RMS normalisations, not one
|
||||
over 10240. A single norm over the full 10240 vector compiles, runs,
|
||||
and is wrong.
|
||||
|
||||
**`mtp.pre_fc_norm_hidden` is the exception, and this document had it
|
||||
wrong.** It is 10240 wide and **ungrouped** — one normalisation over
|
||||
the whole vector. vLLM distinguishes the two cases explicitly: its
|
||||
hyper-connection uses `GroupedGemmaRMSNorm` with
|
||||
`group_size = hidden_size`, and the MTP head uses a plain
|
||||
`GemmaRMSNorm(hidden_size * hc_count)`. Both classes exist in that
|
||||
tree, so the choice is deliberate rather than an oversight. Grouping it
|
||||
would rescale the draft head's input in four independent pieces and
|
||||
show up as a poor acceptance rate, not as an error. See §9.
|
||||
|
||||
Variance is computed in f32 and the `(1 + w)` shift is applied in f32
|
||||
before the multiply — same reasoning as the note already in
|
||||
@@ -501,14 +511,41 @@ mtp.layers.0.* # a complete full_attention
|
||||
|
||||
**Transformers does not implement the MTP head.**
|
||||
`Qwen4ExpPreTrainedModel._keys_to_ignore_on_load_unexpected = [r"^mtp.*"]`
|
||||
— the weights are dropped on load. So for #313 the upstream reference is
|
||||
*not* usable, and the authorities are instead llama.cpp PR #27836
|
||||
(`--spec-type draft-mtp`) and vLLM's `speculative-config
|
||||
{"method": "mtp"}` path. The tensor names above and the two `fc_*`
|
||||
projections make the intended wiring clear (project the next token's
|
||||
embedding and the accepted hidden state, sum, run one decoder layer,
|
||||
reuse `lm_head`), but the accept rule and the proposal depth must come
|
||||
from one of those two implementations, not from this document.
|
||||
— the weights are dropped on load.
|
||||
|
||||
**The wiring, settled against vLLM** (`vllm/models/qwen4_exp/nvidia/
|
||||
mtp.py`, read 2026-09-02). vLLM calls this shape
|
||||
`residual_linear_shared`, as against the `Linear(2H, H)` + repeat that
|
||||
other MTP variants use:
|
||||
|
||||
```
|
||||
e = fc_embedding(pre_fc_norm_embedding(embed(t))) # [T, 2560]
|
||||
h = pre_fc_norm_hidden(h.flatten(-2)) # ONE norm over 10240
|
||||
h = fc_hidden(h.view(T, hc, 2560)) # per stream, shared matrix
|
||||
h = e.unsqueeze(-2) + h # embedding broadcast to all four
|
||||
h = h.flatten(-2) # [T, 10240] -> the decoder layer
|
||||
```
|
||||
|
||||
Three things here are not guessable from the tensor shapes:
|
||||
|
||||
1. **`pre_fc_norm_hidden` is applied to the flattened 10240 and is
|
||||
ungrouped** — see §8, which this document previously got wrong.
|
||||
2. **The hidden state is the *pre-final-mixer* multi-stream**, i.e. `h`
|
||||
after the last decoder layer and *before* `hyper_connection_mixer`
|
||||
collapses it. Not the 2560 the LM head sees. vLLM calls this
|
||||
"scheme A"; on later draft steps the head consumes its own previous
|
||||
multi-stream rather than the target's.
|
||||
3. **The head emits two things per step**: the collapsed `[T, 2560]`
|
||||
for `lm_head`, and the pre-final-mixer `[T, 10240]` for the next
|
||||
draft step.
|
||||
|
||||
PLE is forced off in the draft layer while the HC stream count is kept
|
||||
(`ngram_context=None`), which is consistent with `mtp.layers.0` shipping
|
||||
no `ple.*` tensors.
|
||||
|
||||
The accept rule is standard (greedy prefix match, or the Leviathan
|
||||
rejection scheme) and is not architecture-specific; this document
|
||||
previously conflated it with the wiring above.
|
||||
|
||||
`mtp.layers.0` carries its own full 512-expert MoE — 2.61 B params,
|
||||
5.2 GB at BF16. It is not free to keep resident and its placement is
|
||||
|
||||
Reference in New Issue
Block a user