rob thijssen
9bf13f09dd
All checks were successful
CI / Format (push) Successful in 8s
CI / CUDA type-check (push) Successful in 1m38s
CI / Clippy (push) Successful in 2m17s
CI / Test (push) Successful in 5m17s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
CI / Format (pull_request) Successful in 7s
CI / CUDA type-check (pull_request) Successful in 1m38s
CI / Clippy (pull_request) Successful in 2m15s
CI / Test (pull_request) Successful in 6m54s
CI / Build cortex SRPM (pull_request) Has been skipped
CI / Build neuron SRPM (pull_request) Has been skipped
CI / Publish cortex to COPR (pull_request) Has been skipped
CI / Publish neuron to COPR (pull_request) Has been skipped
CI / Bump version in source (pull_request) Has been skipped
F1 slice 2 — the MoE block itself, CPU/single-GPU:
- arch/qwen3_5/moe.rs: Qwen3_5MoeBlock — top-k router (upstream
softmax-then-topk order, renorm iff norm_topk_prob), per-expert
SwiGLU (reusing Qwen3_5MLP at moe_intermediate_size), and the
always-on shared expert mixed via sigmoid(shared_expert_gate).
Correctness-first host-side scatter dispatch; the fused grouped-GEMM
path is slice 4 behind the same forward signature.
- decoder.rs: MlpKind { Dense, Moe } dispatch on layer_uses_moe,
mirroring AttentionKind.
- linear_attn.rs: fused-checkpoint support — qwen3_next stores
in_proj_qkvz / in_proj_ba interleaved per key-head group (upstream
fix_query_key_value_ordering layout); split_fused_qkvz/ba
de-interleave once at load into the contiguous [Q|K|V] + Z / B + A
layout the forward path (incl. the conv channels) already uses.
Auto-detected via contains_tensor, so Qwen3.6 checkpoints are
untouched.
- mod.rs: text_weight_prefix() — qwen3_next checkpoints put the text
core at `model.*`, Qwen3.6 at `model.language_model.*`; the slice-1
single-GPU MoE guard is removed (TP guard stays until slice 3).
Validation:
- qwen3_next_parity integration test replays a committed tiny
random-weight HF Qwen3NextForCausalLM checkpoint (generated by
script/dump_qwen3_next_tiny.py on beast: transformers 5.9.0,
torch 2.9.1) through neuron's full load path: max_abs 0.000000,
cosine 1.00000000 at f32 — exact parity, pinning the config
normalisation, weight prefix, qkvz/ba de-interleave, hybrid layer
interleaving, and the whole MoE block against upstream.
- Unit tests: scatter forward vs per-token dense reference,
no-shared-expert/no-renorm behaviour, fused-split round-trip, and a
flat-layout end-to-end structural load.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TczcGF7JSjJs8r15RSSGpx