Files
helexa/crates/neuron/tests/fixtures/numerical/README.md
T
grenadeandClaude Opus 5 b3ed41c744
CI / Config consistency (push) Successful in 11s
CI / Classify changes (push) Successful in 23s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / CUDA type-check (push) Successful in 19m34s
CI / Clippy (push) Successful in 2m40s
CI / Test (push) Successful in 6m45s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
test(qwen4_exp): a second parity case, for the paths the first cannot reach
Two fixtures rather than one size of the same test. Each reaches
something the other structurally cannot, and a mutation shows it rather
than the commit message asserting it.

**80 tokens** puts the GatedDeltaNet layers on the chunked prefill
algorithm (#23), which engages at 64 — a different factorisation of the
same recurrence, not the same code with more input. The short case only
ever exercised the recurrent step.

**An eos at position 37** puts a segment boundary inside the n-gram
window, which `shift_right_ignore_eos` must refuse to read across. That
logic is a cummax over eos positions per row, intricate enough to be
worth comparing rather than reasoning about. Making it ignore segments
entirely leaves the short fixture at 0.000002 — it has no eos, so there
is nothing to get wrong — and moves the long one by 0.67. The short
case has no eos deliberately: one that happened to contain an eos would
be testing segment handling by accident and would make the two
redundant.

Both agree with the reference: 2e-6 short, 1.3e-5 long, cosine
1.000000 and matching argmax on each.

The bound is now per case, 1e-5 and 1e-4, with the reason at each call
site. The long case is looser because the chunked factorisation rounds
differently in f32 by construction — a stated reason, not a number
raised until the test passed. It still sits four orders of magnitude
below what the mutations produce, so it discriminates exactly as well:
gate/up swapped moves this fixture by 2.7.

One incidental difference, not a defect: our rotary table is sized by
`max_position_embeddings` and a prompt past it fails inside `narrow`,
where the reference computes cos/sin on the fly and extrapolates. It
cannot arise in production, since `derive_limit` advertises no more
context than the model declares — but it is why the fixture config
declares 128 rather than 64.

Refs #307, #312, #323.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 15:28:23 +03:00

6.9 KiB
Raw Blame History

Numerical-reference fixtures (#15)

Reference tensors captured from the HF transformers implementation by script/dump_reference.py, replayed and compared by tests/numerical_reference.rs. These pin the README's "implemented in this repository, ported against the HuggingFace reference" claim to checked-in numbers.

fixture model case dtype compared by
qwen3_5-0.8b-text Qwen/Qwen3.5-0.8B text (>64-token prompt → chunked GDN prefill) f32 text_logits_match_reference
qwen3_5-0.8b-vision Qwen/Qwen3.5-0.8B 448×448 synthetic image + prompt f32 vision_tower_and_logits_match_reference
qwen3_6-27b-text Qwen/Qwen3.6-27B text bf16 manual (see below)
qwen4_exp-tiny random weights, real architecture text, 12 tokens, no eos f32 qwen4_exp_logits_match_reference
qwen4_exp-tiny-long random weights, real architecture text, 80 tokens, eos at 37 f32 qwen4_exp_long_prompt_logits_match_reference

qwen4_exp-tiny is different, and runs in CI

The others replay a real snapshot and self-skip without NEURON_REF_MODEL_PATH. This one is 44 KB of random weights in the real architecture — 4 layers (3 linear-attention, 1 QSA), PLE on layer 1, 2 experts — generated by transformers itself via script/generate_qwen4_exp_fixture.py. So it is checked in, needs no weights, and runs on every push.

That matters because the whole qwen4_exp architecture was ported from a written spec (#308) and tested against our own reading of it. This is the only thing in the tree that compares it to the implementation it was ported from (#323).

Two properties of the fixture are load-bearing, and both were learned by getting them wrong:

  • Weights are uniform(-0.8, 0.8), not uniform(-0.1, 0.1). Near the origin silu(x) ~ x/2, so the SwiGLU is nearly symmetric and reading the fused expert's gate and up halves the wrong way round moved the logits by 2.4e-4 — inside any tolerance worth writing. At the larger scale the same mutation moves them by 2.7.
  • The generator refuses to emit a fixture containing a QSA selection tie. A block scores exactly 0.0 whenever relu clamps every head, so ties are common with random weights; when the k-th and (k+1)-th scores are equal, which block is dropped is decided by topk's ordering of equal elements, which torch does not specify and whose CPU and CUDA kernels need not agree. Seeds 0, 1 and 4 produce ties; seed 2 (the default) does not. See "the tie-break" below.

Regenerate with the reference from transformers main — the 5.9.0 release does not carry qwen4_exp:

python -m venv --system-site-packages .venv
.venv/bin/pip install --no-deps "transformers @ git+https://github.com/huggingface/transformers@main"
.venv/bin/pip install "tokenizers>=0.23.1,<0.24" "safetensors>=0.8.0"
.venv/bin/python script/generate_qwen4_exp_fixture.py \
    --config crates/neuron/tests/fixtures/numerical/qwen4_exp-tiny/config.json \
    --out crates/neuron/tests/fixtures/numerical/qwen4_exp-tiny

Why there are two cases

They are not the same test at two sizes. Each reaches a path the other structurally cannot, and a mutation demonstrates it:

  • 80 tokens puts the GatedDeltaNet layers on the chunked prefill algorithm (#23), which engages at 64. That is a different factorisation of the same recurrence, not the same code with more input — which is also why the long case's bound is 1e-4 against the short case's 1e-5 (observed 1.3e-5 vs 2e-6). It also gives QSA up to 40 blocks against a budget of 4, so selection discards nearly everything instead of one block at the boundary.
  • The eos at position 37 is a segment boundary the n-gram window must refuse to read across. Making shift_right_ignore_eos ignore segments entirely leaves the short fixture at 0.000002 — it contains no eos, so there is nothing to get wrong — and moves the long one by 0.67.

The short case deliberately has no eos: a fixture that happens to contain one is testing segment handling by accident, and would make the two cases redundant.

The tie-break

Where the reference and this port genuinely differ: at an exact tie in the QSA block scores, topk keeps one block and we keep another. Ours is deterministic (lower block index wins); torch's is unspecified. A tie means every head's dot product was clamped to zero for both blocks — the indexer scored them equally worthless — so either choice is defensible, and there is no stable target to match. The fixture avoids the situation rather than pretending it is resolved.

Running the comparison

On a host with the model snapshot (beast):

NEURON_REF_MODEL_PATH=/archive3/llm-cache/models--Qwen--Qwen3.5-0.8B/snapshots/<rev> \
    cargo test -p neuron --test numerical_reference -- --nocapture

Without NEURON_REF_MODEL_PATH the tests compile and self-skip, so CI stays green without weights.

Why f32 fixtures

f32-vs-f32 isolates implementation differences: observed agreement is text max_abs 0.000 / cosine 1.000000, vision tower cosine 0.999998. Cross-dtype comparisons drown in bf16 rounding chaos through the 27-layer tower (global cosine ~0.997, worst patch ~0.92, worst index unstable across runs) — that is production-dtype noise, not implementation error. The mutation check: rerunning with NEURON_VISION_LEGACY_POS=1 (the deliberately-wrong sequential pos-embed lookup) collapses tower cosine to 0.75 / worst patch 0.28 and fails the test loudly.

The 27B fixture

qwen3_6-27b-text is captured in bf16 on CPU (an f32 27B forward needs ~108 GB; beast has 91 GB free). The automated tests run against the 0.8B because both models execute the same arch modules — the 27B differs only in hyperparameters — and an apples-to-apples 27B replay needs either TP=2 bf16 (idle GPUs, no neuron running) or a bigger-RAM host. Manual procedure when wanted: stop neuron on beast, replay the manifest's token ids through a TP=2 bf16 load, compare argmax + cosine against logits.f32 with bf16-calibrated tolerances.

Regenerating

Regenerate whenever the pinned snapshot or the transformers reference changes; record both versions (in each manifest.json) in the commit message:

# on beast; processor files may be missing from neuron's snapshot —
# point the processor at the repo id with a scratch cache
SNAP=$(ls -d /archive3/llm-cache/models--Qwen--Qwen3.5-0.8B/snapshots/*/ | head -1)
HF_HUB_CACHE=/tmp/hf-ref-cache python3 script/dump_reference.py \
    --model-path "$SNAP" --processor-path Qwen/Qwen3.5-0.8B \
    --case text   --out crates/neuron/tests/fixtures/numerical/qwen3_5-0.8b-text
HF_HUB_CACHE=/tmp/hf-ref-cache python3 script/dump_reference.py \
    --model-path "$SNAP" --processor-path Qwen/Qwen3.5-0.8B \
    --case vision --out crates/neuron/tests/fixtures/numerical/qwen3_5-0.8b-vision