Compare commits

..
Author SHA1 Message Date
grenadeandClaude Opus 5 e04091f37f fix(neuron): say what the isq cache actually did (#322)
CI / Config consistency (push) Successful in 10s
CI / Classify changes (push) Successful in 23s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 13s
CI / Clippy (push) Successful in 2m35s
CI / Test (push) Successful in 8m54s
CI / CUDA type-check (push) Successful in 19m11s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The journal reported only that the cache was enabled for a load. A
load that served every layer from disk and a load that served none of
them looked identical from outside, and the difference between them is
eight minutes. That is the shape of #321 — every light green, no way to
tell whether the thing worked — in code shipped an hour after filing
it.

So one line per load: layers served, layers quantised, bytes read,
bytes written.

It fires from a guard rather than the success path, because a load that
OOMs part way is exactly when the question "how much of that was
re-quantised?" is worth answering — and until #318 lands, every
qwen4_exp load on this hardware is that load.

Measured on beast, same window, same OOM at layer 15, so the only
variable is where the fifteen layers came from:

  quantising                     8m 18.9s
  cache on /archive3 (spinning)  1m 48.4s
  cache on /archive1 (NVMe)      0m 23.1s

The NVMe figure is 21.5x, and it is why the artifact directory is now
symlinked out of the spinning cache onto NVMe the way the hot model
weights already are — one symlink, no config, the mechanism the fleet
already had.

Refs #321, #322.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 19:14:40 +03:00
grenade d0e7f185b6 Merge branch 'feat/322-isq-cache'
build-prerelease / Resolve version stamps + change detection (push) Successful in 20s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m39s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 2m26s
build-prerelease / Build neuron-ada (push) Successful in 2m34s
build-prerelease / Test (push) Successful in 6m53s
build-prerelease / Build neuron-blackwell (push) Successful in 14m47s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m7s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m10s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m10s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 26s
2026-09-07 17:56:07 +03:00
grenadeandClaude Opus 5 bf72ce0ffc fix(neuron): keep the disk check off the path the tests exercise (#322)
CI / Config consistency (push) Successful in 12s
CI / Classify changes (push) Successful in 19s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 11s
CI / Clippy (push) Successful in 2m37s
CI / Test (push) Successful in 6m48s
CI / CUDA type-check (push) Successful in 20m16s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The previous commit made the space rule a pure function and tested it
with literal byte counts, which was right, and then left the two
round-trip tests calling `try_store` — which still ran `df` on the
temporary directory and applied that rule. The tests remained coupled
to how full the machine running them happened to be. Only the margin
had changed: instead of needing 2% of the volume free, they needed
about 24 MB. Same defect, quieter.

So the check moves to `store`, the production entry point, and
`try_store` becomes a write and a rename with no policy in it at all.
The tests call `try_store` and no longer ask the host anything about
itself. A unit test that consults the free space of whatever executes
it is not testing this code.

Also stops leaking the temporary file when a write fails part-way.
There was no cleanup, so a failed write left a partial artifact that
nothing would ever remove — and on the failure most likely to cause
one, a full disk, that dead weight is made of the last free bytes.

Refs #322.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 17:33:50 +03:00
grenadeandClaude Opus 5 22cb84d969 fix(neuron): bound the isq cache write, not the filesystem (#322)
CI / Config consistency (push) Successful in 9s
CI / Classify changes (push) Successful in 17s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 14s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 9m15s
CI / CUDA type-check (push) Successful in 19m4s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
CI failed both cache tests with "the artifact just written" — the load
found nothing where a write had just happened. The write had been
refused and `store` swallows its error, as it must, so the test saw
only the absence.

The rule was wrong. It required 2% of the volume to remain free, which
reads sensibly and fails in the case that matters least and happens
most: on a filesystem already below that line it refused a
six-megabyte write that was in no way responsible for the state. The
runner's /tmp is exactly that filesystem.

Bounding each artifact by half the *remaining* space says the thing
actually meant — no single artifact eats the disk — at every size, and
needs to know nothing about how big the volume is. A 73 GB artifact is
refused onto 100 GB free and allowed onto beast's 2 TB; a few
megabytes is never refused, because a few megabytes is never the
problem.

Two changes so this class of failure is not silent again. The rule is
now a pure function tested against those cases directly, rather than
being reachable only through a filesystem in a particular state — which
is exactly what hid the defect until CI ran it. And the tests call a
fallible `try_store` rather than the swallowing `store`, so a write
that does not happen fails them with its reason instead of surfacing
later as a missing artifact. Production still swallows: a cache that
can fail a load is worse than no cache.

Refs #322.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 17:12:19 +03:00
grenadeandClaude Opus 5 de3f9584e2 feat(neuron): cache in-situ quantisation between loads (#322)
CI / CUDA type-check (push) Blocked by required conditions
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 17s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 13s
CI / Clippy (push) Successful in 2m37s
CI / Test (push) Failing after 9m2s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Quantising at load is a pure function of the checkpoint bytes, the
quant name, and the code that does it. None of those change between
loads, and neuron recomputed the answer every time and dropped it on
unload. For qwen4_exp that is ~32 s per decoder layer — about 26
minutes for a full load — paid again on every restart and every
eviction later reversed.

A hit now skips reading the fused expert tensors at all, which is the
point: the cost avoided is the 5.03 GB read *and* the quantisation,
not just the latter.

The key is (model, revision, quant, quantiser version). The last is not
optional and not paranoia: `bf1e80ed` changed what `quant = "q4k"`
means for a 640-wide row, and an artifact written before it is wrong
after it in a way no checksum of the inputs could catch. Bumping the
version orphans stale artifacts rather than misreading one. A
checkpoint whose revision cannot be identified disables the cache
instead of guessing, because a key that might collide across
checkpoints is worse than recomputing.

One GGUF per layer. GGUF because it records a type per tensor, which is
exactly what the per-row fallback produces — a layer whose down_proj is
Q5_0 while its gate and up are Q4K needs no container invented for it —
and because neuron already reads the format. Per layer because it makes
the cache incrementally useful: a load that runs out of VRAM at layer
16 has still written sixteen layers, and the next attempt starts there.
On hardware where the model does not yet fit at all (#318) that is the
difference between the cache being useless and being the thing that
makes iteration bearable.

It lives inside the source's own cache root, beside the weights it
derives from, rather than behind a config key naming a path. That is
the fleet's existing design — beast already keeps hot weights on NVMe
by symlinking them out of the spinning /archive3 cache, with nothing in
config mapping paths to mounts — and an operator wanting these
artifacts on faster storage does the same to one directory. A second,
differently-shaped way to express that intent would be a second place
to get it wrong.

Every failure is swallowed: a miss, a corrupt artifact, a full disk, an
unwritable directory. The load proceeds by quantising exactly as
before. A cache that can fail a load is worse than no cache. Writes go
through a temp file and a rename so an interrupted one cannot leave a
half-artifact that reads as real, and a write that would take the
filesystem below 2% free is skipped with a warning — proportional
rather than a fixed size, because this lives wherever the weights do
and that ranges from a 7.3 TB array to a developer's checkout.

The round-trip test asserts restored experts are bit-identical in
output, not close: there is no arithmetic between writing and reading
that could legitimately differ. It runs at the checkpoint's real widths
because those decide the types involved — 2560 blocks under Q4K, 640
does not — so it covers a layer whose halves are different quant types,
and asserts that mix is actually present rather than assuming it.

Note the parity fixtures cannot cover this: their expert rows are 4 and
8 wide, far below any block size, so they fall back to F16 and
`QMatMul` keeps non-quantised types as plain tensors with nothing to
cache. The contract is tested where the quantisation is.

Refs #307, #318, #322.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 16:53:14 +03:00
grenade 17394ed8c5 Merge branch 'test/323-parity-long-prompt'
build-prerelease / Resolve version stamps + change detection (push) Successful in 17s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m26s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m37s
build-prerelease / Build neuron-ampere (push) Successful in 2m27s
build-prerelease / Test (push) Successful in 6m39s
build-prerelease / Build neuron-blackwell (push) Successful in 14m44s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m7s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m11s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m17s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 28s
2026-09-07 15:56:38 +03:00
grenadeandClaude Opus 5 b3ed41c744 test(qwen4_exp): a second parity case, for the paths the first cannot reach
CI / Config consistency (push) Successful in 11s
CI / Classify changes (push) Successful in 23s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / CUDA type-check (push) Successful in 19m34s
CI / Clippy (push) Successful in 2m40s
CI / Test (push) Successful in 6m45s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Two fixtures rather than one size of the same test. Each reaches
something the other structurally cannot, and a mutation shows it rather
than the commit message asserting it.

**80 tokens** puts the GatedDeltaNet layers on the chunked prefill
algorithm (#23), which engages at 64 — a different factorisation of the
same recurrence, not the same code with more input. The short case only
ever exercised the recurrent step.

**An eos at position 37** puts a segment boundary inside the n-gram
window, which `shift_right_ignore_eos` must refuse to read across. That
logic is a cummax over eos positions per row, intricate enough to be
worth comparing rather than reasoning about. Making it ignore segments
entirely leaves the short fixture at 0.000002 — it has no eos, so there
is nothing to get wrong — and moves the long one by 0.67. The short
case has no eos deliberately: one that happened to contain an eos would
be testing segment handling by accident and would make the two
redundant.

Both agree with the reference: 2e-6 short, 1.3e-5 long, cosine
1.000000 and matching argmax on each.

The bound is now per case, 1e-5 and 1e-4, with the reason at each call
site. The long case is looser because the chunked factorisation rounds
differently in f32 by construction — a stated reason, not a number
raised until the test passed. It still sits four orders of magnitude
below what the mutations produce, so it discriminates exactly as well:
gate/up swapped moves this fixture by 2.7.

One incidental difference, not a defect: our rotary table is sized by
`max_position_embeddings` and a prompt past it fails inside `narrow`,
where the reference computes cos/sin on the fly and extrapolates. It
cannot arise in production, since `derive_limit` advertises no more
context than the model declares — but it is why the fixture config
declares 128 rather than 64.

Refs #307, #312, #323.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 15:28:23 +03:00
grenade c44dec08d6 Merge branch 'fix/323-layer-type-alias'
build-prerelease / Resolve version stamps + change detection (push) Successful in 18s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m40s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m39s
build-prerelease / Build neuron-ampere (push) Successful in 2m44s
build-prerelease / Test (push) Successful in 6m58s
build-prerelease / Build neuron-blackwell (push) Successful in 14m27s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m12s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m12s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 3m57s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 29s
2026-09-07 15:24:08 +03:00
grenadeandClaude Opus 5 e4961246d2 test(qwen4_exp): logits parity against the upstream reference (#323)
CI / Classify changes (push) Successful in 16s
CI / Config consistency (push) Successful in 8s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m34s
CI / Test (push) Successful in 8m57s
CI / CUDA type-check (push) Successful in 18m59s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The architecture was ported from a written spec and every test asserts
it does what we read that spec to mean. This compares it to the
implementation it was ported from: max_abs 0.000002, cosine 1.000000,
same argmax, through all four novel blocks composed.

Unlike the other fixtures here it needs no snapshot and does not
self-skip. It is 44 KB of random weights in the real architecture — 4
layers, 3 linear-attention and 1 QSA, PLE on layer 1 — generated *by*
transformers, so it is checked in and runs on every push. Generating
the weights from the reference rather than by hand means a tensor name
we cannot find is a finding about our loader, not a fixture that
drifted.

Two properties of the fixture are load-bearing and both were learned by
getting them wrong first.

The weights are uniform(-0.8, 0.8) rather than the (-0.1, 0.1) I
started with. Near the origin silu(x) ~ x/2, so the SwiGLU is nearly
symmetric: with small weights, reading the fused expert's gate and up
halves the wrong way round — the one error moe.rs's own comment warns
about — moved the logits by 2.4e-4 and passed. At this scale the same
mutation moves them by 2.7 and takes the argmax with it. A fixture that
cannot express a defect is not evidence about it.

The tolerance is 1e-5 against an observed 2e-6, rather than a
comfortable 1e-3. The same reasoning: at 1e-3 the gate/up swap passed.
A bound wide enough to feel safe is wide enough to hide the thing it
was written for.

The generator refuses to emit a fixture whose QSA selection rests on a
tie. Blocks score exactly 0.0 whenever relu clamps every head, which is
common with random weights, and when the k-th and (k+1)-th scores are
equal it is `topk`'s ordering of equal elements that picks the survivor
— which torch does not specify and whose CPU and CUDA kernels need not
agree. Seeds 0, 1 and 4 tie; seed 2 does not. This is the one place we
genuinely differ from the reference, and it is benign: a tie means the
indexer scored both blocks equally worthless. The fixture avoids the
situation rather than pretending it is resolved.

On failure the test breaks the error down per position before
asserting. Localised means a discrete choice went differently, spread
means the arithmetic drifted; knowing which cost an hour the first
time.

Also carries the `qwen_sparse_attention` layer-type alias, found while
standing the reference up.

Refs #307, #311, #312, #323.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 15:03:57 +03:00
grenadeandClaude Opus 5 d96f4e8fa1 fix(qwen4_exp): read both of upstream's names for a full-attention layer
CI / Test (push) Blocked by required conditions
CI / CUDA type-check (push) Blocked by required conditions
CI / Config consistency (push) Successful in 9s
CI / Classify changes (push) Successful in 24s
CI / Format (push) Successful in 11s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Clippy (push) Successful in 2m55s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Found while standing up the parity reference (#323). The shipped
`config.json` spells its attention layers `full_attention` — 36 linear
to 12 full, checked against the checkpoint on beast. But
`Qwen4ExpTextConfig` *derives* the array as `qwen_sparse_attention`
when a config omits it, which `transformers` main does for any config
carrying only `full_attention_interval`.

We matched the shipped spelling alone. On a config using the derived
one, `is_full_attention` is false for every layer and the model is
built as 48 linear-attention layers — a 3:1-linear architecture read as
4:1, which is wrong in a way that surfaces somewhere else entirely,
when tensors that should exist do not.

Nothing is broken today: the checkpoint we serve uses the spelling we
read, and the derivation we do ourselves emits it too. This is the case
where someone republishes a config, or a second qwen4_exp checkpoint
appears, and the failure has no obvious relationship to its cause.

Refs #307, #323.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 14:52:48 +03:00
grenade d96e2fcfb7 Merge branch 'fix/323-qsa-scores-in-f32'
build-prerelease / Resolve version stamps + change detection (push) Successful in 20s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m42s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m30s
build-prerelease / Build neuron-ampere (push) Successful in 2m35s
build-prerelease / Test (push) Successful in 7m17s
build-prerelease / Build neuron-blackwell (push) Successful in 14m27s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m4s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m9s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m12s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 27s
2026-09-07 14:29:47 +03:00
grenadeandClaude Opus 5 ea532c24dd fix(qwen4_exp): score QSA blocks in f32, as upstream does (#323)
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 16s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 14s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 8m49s
CI / CUDA type-check (push) Successful in 19m12s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
First finding from the parity work, and found by reading the reference
rather than running it: `modeling_qwen4_exp.py` casts both operands
before the indexer matmul,

    scores = torch.matmul(q.float(), block_key_states.float().transpose(-1, -2))
    scores = torch.relu(scores).sum(dim=-1) / math.sqrt(self.index_head_dim)

and we ran the matmul, the relu and the sum over heads in the model
dtype, casting to f32 only afterwards.

It is not a rounding nicety. These scores feed a top-k over as many as
65,536 blocks and the output is the set of positions a query may attend
— a discrete choice, not a weighting. Two blocks whose f32 scores
differ by less than a bf16 tick compare equal, and `select`'s index
tie-break silently decides which past the model sees.

The same argument is already written down two functions up, on `pool`:
"near 0.25 a bf16 tick is 0.002, so a bf16 accumulation and an f32 one
land on the same wrong number". We applied it to the pooling and not to
the scoring, and the unit tests could not see the difference because
they all pass f32 tensors — the production path is bf16 and is the only
place it mattered.

Both the batched production path and the `BlockSelector::scores` helper
now upcast. They are separate implementations of the same arithmetic,
which is its own hazard, and this is the second time that has cost
something.

The test's construction is the argument: four heads contributing 1.0,
with one head of one block contributing 1.0078125 = 1 + 2^-7, which is
exact in bf16. The sums are 4.0 and 4.0078125, and bf16's tick near 4.0
is 2^-5 = 0.03125, so the second rounds onto the first. The inputs
survive the dtype; the comparison does not. Verified the test fails
without the cast.

Refs #307, #311, #323.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 14:08:35 +03:00
grenade 1ae3ba721e Merge branch 'feat/310-mmap-ngram-table'
build-prerelease / Resolve version stamps + change detection (push) Successful in 17s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m42s
build-prerelease / Build helexa-bench binary (push) Successful in 2m24s
build-prerelease / Build neuron-ampere (push) Successful in 2m44s
build-prerelease / Build cortex binary (push) Successful in 3m31s
build-prerelease / Build helexa-angels binary (push) Successful in 3m30s
build-prerelease / Build helexa-upstream binary (push) Successful in 3m32s
build-prerelease / Test (push) Successful in 9m56s
build-prerelease / Build neuron-blackwell (push) Successful in 15m35s
build-prerelease / Package cortex RPM (push) Successful in 26s
build-prerelease / Package helexa-bench RPM (push) Successful in 28s
build-prerelease / Package helexa-upstream RPM (push) Successful in 26s
build-prerelease / Package helexa-angels RPM (push) Successful in 27s
build-prerelease / Build helexa-router binary (push) Successful in 1m27s
build-prerelease / Build helexa-tools binary (push) Successful in 1m35s
build-prerelease / Package helexa-router RPM (push) Successful in 18s
build-prerelease / Build neuron-ada (push) Successful in 2m13s
build-prerelease / Package helexa-tools RPM (push) Successful in 20s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m6s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m5s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m12s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 30s
2026-09-07 09:22:44 +03:00
grenadeandClaude Opus 5 7dfe18b1f5 feat(qwen4_exp): load the n-gram table first, so its failures are cheap
CI / Config consistency (push) Successful in 10s
CI / Classify changes (push) Successful in 18s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 9s
CI / CUDA type-check (push) Successful in 21m56s
CI / Clippy (push) Successful in 2m57s
CI / Test (push) Successful in 9m35s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
`load_ngram` ran after all 48 decoder layers. Now that the table is
mapped rather than made resident, where it sits in the load order costs
nothing — and everything it can get wrong is a tensor name or an extent
that disagrees with the file it is in.

Those are exactly the failures worth having in the first second. Loading
it last meant a checkpoint whose shard names had moved would say so only
after 48 layers of experts had been quantised — about 26 minutes — and
on the hardware we have, where those experts do not currently fit, it
would never say so at all. The mapping could not be validated against
the real 131-shard checkpoint until #318 lands, which is a poor reason
to not know whether it works.

No behaviour changes: the table is a lookup with no dependency on the
layers, and the layers do not consume it until forward.

Refs #307, #310.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 08:46:14 +03:00
grenade a79b9ae5ad Merge branch 'fix/quant-fallback-match-llamacpp'
build-prerelease / Resolve version stamps + change detection (push) Successful in 22s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 2m23s
build-prerelease / Build neuron-ada (push) Successful in 2m31s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m58s
build-prerelease / Test (push) Successful in 7m25s
build-prerelease / Build neuron-blackwell (push) Successful in 16m38s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m17s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m28s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m39s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 29s
2026-09-07 08:44:13 +03:00
grenadeandClaude Opus 5 8c8d20c248 feat(qwen4_exp): map the n-gram table instead of making it resident (#310)
CI / Test (push) Blocked by required conditions
CI / CUDA type-check (push) Blocked by required conditions
CI / Config consistency (push) Successful in 9s
CI / Classify changes (push) Successful in 22s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 11s
CI / Clippy (push) Successful in 2m37s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
The PLE table is 102.4 GB of BF16 across 128 shards — measured from the
checkpoint's own safetensors headers, 128 x [2500012, 160] at 0.80 GB
each. `ShardedNGramTable` puts all of it wherever the VarBuilder
points, which for a CUDA load is a 32 GB card. It is 28.4% of the
checkpoint and the one component in-situ quantisation never reaches, so
no quant choice shrinks it and no amount of tensor parallelism fits it.

A token needs sixteen rows of 160 values from it. About 5 KB, out of
102.4 GB. Almost the entire table is cold at any instant, so the page
cache is the right thing to decide residency and the right answer is to
decide nothing: map the shards where they already lie in the checkpoint
and copy the requested rows out of the mapping.

Mapping the checkpoint's own tensors in place, rather than building a
separate artifact, is FreeToken's `source_from_safetensors` approach.
It works here because the shards are already exactly the shape a gather
wants — contiguous, equal-stride, row-major, one extent each — so there
is nothing to convert and nothing to keep in sync. Extents are resolved
once at load from the safetensors metadata; a gather is then arithmetic
and a memcpy.

The tensor-name prefix comes from the VarBuilder that would otherwise
have loaded the shards, not from the layer index rebuilt by hand. The
two cannot disagree that way, and disagreeing means reading 102.4 GB of
the wrong thing.

`ShardedNGramTable` stays, as the oracle. The new test is the
differential #310 asks for: both tables over the same fixture, every
row of every shard, plus an unsorted request that crosses shard
boundaries in both directions and repeats a row — the oracle sorts
internally and inverts the permutation, the mapped path does not, so a
disagreement there is precisely that inverse being wrong on one side.
Verified it fails if the data offset drops the 8-byte header length.

The whole qwen4_exp suite now loads through the mapped table rather
than the oracle, so the loader tests exercise the residency that
serves.

This does not by itself make the model loadable: the routed experts are
73.0 GB against 63.7 GB of VRAM, which is #318. It removes the larger
of the two blockers, and the one that no other decision could shrink.

Refs #307, #310, #318.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 08:40:44 +03:00
grenadeandClaude Opus 5 15c664c5c3 fix(quant): use llama.cpp's fallback table, which spends bits rather than saving them
CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 20s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 11s
CI / Clippy (push) Successful in 2m31s
CI / Test (push) Successful in 8m59s
CI / CUDA type-check (push) Successful in 19m19s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
`bf1e80ed` said it met the 640-wide reduction "the way llama.cpp meets
the same wall" and did not. Upstream's `tensor_type_fallback`
(src/llama-quant.cpp) maps Q4K -> Q5_0 and Q5K -> Q5_1; that commit
used Q4_0 and Q5_0. Only Q2K/Q3K -> Q4_0 and Q6K -> Q8_0 matched.

The difference is a principle, not two table cells. Every upstream
fallback *increases* bits per weight: a k-quant carries a two-level
scale hierarchy the legacy 32-block types do not, so it is better than
a legacy type of equal width, and dropping Q4K (4.5 bpw) to Q4_0 (also
4.5) holds memory constant by taking the quality hit in silence.
Upstream pays a bit instead. We were taking the silent trade, in the
one direction that flatters the constraint we are straining against,
having asserted conformance to a table nobody had read.

It is not free. `down_proj` is exactly a third of the 120.8 B routed
parameters, so Q5_0 rather than Q4_0 costs +5.0 GB across the expert
bank: 67.9 -> 73.0 GB. Every published number that rests on the expert
footprint moves with it, and the model was already over both cards
combined, so this makes host residency more necessary rather than less.
That is the right way round — the cheaper table did not make the model
fit, it just made the shortfall look smaller.

The undivisible-by-32 case now falls back to F16 as upstream does,
rather than erroring. It is unreachable for this checkpoint (640 and
2560 are both multiples of 32) and vanishingly rare in general, but a
3.5x blowup on a tensor must be loud rather than fatal, so it warns.

Also corrects the record: nothing here is GGUF-incompatible. GGUF
stores a ggml_type per tensor and mixed types within a file are the
norm — "Q4_K_M" names a mixture, not a uniform type. A per-row
substitution is just the tensor's type. What still differs from a file
llama.cpp would write is the *selection* policy above this function
(their per-tensor upgrades and imatrix weighting), which is a quality
question for #315, not a format one.

Refs #307, #315, #322.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 08:22:59 +03:00
grenade 7593ab3d5d Merge branch 'fix/qwen4-exp-expert-row-width'
build-prerelease / Resolve version stamps + change detection (push) Successful in 22s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m39s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m35s
build-prerelease / Build neuron-ampere (push) Successful in 2m39s
build-prerelease / Test (push) Successful in 9m16s
build-prerelease / Build neuron-blackwell (push) Successful in 14m42s
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m15s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m15s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m14s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 28s
2026-09-06 21:21:11 +03:00
grenadeandClaude Opus 5 bf1e80ed55 fix(qwen4_exp): 640 is not a whole number of k-quant blocks
CI / Config consistency (push) Successful in 11s
CI / Classify changes (push) Successful in 23s
CI / Format (push) Successful in 12s
CI / Clippy (push) Successful in 2m37s
CI / Test (push) Successful in 7m2s
CI / CUDA type-check (push) Successful in 19m17s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The first real cold load on beast got through routing, resolved all
131 shards, parsed the config, reached decoder layer 0 — and died
there:

    quantize experts to Q4K: quantized tensor must have their last dim
    divisible by block size [2560, 640] 256

ggml quantises along the last dimension and a row must be a whole
number of blocks. `gate_up` reduces over `hidden_size` = 2560, which
is ten blocks of 256. `down_proj` reduces over `moe_intermediate_size`
= 640, which is two and a half. So `quant = "q4k"` could never have
loaded this model, on any card, whatever else was fixed first.

It is not an orientation mistake — `[2560, 640]` is the `[out, in]`
layout a `QMatMul` wants, and 640 is genuinely the reduction. The
constraint is the number itself. llama.cpp meets the same wall and
answers it the same way: substitute a 32-block type for the row that
will not divide. Q4_0 and Q4K are both 4.5 bpw, so the memory
arithmetic every placement decision rests on is unchanged; the trade
is quality, on one of the three expert matrices, against not loading
at all. A row that divides neither 256 nor 32 is an error, because
silently leaving those weights dense would be a memory surprise rather
than a mercy.

The substitution is decided once per layer from the shapes rather than
per expert -- 512 identical lines a layer would bury the one fact
worth reading.

The test uses the checkpoint's own widths, 2560 and 640, scaled down
only in expert count. The fixture that missed this was 256 in both
dimensions -- the narrowest a k-quant accepts -- so every row divided
and the two halves were indistinguishable. A test whose fixture cannot
express the failure is not evidence about it.

Refs #307, #315, #320.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-06 20:59:39 +03:00
grenade a14d1af447 Merge branch 'fix/qwen4-exp-load-reporting-and-pool-trim'
build-prerelease / Resolve version stamps + change detection (push) Successful in 21s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m43s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m35s
build-prerelease / Build neuron-ampere (push) Successful in 2m36s
build-prerelease / Test (push) Successful in 10m54s
build-prerelease / Build neuron-blackwell (push) Successful in 14m28s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m7s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m16s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m8s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 30s
2026-09-06 19:58:00 +03:00
grenadeandClaude Opus 5 f1af926c30 fix(qwen4_exp): route the arch on the path that actually loads it
CI / Config consistency (push) Successful in 12s
CI / Classify changes (push) Successful in 27s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 15s
CI / Clippy (push) Successful in 2m39s
CI / Test (push) Successful in 6m49s
CI / CUDA type-check (push) Successful in 19m10s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
`2f2f306c` added qwen4_exp to `DENSE_SUPPORTED_MODEL_TYPES` and to
`candle::load_arch_dense`. That is the CPU fallback. Every CUDA load
goes through the device worker's own `load_dense_inner`, which had no
arm for it — so on a GPU host the model passed preflight and then hit

    unrouted supported model_type 'qwen4_exp' — DENSE_SUPPORTED_MODEL_TYPES
    and load_dense_inner must stay in sync

in about 200 ms, having read none of the 360 GB. The bail's own comment
is the specification this violated, and it was written before there was
a second loader to violate it.

The arm mirrors the CPU one, and the job now carries the parsed
`quant`. It has to: the worker load never had a quant parameter, so
even a routed qwen4_exp would have loaded BF16 experts — 241.6 GB of
them, measured from the checkpoint's own safetensors headers, against
a 32.6 GB card. In-situ quantisation reaching only the CPU path is the
same defect #320 fixed one layer up, and it would have looked like
#320's fix had not worked.

The test asserts every model_type the preflight admits has an arm here,
by driving a bare `{"model_type": ...}` at the loader and requiring the
failure to be anything other than "unrouted". Adding a name to the list
with no loader behind it now fails in CI rather than on a GPU host at
load time — which is the only place the difference between the two
loaders was observable.

Refs #307, #315, #320.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-06 19:36:46 +03:00
grenadeandClaude Opus 5 430b6be62b fix(neuron): let the loaded model answer for itself, and give the pool back
CI / Test (push) Blocked by required conditions
CI / Config consistency (push) Successful in 9s
CI / Classify changes (push) Successful in 21s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / CUDA type-check (push) Successful in 19m23s
CI / Format (push) Successful in 12s
CI / Clippy (push) Successful in 2m41s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Three things the first qwen4_exp cold load would have got wrong, none
of which the load itself has to reach to matter.

The single-GPU path decided prefix-snapshot capability by matching
config.json's model_type against a list of two, while
ModelArch::supports_kv_snapshot had grown to three. So qwen4_exp
loaded with snapshots reported unavailable and batched decode silently
off — the work that landed the snapshot was reachable only through a
list somewhere else that nobody had reason to look at. The dense load
reply now carries the arch's own answer (DenseLoad), taken while the
model is still in hand on the worker thread, which is the only place
that can be asked. The TP path still re-derives it; that is correct
for the two archs it serves today and it is commented as the hazard it
is, because #319 will add a third.

Nothing built a context profile for qwen4_exp either, so derive_limit
had no physics and the model would have advertised the static prompt
cap. The profile is the qwen3_5 one plus a term: the QSA indexer keeps
a second cache on every full-attention layer, 12 x 1 x 128 x 2 bytes =
3 KiB/token beside the main cache's 24. Advertising against the main
cache alone overstates what the card can hold by an eighth, which is
the #281 failure mode. The test drives both builders off the
checkpoint's verbatim config.json and asserts each declines the
other's, since the load path now tries them in turn.

And a load that fails part-way left its bytes in cudarc's pool. The
image path learned this at the benjy cold-swap and trims after every
load and drop; the text path never did. A dense load that runs out of
VRAM has typically allocated tens of gigabytes first, and the model
that was evicted to make room for it needs them back — so LoadGguf,
LoadDense, DropArch and DropTp now trim the way LoadImage and
DropImage always have.

Refs #307, #310, #281.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-06 19:10:08 +03:00
grenade af40bac9b8 Merge branch 'fix/320-isq-reachable-and-log-levels'
build-prerelease / Resolve version stamps + change detection (push) Successful in 18s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Lint (fmt + clippy) (push) Successful in 4m7s
build-prerelease / Test (push) Successful in 9m15s
build-prerelease / Build neuron-ampere (push) Successful in 2m11s
build-prerelease / Build neuron-ada (push) Successful in 2m14s
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Successful in 14m18s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m6s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 29s
2026-09-06 18:03:15 +03:00
grenade d5738bfb89 fix(neuron): read quant as ISQ when the repo has no GGUF (#320)
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 14s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m36s
CI / Test (push) Successful in 9m44s
CI / CUDA type-check (push) Successful in 19m12s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The single-GPU load paths treated a non-empty `quant` as "select the
GGUF whose filename contains this string", while the TP path treats the
same field as in-situ quantisation of safetensors. A model published
only as safetensors is therefore loadable quantised on two cards and
not on one, which is backwards — the single card is the one that needs
the quantisation.

Fall through to the dense loader (which applies ISQ) when the repo has
no GGUF at all but does have safetensors. A repo that does carry GGUFs
keeps the old behaviour exactly, including the old error when none of
them match: that case is a typo, not an ISQ request, and silently
turning it into a full-precision load would be a worse answer than the
error.

Also drop the per-model context-limit line from debug to trace. Every
input to it is fixed at load, so it repeats byte-identically on every
`GET /models` poll — roughly 8,600 lines a day per model saying the
same thing, on top of whatever you were actually reading the journal
for.

And set RUST_LOG=debug for neuron fleet-wide from deploy.yml, in the
model.conf drop-in it already manages. The code default is `info`,
which is too coarse to reconstruct a load or a routing decision after
the fact; before this, one host had `debug` because it was set there by
hand, and the others quietly did not. A verbosity you have to remember
to set is one you discover you did not have at the moment you needed
it. It sorts after local.conf, so the managed value wins.
2026-09-03 16:22:19 +03:00
grenade ecbfe22124 Merge branch 'feat/qwen4-exp-quantised-experts'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m45s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m27s
build-prerelease / Build neuron-ampere (push) Successful in 2m28s
build-prerelease / Test (push) Successful in 10m36s
build-prerelease / Build neuron-blackwell (push) Successful in 14m47s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m3s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m11s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m10s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 24s
2026-09-02 19:52:10 +03:00
grenade 3e5d39807d fix(qwen4_exp): point the CUDA leader load at the moved parse_quant_string
CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 16s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m32s
CI / Test (push) Successful in 9m9s
CI / CUDA type-check (push) Successful in 19m20s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Moving the function out of tp/worker.rs left dispatch.rs calling it
through worker's `use` import, which is private — so the call compiled
nowhere except in my head. The CUDA type-check caught it; the CPU
build cannot, because that call site is cfg(cuda) and every local check
I ran was blind to it.

That is what the job is for, and it is the second time this port has
relied on it. Worth remembering that "fmt, clippy and test are green"
means green on roughly half of this crate.
2026-09-02 19:30:08 +03:00
grenade 05a5aff25e feat(qwen4_exp): quantise the experts in situ, which is what makes it loadable
CI / Config consistency (push) Successful in 10s
CI / Classify changes (push) Successful in 20s
CI / Format (push) Successful in 12s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Clippy (push) Successful in 2m46s
CI / Test (push) Successful in 12m52s
CI / CUDA type-check (push) Failing after 19m32s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
120.8 B routed parameters are 241.6 GB at BF16 and 67.9 GB at q4k,
against 64 GB of VRAM and roughly 110 GB of usable host RAM. Everything
else in this port has been waiting on that arithmetic.

The experts are quantised out of the banked tensors rather than out of
the checkpoint, which is what bounds the peak: one layer's fused pair is
5.03 GB, and it is dropped once its 512 experts are quantised, so the
full 241.6 GB never exists at once. That is the reason the previous
commit banked them.

q4k was chosen on evidence rather than taste. fp8 experts are excluded
by arithmetic — 120.8 GB of experts plus 51.2 GB of PLE tables is
176.9 GB, which does not fit even counting host RAM. NVFP4 and IQ4_XS
are memory-equivalent to q4k at ~4-4.5 bpw, so they separate only on
kernels, and neuron already implements the ggml k-quants through the
same in-situ path the 27B and the image model use in production. NVFP4
is the optimisation to race later, against a *batched* measurement:
FreeToken measured it losing to plain Triton at M=1 for expert widths
below 1024, and this model is 640.

parse_quant_string moves out from behind cfg(feature = "cuda"). A
catalogue quant name means the same thing whichever loader reads it, and
it was unavailable to precisely the path that now needs it.

One trap worth the comment it got: QTensor::quantize reads
src.storage() — the whole backing buffer, ignoring shape and offset — so
a view into the fused tensor hands it every expert's bytes while the
shape claims one, and candle panics on a block count deep inside
from_float. Neither contiguous() nor copy() helps, because a slice along
dim 0 is already contiguous and both are then free to keep sharing the
parent buffer. The source has to own its storage outright.

The test uses 256-wide tensors because the k-quants block on 256
elements, so a toy fixture cannot be quantised at all — and it compares
against an unrelated expert as well as against the dense one, so a
quantised path that returned something smoothly wrong would not pass.

Refs #307, #315, #259.
2026-09-02 18:52:24 +03:00
grenade cd74eeaf1c Merge branch 'feat/qwen4-exp-expert-bank'
build-prerelease / Resolve version stamps + change detection (push) Successful in 22s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m36s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Test (push) Successful in 9m29s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 2m25s
build-prerelease / Build neuron-ada (push) Successful in 2m28s
build-prerelease / Build neuron-blackwell (push) Successful in 17m0s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m5s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m9s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m5s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 29s
2026-09-02 18:16:31 +03:00
grenade 60bbe88ba7 feat(qwen4_exp): bank the experts instead of slicing them apart
CI / Config consistency (push) Successful in 14s
CI / Classify changes (push) Successful in 28s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 12s
CI / Clippy (push) Successful in 2m59s
CI / Test (push) Successful in 10m36s
CI / CUDA type-check (push) Successful in 20m31s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Slicing the checkpoint's two fused tensors into 512 per-expert modules
copies 241.6 GB at BF16, 5.03 GB per layer. That fits in neither
beast's 64 GB of VRAM nor its 123 GB of host RAM, and no quantisation
rescues it — #309 measured 59.4 GB of 63.7 used at 4.25 bpw with only
four layers' experts offloaded. The loader was the reason nothing could
be loaded.

So the block now takes the experts in whichever shape the checkpoint
gives, and views one per routed token. The routing is untouched: it was
already correct and duplicating it to get a second storage layout would
have invited exactly the kind of drift that is hard to see — softmax in
f32 over all 512, top-k, renormalise, sigmoid-gated shared expert.

Banking costs nothing per call. A slice along dim 0 of a contiguous
tensor is contiguous, and x.matmul(w.t()) is what Linear does anyway,
so the views feed the same GEMMs without a copy.

Because "we only changed where the bytes live" is a claim rather than a
fact, the same weights are expressed both ways and asserted to compute
the same function. Reading the gate and up halves in the wrong order is
pinned separately, since the banked path would get that wrong as
silently as the sliced one did.

It is also the layout a host-resident expert bank with a device-side
gather wants (#318), so that work starts from this rather than having
to undo the slicing first.

One consequence worth noting: MLP::from_weights has no caller outside
tests any more, so it goes back behind cfg(test).

Refs #307, #312, #318.
2026-09-02 17:06:44 +03:00
grenade 499ed5013e Merge branch 'feat/qwen4-exp-mtp-head'
build-prerelease / Resolve version stamps + change detection (push) Successful in 26s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m59s
build-prerelease / Test (push) Successful in 9m52s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 3m15s
build-prerelease / Build neuron-ada (push) Successful in 3m22s
build-prerelease / Build neuron-blackwell (push) Successful in 16m5s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m6s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m14s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m24s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 29s
2026-09-02 14:43:04 +03:00
grenade 0ab159e144 Merge branch 'docs/mtp-wiring-settled' 2026-09-02 14:43:04 +03:00
grenade e3a82e412d Merge branch 'feat/qwen4-exp-snapshot' 2026-09-02 14:43:04 +03:00
31 changed files with 2661 additions and 214 deletions
+27 -15
View File
@@ -317,22 +317,36 @@ jobs:
# restart decision below, exactly like the drop-in.
neuron_toml_changed="${{ steps.neuronconf.outputs.changed }}"
# ── Desired per-model systemd drop-in ─────────────────────────
# model.conf carries NEURON_MAX_PROMPT_TOKENS so the context cap
# is deterministic per host and rolled out (with a restart) by
# ── Desired per-host systemd drop-in ──────────────────────────
# model.conf carries the environment neuron is started with, so
# it is deterministic per host and rolled out (with a restart) by
# this workflow, not hand-edited. It sorts after local.conf, so a
# deploy-managed value wins over any manual local override of the
# same variable. See doc/context-limits.md.
# same variable.
#
# RUST_LOG is fleet-uniform and unconditional. The code default
# is `info`, which is too coarse to diagnose a load or a routing
# decision after the fact, and before this it was `debug` on
# exactly one host because someone set it there by hand. A
# verbosity you have to remember to set is a verbosity you find
# out you did not have at the moment you needed it. Bumping a
# module (`debug,neuron::harness::candle=trace`) is a one-line
# change here, applied everywhere, with an audit trail.
#
# NEURON_MAX_PROMPT_TOKENS is per-host and stays conditional;
# see doc/context-limits.md.
conf=/etc/systemd/system/neuron.service.d/model.conf
config_changed=${neuron_toml_changed:-0}
rust_log="debug"
desired=$(printf '%s\n%s\n%s' \
"# Managed by .gitea/workflows/deploy.yml - do not edit by hand." \
"[Service]" \
"Environment=RUST_LOG=${rust_log}")
if [ -n "${max_prompt_tokens}" ]; then
desired=$(printf '%s\n%s\n%s\n%s' \
"# Managed by .gitea/workflows/deploy.yml - do not edit by hand." \
"# Per-model context cap; see doc/context-limits.md." \
"[Service]" \
desired=$(printf '%s\n%s' "${desired}" \
"Environment=NEURON_MAX_PROMPT_TOKENS=${max_prompt_tokens}")
[ "${desired}" = "$(cat "${conf}" 2>/dev/null || true)" ] || config_changed=1
fi
[ "${desired}" = "$(cat "${conf}" 2>/dev/null || true)" ] || config_changed=1
# ── Desired registry drop-in ──────────────────────────────────
# Every model fetch goes through rustingface rather than
@@ -411,15 +425,13 @@ jobs:
echo "installed=${installed} published=${latest:-unknown} pkg_changed=${pkg_changed} config_changed=${config_changed} — deploying"
# Write the drop-in (staged in gitea_ci's dir, installed root-owned).
# Guarded on max_prompt_tokens as well as config_changed: since
# a neuron.toml change also sets config_changed, an entry with
# no max_prompt_tokens would otherwise reach this with `desired`
# unset.
if [ "${config_changed}" -eq 1 ] && [ -n "${max_prompt_tokens}" ]; then
# `desired` is always set now that RUST_LOG is unconditional, so
# config_changed alone is the guard.
if [ "${config_changed}" -eq 1 ]; then
printf '%s\n' "${desired}" > /var/lib/gitea_ci/model.conf
sudo /usr/bin/install -o root -g root -m 0644 -D /var/lib/gitea_ci/model.conf "${conf}"
rm -f /var/lib/gitea_ci/model.conf
echo "applied ${conf}: NEURON_MAX_PROMPT_TOKENS=${max_prompt_tokens}"
echo "applied ${conf}: RUST_LOG=${rust_log} NEURON_MAX_PROMPT_TOKENS=${max_prompt_tokens:-<unset>}"
fi
# Staged 0600 in gitea_ci's own dir before install, so the token
Generated
+1
View File
@@ -3147,6 +3147,7 @@ dependencies = [
"half",
"hf-hub",
"image",
"memmap2",
"minijinja",
"minijinja-contrib",
"rayon",
+6
View File
@@ -104,6 +104,12 @@ minijinja-contrib = { version = "2", features = ["pycompat"] }
# tp `fused_load` module to read per-rank slices of fused QKV tensors
# without materialising the full tensor on device.
safetensors = "0.8"
# The qwen4_exp PLE n-gram table is 102.4 GB of BF16 across 128 shards
# (#310). It is a pure gather of ~5 KB per token, so it is read straight
# out of the checkpoint by mmap rather than made resident anywhere —
# the page cache decides what stays warm, and almost all of the table is
# cold at any instant.
memmap2 = "0.9"
# Vision capability for Qwen3.6 (Stage A of the vision plan in
# doc/vision-qwen3_6-spec.md). `image` decodes PNG/JPEG/etc from
# the bytes embedded in `data:image/...;base64,...` content parts;
@@ -19,9 +19,12 @@ pub struct Qwen3_5MLP {
impl Qwen3_5MLP {
/// Construct directly from pre-built projections.
///
/// `qwen4_exp` ships its routed experts as fused 3D tensors rather
/// than per-expert modules, so its loader slices them and builds the
/// experts here — same SwiGLU, different storage.
/// Test-only again: `qwen4_exp` briefly built its 512 experts this
/// way before the fused tensors stayed banked (`Experts::Banked`),
/// which removed the last caller outside tests. The differential
/// test that proves banking neutral still needs it, to express the
/// same weights in the layout it is being compared against.
#[cfg(test)]
pub(crate) fn from_weights(gate_proj: Linear, up_proj: Linear, down_proj: Linear) -> Self {
Self {
gate_proj,
+573 -13
View File
@@ -29,18 +29,289 @@
//! replaces the loop behind the same `forward` signature.
use anyhow::{Context, Result};
use candle_core::{DType, Module, Tensor};
use candle_core::quantized::{GgmlDType, QMatMul, QTensor};
use candle_core::{DType, IndexOp, Module, Tensor};
use candle_nn::Linear;
use candle_nn::var_builder::ShardedVarBuilder;
use super::TextConfig;
use super::mlp::Qwen3_5MLP;
/// How a layer's routed experts are stored.
///
/// Two shapes, because the right one depends on whether the model fits.
///
/// `PerExpert` is the Qwen3-Next layout: one module per expert, sliced
/// out at load. Fine when the weights are device-resident anyway.
///
/// `Banked` keeps the checkpoint's own fused tensors whole and takes a
/// view per routed expert. `qwen4_exp` needs it: slicing its 512
/// experts into modules copies **241.6 GB** at BF16 — 5.03 GB per
/// layer — which fits in neither 64 GB of VRAM nor 123 GB of host RAM,
/// at any quantisation. It is also the layout a host-resident bank with
/// a device-side gather wants (#318), so the fused form is where that
/// work starts rather than something it would have to undo.
///
/// Banking costs nothing per call: a slice along dim 0 of a contiguous
/// tensor is contiguous, and `x.matmul(w.t())` is what `Linear` does
/// anyway, so the views feed the same GEMMs without a copy.
pub(crate) enum Experts {
PerExpert(Vec<Qwen3_5MLP>),
/// Quantised in situ at load (#315). One `(gate, up, down)` triple
/// per expert.
///
/// This is what makes `qwen4_exp` loadable at all: 120.8 B routed
/// parameters are 241.6 GB at BF16 and 67.9 GB at q4k, against
/// beast's 64 GB of VRAM and ~110 GB of usable host RAM. Nothing
/// about the arithmetic works until they shrink.
///
/// Quantising *from* the banked tensor is what keeps the peak
/// bounded: one layer's fused BF16 tensor is 5.03 GB, and it is
/// dropped once its 512 experts are quantised, so the full 241.6 GB
/// never exists at once.
Quantized(Vec<QuantizedExpert>),
Banked {
/// `(num_experts, 2 * intermediate, hidden)` — gate rows first,
/// then up. Reversing them computes `silu(up) * gate`, which is
/// a different function of the same weights (#312).
gate_up: Tensor,
/// `(num_experts, hidden, intermediate)`
down: Tensor,
intermediate: usize,
},
}
/// One expert's three projections, quantised.
pub(crate) struct QuantizedExpert {
gate: QMatMul,
up: QMatMul,
down: QMatMul,
}
/// The quantisation type actually usable for rows of `row` elements.
///
/// ggml quantises along the last dimension, and a row must be a whole
/// number of blocks. The k-quants (`Q2K`..`Q8K`) block on 256; the
/// older types block on 32. `qwen4_exp`'s `down_proj` reduces over
/// `moe_intermediate_size = 640`, which is 2.5 blocks of 256 — so a
/// `quant = "q4k"` load fails on the very first layer with
/// `quantized tensor must have their last dim divisible by block size
/// [2560, 640] 256`, having read 131 shards to get there.
///
/// **This table is llama.cpp's**, from `tensor_type_fallback` in
/// `src/llama-quant.cpp`, and the principle behind it is the part
/// worth stating: **every fallback spends bits rather than saving
/// them.** A k-quant carries a two-level scale hierarchy that the
/// legacy 32-block types do not, so it is better than a legacy type at
/// equal width — dropping `Q4K` (4.5 bpw) to `Q4_0` (also 4.5) would
/// hold the memory constant and take the quality hit silently.
/// Upstream declines that trade and pays a bit instead:
///
/// | requested | fallback | bpw |
/// |---|---|---|
/// | `Q2K`, `Q3K` | `Q4_0` | 2.63 / 3.44 -> 4.5 |
/// | `Q4K` | `Q5_0` | 4.5 -> 5.5 |
/// | `Q5K` | `Q5_1` | 5.5 -> 6.0 |
/// | `Q6K`, `Q8K` | `Q8_0` | 6.56 -> 8.5 |
///
/// For `qwen4_exp` that is not free: `down_proj` is exactly a third of
/// the 120.8 B routed parameters, so `Q5_0` rather than `Q4_0` costs
/// **+5.0 GB** across the expert bank (67.9 -> 73.0 GB). We take it
/// anyway. The cheaper table was an unmeasured divergence in the one
/// direction that flatters the constraint we are straining against,
/// and this matrix is the one a mixed k-quant scheme normally treats
/// as quality-critical. If the 5 GB turns out to buy nothing, #315 is
/// where that gets measured and reversed on evidence.
///
/// A row that divides 32 either — vanishingly rare, and impossible for
/// this checkpoint since 640 and 2560 are both multiples of 32 — falls
/// back to F16 as upstream does, loudly. It is a 3.5x blowup on that
/// tensor, so it must never be quiet.
pub(crate) fn quant_for_row(requested: GgmlDType, row: usize) -> candle_core::Result<GgmlDType> {
if row.is_multiple_of(requested.block_size()) {
return Ok(requested);
}
let fallback = match requested {
GgmlDType::Q2K | GgmlDType::Q3K => GgmlDType::Q4_0,
GgmlDType::Q4K => GgmlDType::Q5_0,
GgmlDType::Q5K => GgmlDType::Q5_1,
GgmlDType::Q6K | GgmlDType::Q8K => GgmlDType::Q8_0,
// Already a 32-block type (or not a quant at all): there is no
// smaller block to demote to, so the F16 check below decides.
other => other,
};
if row.is_multiple_of(fallback.block_size()) {
return Ok(fallback);
}
tracing::warn!(
?requested,
?fallback,
row,
"row divides neither the requested block size nor its 32-wide \
fallback; storing this tensor as F16 — 3.5x the bytes"
);
Ok(GgmlDType::F16)
}
impl Experts {
pub(crate) fn len(&self) -> usize {
match self {
Experts::PerExpert(v) => v.len(),
Experts::Banked { gate_up, .. } => gate_up.dims()[0],
Experts::Quantized(v) => v.len(),
}
}
/// Quantise a banked pair into per-expert triples.
///
/// Takes the fused tensors by reference and drops nothing itself —
/// the caller owns the peak, and should hold one layer at a time.
///
/// The two halves may not end up the same dtype: see
/// [`quant_for_row`]. `qwen4_exp`'s `down_proj` reduces over 640,
/// which no k-quant can block.
pub(crate) fn quantize_banked(
gate_up: &Tensor,
down: &Tensor,
intermediate: usize,
dtype: GgmlDType,
) -> candle_core::Result<Self> {
let n = gate_up.dims()[0];
// Decided once, from the shapes, rather than per expert —
// every expert in a layer has the same two row widths, and
// 512 identical log lines per layer would bury the fact.
let gate_up_dtype = quant_for_row(dtype, gate_up.dims()[2])?;
let down_dtype = quant_for_row(dtype, down.dims()[2])?;
if gate_up_dtype != dtype || down_dtype != dtype {
tracing::info!(
requested = ?dtype,
gate_up_row = gate_up.dims()[2],
gate_up_dtype = ?gate_up_dtype,
down_row = down.dims()[2],
down_dtype = ?down_dtype,
"expert quantisation: a row width does not divide the requested \
block size; substituting a 32-block type for that half"
);
}
let mut out = Vec::with_capacity(n);
for e in 0..n {
let gu = gate_up.i(e)?;
// The source must own its storage outright.
// `QTensor::quantize` reads `src.storage()` — the whole
// backing buffer, ignoring shape and offset — so a view into
// the fused tensor hands it every expert's bytes while the
// shape claims one, and candle panics on a block-count
// mismatch inside `from_float`. Neither `contiguous()` nor
// `copy()` saves you here: a slice along dim 0 is already
// contiguous, so both are free to hand back something still
// sharing the parent's buffer. Round-tripping through a Vec
// is the part that actually allocates, and it is also where
// the f32 the quantiser wants comes from.
let q = |t: Tensor, dtype: GgmlDType| -> candle_core::Result<QMatMul> {
let dims = t.dims().to_vec();
let vals = t.to_dtype(DType::F32)?.flatten_all()?.to_vec1::<f32>()?;
let owned = Tensor::from_vec(vals, dims, t.device())?;
QMatMul::from_qtensor(QTensor::quantize(&owned, dtype)?)
};
out.push(QuantizedExpert {
gate: q(gu.narrow(0, 0, intermediate)?, gate_up_dtype)?,
up: q(gu.narrow(0, intermediate, intermediate)?, gate_up_dtype)?,
down: q(down.i(e)?, down_dtype)?,
});
}
Ok(Experts::Quantized(out))
}
/// The quantised tensors, named for the on-disk cache (#322).
///
/// `None` for any layout that is not quantised — there is nothing
/// to cache about weights that were never transformed.
pub(crate) fn cache_entries(&self) -> Option<Vec<(String, std::sync::Arc<QTensor>)>> {
let Experts::Quantized(v) = self else {
return None;
};
let of = |m: &QMatMul| match m {
QMatMul::QTensor(t) => Some(t.clone()),
// A dequantised-on-load QMatMul has no quantised tensor to
// write. `CANDLE_DEQUANTIZE_ALL` produces these; caching
// half a layer would be worse than caching none of it.
_ => None,
};
let mut out = Vec::with_capacity(v.len() * 3);
for (i, q) in v.iter().enumerate() {
out.push((format!("e{i}.gate"), of(&q.gate)?));
out.push((format!("e{i}.up"), of(&q.up)?));
out.push((format!("e{i}.down"), of(&q.down)?));
}
Some(out)
}
/// Rebuild from what [`Self::cache_entries`] wrote.
///
/// Errors rather than filling gaps: a layer missing an expert is a
/// truncated artifact, and serving it would mean one expert of 512
/// quietly doing nothing.
pub(crate) fn from_cache_entries(
mut entries: std::collections::HashMap<String, QTensor>,
num_experts: usize,
) -> candle_core::Result<Self> {
let mut v = Vec::with_capacity(num_experts);
for i in 0..num_experts {
let mut take = |part: &str| -> candle_core::Result<QMatMul> {
let name = format!("e{i}.{part}");
match entries.remove(&name) {
Some(t) => QMatMul::from_qtensor(t),
None => candle_core::bail!("cached expert layer is missing '{name}'"),
}
};
v.push(QuantizedExpert {
gate: take("gate")?,
up: take("up")?,
down: take("down")?,
});
}
Ok(Experts::Quantized(v))
}
/// Expert `e` applied to the rows routed to it.
pub(crate) fn forward_one(&self, e: usize, xs: &Tensor) -> candle_core::Result<Tensor> {
match self {
Experts::PerExpert(v) => v[e].forward(xs),
Experts::Banked {
gate_up,
down,
intermediate,
} => {
let gu = gate_up.i(e)?;
let gate_w = gu.narrow(0, 0, *intermediate)?;
let up_w = gu.narrow(0, *intermediate, *intermediate)?;
let lhs = candle_nn::ops::silu(&xs.matmul(&gate_w.t()?)?)?;
let rhs = xs.matmul(&up_w.t()?)?;
(lhs * rhs)?.matmul(&down.i(e)?.t()?)
}
Experts::Quantized(v) => {
let q = &v[e];
// The quantised kernels accumulate in f32 whatever the
// activation dtype, so the cast is explicit here rather
// than implied — and the result comes back in the
// caller's dtype so the routing arithmetic upstream
// does not have to know which layout it got.
let dtype = xs.dtype();
let xs32 = xs.to_dtype(DType::F32)?;
let lhs = candle_nn::ops::silu(&q.gate.forward(&xs32)?)?;
let rhs = q.up.forward(&xs32)?;
q.down.forward(&(lhs * rhs)?)?.to_dtype(dtype)
}
}
}
}
pub struct Qwen3_5MoeBlock {
/// Router: `(num_experts, hidden)`, checkpoint name `mlp.gate`.
gate: Linear,
/// Routed experts, checkpoint names `mlp.experts.{i}.{gate,up,down}_proj`.
experts: Vec<Qwen3_5MLP>,
/// Routed experts. Per-expert modules for `qwen3_5`; the
/// checkpoint's fused tensors for `qwen4_exp` — see [`Experts`].
experts: Experts,
/// Always-on expert, `mlp.shared_expert.*`. `None` when the config
/// declares no shared expert (Qwen3-30B-A3B style).
shared_expert: Option<Qwen3_5MLP>,
@@ -62,7 +333,7 @@ impl Qwen3_5MoeBlock {
/// `qwen3_5` config.
pub(crate) fn from_parts(
gate: Linear,
experts: Vec<Qwen3_5MLP>,
experts: Experts,
shared_expert: Option<Qwen3_5MLP>,
shared_expert_gate: Option<Linear>,
num_experts_per_tok: usize,
@@ -129,7 +400,7 @@ impl Qwen3_5MoeBlock {
Ok(Self {
gate,
experts,
experts: Experts::PerExpert(experts),
shared_expert,
shared_expert_gate,
num_experts_per_tok: cfg.num_experts_per_tok,
@@ -201,13 +472,13 @@ impl Module for Qwen3_5MoeBlock {
)?;
let mut ys = xs_flat.zeros_like()?;
for (e, expert) in self.experts.iter().enumerate() {
for e in 0..self.experts.len() {
if tokens_for[e].is_empty() {
continue;
}
let rows = Tensor::new(tokens_for[e].as_slice(), xs.device())?;
let picked = xs_flat.index_select(&rows, 0)?;
let out = expert.forward(&picked)?;
let out = self.experts.forward_one(e, &picked)?;
let w = Tensor::new(weights_for[e].as_slice(), xs.device())?
.to_dtype(out.dtype())?
.reshape(((), 1))?;
@@ -252,7 +523,7 @@ mod tests {
let block = Qwen3_5MoeBlock {
gate: Linear::new(randn(&[n_exp, hidden]), None),
experts: (0..n_exp).map(|_| rand_mlp(hidden, inter)).collect(),
experts: Experts::PerExpert((0..n_exp).map(|_| rand_mlp(hidden, inter)).collect()),
shared_expert: Some(rand_mlp(hidden, inter)),
shared_expert_gate: Some(Linear::new(randn(&[1, hidden]), None)),
num_experts_per_tok: top_k,
@@ -283,8 +554,9 @@ mod tests {
let mut expect = vec![0f32; hidden];
for &e in selected {
let w = probs[e] / denom;
let out: Vec<f32> = block.experts[e]
.forward(&row)
let out: Vec<f32> = block
.experts
.forward_one(e, &row)
.unwrap()
.flatten_all()
.unwrap()
@@ -336,7 +608,7 @@ mod tests {
let (hidden, inter, n_exp) = (4, 2, 3);
let block = Qwen3_5MoeBlock {
gate: Linear::new(randn(&[n_exp, hidden]), None),
experts: (0..n_exp).map(|_| rand_mlp(hidden, inter)).collect(),
experts: Experts::PerExpert((0..n_exp).map(|_| rand_mlp(hidden, inter)).collect()),
shared_expert: None,
shared_expert_gate: None,
num_experts_per_tok: 1,
@@ -368,8 +640,9 @@ mod tests {
.max_by(|&a, &b| exps[a].partial_cmp(&exps[b]).unwrap())
.unwrap();
let w = exps[best] / sum;
let out: Vec<f32> = block.experts[best]
.forward(&xs.reshape(((), hidden)).unwrap())
let out: Vec<f32> = block
.experts
.forward_one(best, &xs.reshape(((), hidden)).unwrap())
.unwrap()
.flatten_all()
.unwrap()
@@ -383,4 +656,291 @@ mod tests {
);
}
}
/// Banking is a storage change, so the two layouts must compute the
/// same function. Same weights, expressed both ways, asserted
/// identical — otherwise "we only changed where the bytes live" is
/// a claim rather than a fact.
#[test]
fn banked_experts_equal_per_expert_modules() {
let (n_experts, hidden, inter) = (3usize, 4usize, 2usize);
// One source of truth for the weights, laid out both ways.
let gate_up = randn(&[n_experts, inter * 2, hidden]);
let down = randn(&[n_experts, hidden, inter]);
let per_expert: Vec<Qwen3_5MLP> = (0..n_experts)
.map(|e| {
let gu = gate_up.i(e).unwrap();
Qwen3_5MLP::from_weights(
Linear::new(gu.narrow(0, 0, inter).unwrap().contiguous().unwrap(), None),
Linear::new(
gu.narrow(0, inter, inter).unwrap().contiguous().unwrap(),
None,
),
Linear::new(down.i(e).unwrap().contiguous().unwrap(), None),
)
})
.collect();
let sliced = Experts::PerExpert(per_expert);
let banked = Experts::Banked {
gate_up,
down,
intermediate: inter,
};
assert_eq!(sliced.len(), banked.len());
let xs = randn(&[5, hidden]);
for e in 0..n_experts {
let a: Vec<f32> = sliced
.forward_one(e, &xs)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
let b: Vec<f32> = banked
.forward_one(e, &xs)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
assert_eq!(a.len(), b.len());
for (x, y) in a.iter().zip(b.iter()) {
assert!(
(x - y).abs() < 1e-5,
"expert {e} differs between layouts: {a:?} vs {b:?}"
);
}
}
}
/// And the halves are not interchangeable: reading `gate_up` the
/// other way round computes silu(up) * gate, which the banked path
/// would do just as silently as the sliced one.
#[test]
fn banked_reads_the_gate_before_the_up() {
let dev = Device::Cpu;
// gate row picks channel 0, up row picks channel 1.
let gate_up = Tensor::from_vec(vec![1.0f32, 0.0, 0.0, 1.0], (1, 2, 2), &dev).unwrap();
let down = Tensor::from_vec(vec![1.0f32, 0.0], (1, 2, 1), &dev).unwrap();
let banked = Experts::Banked {
gate_up,
down,
intermediate: 1,
};
let xs = Tensor::from_vec(vec![1.0f32, 2.0], (1, 2), &dev).unwrap();
let got: Vec<f32> = banked
.forward_one(0, &xs)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
// silu(gate.x) * (up.x) = silu(1) * 2
let want = (1.0f32 / (1.0 + (-1.0f32).exp())) * 2.0;
assert!((got[0] - want).abs() < 1e-5, "got {got:?} want {want}");
// the swapped reading would be silu(2) * 1
let swapped = (2.0f32 / (1.0 + (-2.0f32).exp())) * 1.0;
assert!(
(got[0] - swapped).abs() > 1e-3,
"this is silu(up) * gate: {got:?}"
);
}
/// In-situ quantisation has to approximate the dense expert, not
/// merely produce numbers of the right shape.
///
/// Dimensions are 256 because the k-quants block on 256 elements —
/// a tiny fixture silently cannot be quantised at all, which is the
/// first thing that bites when this is tried on a toy model.
///
/// Experts restored from the on-disk cache compute the same
/// function as the ones that filled it (#322).
///
/// The cache's whole risk is that it returns something plausible:
/// the model loads, it serves, and its weights are quietly not the
/// ones it was asked for. So this asserts the restored experts are
/// *bit-identical* in output, not close — there is no arithmetic
/// between writing and reading that could legitimately differ.
///
/// At the checkpoint's real widths, because they are what decide
/// the types involved: 2560 blocks under Q4K, 640 does not and
/// falls back to Q5_0, so this exercises a layer whose halves are
/// *different* quantisation types — exactly the case a container
/// storing one type per file could not represent, and the reason
/// the artifact is GGUF.
#[test]
fn cached_experts_compute_the_same_function() {
let dev = Device::Cpu;
let (n_experts, hidden, inter) = (2usize, 2560usize, 640usize);
let gate_up = randn(&[n_experts, inter * 2, hidden]);
let down = randn(&[n_experts, hidden, inter]);
let fresh = Experts::quantize_banked(&gate_up, &down, inter, GgmlDType::Q4K).unwrap();
let entries = fresh
.cache_entries()
.expect("quantised experts must be cacheable");
// The two halves really did take different types; if they had
// not, this test would not be covering what it claims to.
let dtypes: std::collections::HashSet<GgmlDType> =
entries.iter().map(|(_, t)| t.dtype()).collect();
assert!(
dtypes.len() > 1,
"expected a mix of quant types across gate/up and down, got {dtypes:?}"
);
let dir = tempfile::tempdir().unwrap();
let cache =
crate::harness::isq_cache::IsqCache::new(dir.path(), "test/model", Some("rev0"), "q4k")
.unwrap();
let refs: Vec<(&str, &QTensor)> = entries
.iter()
.map(|(n, t)| (n.as_str(), t.as_ref()))
.collect();
cache
.try_store("layer-0", &refs)
.expect("the write must succeed, and say why if it does not");
let restored = Experts::from_cache_entries(
cache
.load("layer-0", &dev)
.expect("the artifact just written"),
n_experts,
)
.unwrap();
let xs = randn(&[3, hidden]);
for e in 0..n_experts {
let a: Vec<f32> = fresh
.forward_one(e, &xs)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
let b: Vec<f32> = restored
.forward_one(e, &xs)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
assert_eq!(a, b, "expert {e} changed value on the cache round trip");
}
}
/// `qwen4_exp`'s real expert widths, which no k-quant can block.
///
/// `moe_intermediate_size` is 640, so `down_proj` is `[2560, 640]`
/// and its 640-element rows are 2.5 blocks of 256. The load died
/// on beast at decoder layer 0 with `quantized tensor must have
/// their last dim divisible by block size [2560, 640] 256`, after
/// resolving all 131 shards.
///
/// The fixture that missed it was 256 wide in both dimensions —
/// the smallest width a k-quant accepts — so every row divided
/// and the orientation could not matter. This one uses the
/// checkpoint's own numbers, scaled down only in expert count.
#[test]
fn the_real_expert_widths_quantise_despite_a_640_wide_reduction() {
assert_eq!(
quant_for_row(GgmlDType::Q4K, 2560).unwrap(),
GgmlDType::Q4K,
"gate_up reduces over hidden_size, which is 10 whole blocks"
);
assert_eq!(
quant_for_row(GgmlDType::Q4K, 640).unwrap(),
GgmlDType::Q5_0,
"down reduces over moe_intermediate_size, which is not"
);
// llama.cpp's `tensor_type_fallback` table, which spends bits
// rather than saving them: a legacy 32-block type is worse than
// a k-quant at equal width, so upstream pays one to avoid the
// quality cliff. Q4_0 above would be 4.5 bpw like Q4K and would
// save 5.0 GB across the expert bank — that is the trade we
// declined, and this is where it is pinned.
assert_eq!(quant_for_row(GgmlDType::Q5K, 640).unwrap(), GgmlDType::Q5_1);
assert_eq!(quant_for_row(GgmlDType::Q2K, 640).unwrap(), GgmlDType::Q4_0);
assert_eq!(quant_for_row(GgmlDType::Q3K, 640).unwrap(), GgmlDType::Q4_0);
assert_eq!(quant_for_row(GgmlDType::Q6K, 640).unwrap(), GgmlDType::Q8_0);
assert_eq!(
quant_for_row(GgmlDType::Q8_0, 640).unwrap(),
GgmlDType::Q8_0
);
// 100 divides neither 256 nor 32: F16, as upstream does.
assert_eq!(quant_for_row(GgmlDType::Q4K, 100).unwrap(), GgmlDType::F16);
// And the whole path runs at those widths.
let (n_experts, hidden, inter) = (2usize, 2560usize, 640usize);
let gate_up = randn(&[n_experts, inter * 2, hidden]);
let down = randn(&[n_experts, hidden, inter]);
let quantised = Experts::quantize_banked(&gate_up, &down, inter, GgmlDType::Q4K)
.expect("the real widths must quantise");
let xs = randn(&[2, hidden]);
let got = quantised.forward_one(0, &xs).expect("forward");
assert_eq!(got.dims(), &[2, hidden]);
assert!(
got.flatten_all()
.unwrap()
.to_vec1::<f32>()
.unwrap()
.iter()
.all(|v| v.is_finite()),
"a substituted dtype must still produce finite output"
);
}
/// The assertion is relative error against the dense output, with a
/// noise baseline: a q8_0 expert should be close, a q4k expert
/// looser, and both far nearer the truth than an unrelated expert
/// is. Without that last comparison a "quantised" path that
/// returned anything smoothly wrong would pass.
#[test]
fn quantised_experts_approximate_the_dense_ones() {
let (n_experts, hidden, inter) = (2usize, 256usize, 256usize);
let gate_up = randn(&[n_experts, inter * 2, hidden]);
let down = randn(&[n_experts, hidden, inter]);
let banked = Experts::Banked {
gate_up: gate_up.clone(),
down: down.clone(),
intermediate: inter,
};
let xs = randn(&[4, hidden]);
let rel_err = |a: &[f32], b: &[f32]| -> f32 {
let num: f32 = a.iter().zip(b).map(|(x, y)| (x - y) * (x - y)).sum();
let den: f32 = a.iter().map(|x| x * x).sum::<f32>().max(1e-12);
(num / den).sqrt()
};
let out = |e: &Experts, i: usize| -> Vec<f32> {
e.forward_one(i, &xs)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap()
};
let dense0 = out(&banked, 0);
// Baseline: a different expert on the same input. Any quantised
// reading must beat this by a wide margin.
let noise = rel_err(&dense0, &out(&banked, 1));
for (dtype, tol) in [(GgmlDType::Q8_0, 0.05f32), (GgmlDType::Q4K, 0.20f32)] {
let q = Experts::quantize_banked(&gate_up, &down, inter, dtype).unwrap();
assert_eq!(q.len(), n_experts);
let err = rel_err(&dense0, &out(&q, 0));
assert!(err < tol, "{dtype:?} relative error {err:.4} exceeds {tol}");
assert!(
err < noise / 2.0,
"{dtype:?} error {err:.4} is not clearly better than an unrelated \
expert ({noise:.4}) — the quantised path may not be reading \
this expert at all"
);
}
}
}
@@ -197,10 +197,18 @@ impl TextConfig {
.map(|d| d.eq_ignore_ascii_case("float32") || d.eq_ignore_ascii_case("f32"))
}
/// Both spellings upstream uses. The shipped checkpoint's
/// `layer_types` says `full_attention` (36 linear, 12 full), but
/// `Qwen4ExpTextConfig` *derives* the array with
/// `qwen_sparse_attention` when the checkpoint omits it. Matching
/// only the first is a silent misclassification: every layer would
/// read as linear attention, and a model that is 3:1 linear would
/// look almost right until its full-attention tensors were not
/// found.
pub fn is_full_attention(&self, layer: usize) -> bool {
self.layer_types
.get(layer)
.is_some_and(|t| t == "full_attention")
.is_some_and(|t| t == "full_attention" || t == "qwen_sparse_attention")
}
fn fill_layer_types(&mut self) -> Result<()> {
@@ -474,6 +482,35 @@ mod tests {
}
/// A one-indexed field that reads 0 is a config written against the
/// Upstream has two names for the same layer, and we must read both.
///
/// The shipped `config.json` spells the attention layers
/// `full_attention`. `Qwen4ExpTextConfig` derives them as
/// `qwen_sparse_attention` when the checkpoint omits the array —
/// observed directly from `transformers` `main`, which reports
/// `['linear_attention', ..., 'qwen_sparse_attention']` for a config
/// carrying only `full_attention_interval`.
///
/// Reading only the shipped spelling would classify every layer as
/// linear on such a config. That is not a load error at the point it
/// happens — it is a model built with the wrong 12 layers, which
/// fails later and elsewhere.
#[test]
fn both_upstream_spellings_of_a_full_attention_layer_are_read() {
let mut cfg = Config::from_config_json(SHIPPED).unwrap().text_config;
cfg.layer_types = vec!["linear_attention".into(), "full_attention".into()];
assert!(!cfg.is_full_attention(0));
assert!(cfg.is_full_attention(1), "the shipped spelling");
cfg.layer_types = vec!["linear_attention".into(), "qwen_sparse_attention".into()];
assert!(!cfg.is_full_attention(0));
assert!(
cfg.is_full_attention(1),
"the derived spelling must classify as full attention too"
);
}
/// wrong convention, not a layer.
#[test]
fn rejects_a_zero_ple_layer_id() {
@@ -41,6 +41,8 @@ use crate::harness::arch::qwen3_5::linear_attn::GatedDeltaNet;
use crate::harness::arch::qwen3_5::moe::Qwen3_5MoeBlock;
use crate::harness::arch::qwen3_5::rope::RotaryEmbedding;
use candle_core::quantized::GgmlDType;
use super::config::TextConfig;
use super::full_attn::Attention;
use super::hyper::HyperConnection;
@@ -72,7 +74,9 @@ impl DecoderLayer {
cfg: &TextConfig,
rotary: Arc<RotaryEmbedding>,
layer_idx: usize,
quant: Option<GgmlDType>,
vb: &ShardedVarBuilder,
isq_cache: Option<&crate::harness::isq_cache::IsqCache>,
) -> Result<Self> {
let layer_type = cfg
.layer_types
@@ -89,7 +93,9 @@ impl DecoderLayer {
rotary,
layer_type,
cfg.ple_layers().contains(&layer_idx),
quant,
vb,
isq_cache,
)
}
@@ -104,7 +110,9 @@ impl DecoderLayer {
rotary: Arc<RotaryEmbedding>,
layer_type: &str,
with_ple: bool,
quant: Option<GgmlDType>,
vb: &ShardedVarBuilder,
isq_cache: Option<&crate::harness::isq_cache::IsqCache>,
) -> Result<Self> {
let mixer = match layer_type {
"full_attention" => {
@@ -150,7 +158,7 @@ impl DecoderLayer {
attn_hc: hc("attn_hyper_connection")?,
mlp_hc: hc("mlp_hyper_connection")?,
mixer,
mlp: moe::load(cfg, &vb.pp("mlp"))?,
mlp: moe::load(cfg, quant, &vb.pp("mlp"), isq_cache)?,
ple,
})
}
@@ -34,6 +34,7 @@
//! See `doc/qwen4_exp-port-spec.md`.
use anyhow::{Context, Result, ensure};
use candle_core::quantized::GgmlDType;
use candle_core::{DType, Device, IndexOp, Module, Tensor};
use candle_nn::var_builder::ShardedVarBuilder;
use candle_nn::{Embedding, Linear};
@@ -44,7 +45,7 @@ use crate::harness::arch::qwen3_5::rope::RotaryEmbedding;
use super::config::{Config, TextConfig};
use super::decoder::DecoderLayer;
use super::hyper::HyperConnection;
use super::ple::{NGramHasher, NGramTable, ShardedNGramTable};
use super::ple::{MmapNGramTable, NGramHasher, NGramTable, ShardedNGramTable};
use crate::harness::arch::snapshot::{KvCacheSnapshot, PleSnapshot};
/// The hashed n-gram lookup, hoisted to the model because it is a
@@ -124,11 +125,23 @@ pub struct Qwen4ExpForCausalLM {
}
impl Qwen4ExpForCausalLM {
/// `quant` quantises the routed experts in situ (#315). `None`
/// keeps the checkpoint's precision, which for this architecture is
/// useful only on a model small enough to hold: 241.6 GB of BF16
/// experts fit neither 64 GB of VRAM nor 123 GB of host RAM.
/// `safetensors_paths` is the checkpoint's own file set, used only
/// to mmap the PLE n-gram table where it already lies (#310). Empty
/// falls back to [`ShardedNGramTable`], which puts 102.4 GB
/// wherever `vb` points — the oracle, not a residency to serve
/// from.
pub fn load(
cfg: &Config,
dtype: DType,
quant: Option<GgmlDType>,
device: &Device,
vb: &ShardedVarBuilder,
safetensors_paths: &[std::path::PathBuf],
isq_cache: Option<&crate::harness::isq_cache::IsqCache>,
) -> Result<Self> {
let text = &cfg.text_config;
// `RotaryEmbedding` reads a qwen3_5 config; the rope block is the
@@ -144,10 +157,29 @@ impl Qwen4ExpForCausalLM {
let embed_tokens = Embedding::new(embed_weight, text.hidden_size);
let layers_vb = root.pp("layers");
// Before the layers, not after them. The table is mapped rather
// than made resident (#310), so its position in the load order
// costs nothing — but everything it can get wrong is a name or
// an extent, and those are worth finding in the second before
// the expert loop rather than the half-hour after it. Loading
// it last meant a checkpoint whose shard names had moved would
// report that only once 48 layers of experts had been
// quantised, or — since they do not currently fit — never.
let ngram = match text.ple_layers().first() {
Some(layer) => Some(load_ngram(
text,
&layers_vb.pp(*layer).pp("ple"),
safetensors_paths,
device,
)?),
None => None,
};
let mut layers = Vec::with_capacity(text.num_hidden_layers);
for i in 0..text.num_hidden_layers {
layers.push(
DecoderLayer::load(text, rotary.clone(), i, &layers_vb.pp(i))
DecoderLayer::load(text, rotary.clone(), i, quant, &layers_vb.pp(i), isq_cache)
.with_context(|| format!("load decoder layer {i}"))?,
);
}
@@ -169,11 +201,6 @@ impl Qwen4ExpForCausalLM {
.get((text.vocab_size, text.hidden_size), "weight")
.context("load 'lm_head.weight'")?;
let ngram = match text.ple_layers().first() {
Some(layer) => Some(load_ngram(text, &layers_vb.pp(*layer).pp("ple"))?),
None => None,
};
Ok(Self {
embed_tokens,
layers,
@@ -350,7 +377,12 @@ impl Qwen4ExpForCausalLM {
}
}
fn load_ngram(cfg: &TextConfig, ple_vb: &ShardedVarBuilder) -> Result<NGramEmbedding> {
fn load_ngram(
cfg: &TextConfig,
ple_vb: &ShardedVarBuilder,
safetensors_paths: &[std::path::PathBuf],
device: &Device,
) -> Result<NGramEmbedding> {
let vb = ple_vb.pp("ple_embedding");
let heads = cfg.ngram_heads();
let head_dim = cfg.ngram_head_dim();
@@ -376,12 +408,28 @@ fn load_ngram(cfg: &TextConfig, ple_vb: &ShardedVarBuilder) -> Result<NGramEmbed
cfg.split_ngram_parts
);
let table = ShardedNGramTable::load(
&vb.pp("ngram_embedding"),
cfg.split_ngram_parts,
rows,
head_dim,
)?;
// The shard names are whatever this VarBuilder would have asked
// for, so the prefix is taken from it rather than rebuilt from the
// layer index — the two cannot disagree that way, and getting it
// wrong is a 102.4 GB read of the wrong thing.
let ngram_vb = vb.pp("ngram_embedding");
let table: Box<dyn NGramTable> = if safetensors_paths.is_empty() {
Box::new(ShardedNGramTable::load(
&ngram_vb,
cfg.split_ngram_parts,
rows,
head_dim,
)?)
} else {
Box::new(MmapNGramTable::open(
safetensors_paths,
&ngram_vb.prefix(),
cfg.split_ngram_parts,
rows,
head_dim,
device,
)?)
};
Ok(NGramEmbedding {
hasher: NGramHasher::new(
@@ -392,7 +440,7 @@ fn load_ngram(cfg: &TextConfig, ple_vb: &ShardedVarBuilder) -> Result<NGramEmbed
offsets,
cfg.eos_token_id,
)?,
table: Box::new(table),
table,
heads,
head_dim,
carried: Vec::new(),
@@ -751,7 +799,19 @@ mod tests {
candle_core::safetensors::save(&tensors(&cfg.text_config), &path).unwrap();
let vb =
unsafe { ShardedSafeTensors::var_builder(&[&path], DType::F32, &Device::Cpu).unwrap() };
let model = Qwen4ExpForCausalLM::load(&cfg, DType::F32, &Device::Cpu, &vb).unwrap();
// Passing the path exercises the mmap table (#310), which is
// the residency production uses — the whole suite then runs
// against the gather that actually serves, not the oracle.
let model = Qwen4ExpForCausalLM::load(
&cfg,
DType::F32,
None,
&Device::Cpu,
&vb,
std::slice::from_ref(&path),
None,
)
.unwrap();
(dir, model)
}
@@ -1015,7 +1075,7 @@ mod tests {
let vb =
unsafe { ShardedSafeTensors::var_builder(&[&path], DType::F32, &Device::Cpu).unwrap() };
let rotary = Arc::new(rotary_for(&cfg.text_config, DType::F32, &Device::Cpu).unwrap());
let mut head = MtpHead::load(&cfg.text_config, rotary.clone(), &vb).unwrap();
let mut head = MtpHead::load(&cfg.text_config, rotary.clone(), None, &vb).unwrap();
let prompt = Tensor::from_vec(vec![5u32, 9, 3, 12], (1, 4), &Device::Cpu).unwrap();
let (_logits, hidden) = model.forward_with_hidden(&prompt, 0).unwrap();
@@ -1072,7 +1132,7 @@ mod tests {
let vb =
unsafe { ShardedSafeTensors::var_builder(&[&path], DType::F32, &Device::Cpu).unwrap() };
let rotary = Arc::new(rotary_for(&cfg.text_config, DType::F32, &Device::Cpu).unwrap());
let mut head = MtpHead::load(&cfg.text_config, rotary.clone(), &vb).unwrap();
let mut head = MtpHead::load(&cfg.text_config, rotary.clone(), None, &vb).unwrap();
let h = cfg.text_config.hidden_size;
let collapsed = Tensor::zeros((1, 1, h), DType::F32, &Device::Cpu).unwrap();
+102 -79
View File
@@ -22,26 +22,46 @@
//! of error that produces a model that talks and reasons slightly
//! worse. It is asserted below rather than commented.
//!
//! The fused layout is the better one for a grouped GEMM, and keeping
//! it would be the faster path; that is an algorithm change, and the
//! spec is explicit that this stage is not it. Sparsity is extreme —
//! 10 of 512 with an intermediate of 640 — so the routing, not the
//! GEMM shape, is what dominates today.
//! ## Why the fused tensors stay fused
//!
//! Slicing them into 512 per-expert modules copies **241.6 GB** at
//! BF16, 5.03 GB per layer. That fits in neither beast's 64 GB of VRAM
//! nor its 123 GB of host RAM, and no quantisation rescues it: #309
//! measured 59.4 GB of 63.7 used at 4.25 bpw with only *four* layers'
//! experts offloaded. A model that cannot be held cannot be loaded.
//!
//! So the block takes `Experts::Banked` and views one expert per
//! routed token instead. A slice along dim 0 of a contiguous tensor is
//! contiguous, and `x.matmul(w.t())` is what `Linear` does anyway, so
//! the views feed the same GEMMs with no copy. It is also the layout a
//! host-resident bank with a device-side gather wants (#318).
//!
//! See `doc/qwen4_exp-port-spec.md` §6.
use anyhow::{Context, Result};
use candle_core::{IndexOp, Tensor};
use candle_core::Tensor;
use candle_core::quantized::GgmlDType;
use candle_nn::Linear;
use candle_nn::var_builder::ShardedVarBuilder;
use crate::harness::arch::qwen3_5::mlp::Qwen3_5MLP;
use crate::harness::arch::qwen3_5::moe::Qwen3_5MoeBlock;
use crate::harness::arch::qwen3_5::moe::{Experts, Qwen3_5MoeBlock};
use super::config::TextConfig;
/// `vb` should be `.pp(...)`-ed to the layer's `mlp` prefix.
pub fn load(cfg: &TextConfig, vb: &ShardedVarBuilder) -> Result<Qwen3_5MoeBlock> {
///
/// `quant` quantises the routed experts in situ (#315). Without it the
/// experts stay banked at the checkpoint's precision, which is only
/// useful for a model small enough to hold — 120.8 B routed parameters
/// are 241.6 GB at BF16 and fit nowhere on this hardware, so a real
/// load of `Qwen3.8-Flash-Next` always passes one.
pub fn load(
cfg: &TextConfig,
quant: Option<GgmlDType>,
vb: &ShardedVarBuilder,
isq_cache: Option<&crate::harness::isq_cache::IsqCache>,
) -> Result<Qwen3_5MoeBlock> {
let (h, inter) = (cfg.hidden_size, cfg.moe_intermediate_size);
anyhow::ensure!(
cfg.num_experts > 0 && cfg.num_experts_per_tok > 0 && inter > 0,
@@ -59,13 +79,64 @@ pub fn load(cfg: &TextConfig, vb: &ShardedVarBuilder) -> Result<Qwen3_5MoeBlock>
);
let experts_vb = vb.pp("experts");
let gate_up = experts_vb
.get((cfg.num_experts, inter * 2, h), "gate_up_proj")
.with_context(|| format!("load '{}/gate_up_proj'", experts_vb.prefix()))?;
let down = experts_vb
.get((cfg.num_experts, h, inter), "down_proj")
.with_context(|| format!("load '{}/down_proj'", experts_vb.prefix()))?;
let experts = split_fused_experts(&gate_up, &down, inter)?;
// The cache key is the layer's own tensor prefix, so it cannot
// collide across layers and needs no separate index to be passed
// down alongside it.
let cache_key = experts_vb.prefix().replace(['.', '/'], "-");
// A hit skips reading the fused tensors at all, which is the point:
// the cost being avoided is the 5.03 GB read *and* the ~32 s of
// quantisation, not just the latter.
let cached =
match (quant, isq_cache) {
(Some(_), Some(cache)) => cache.load(&cache_key, vb.device()).and_then(|entries| {
match Experts::from_cache_entries(entries, cfg.num_experts) {
Ok(e) => Some(e),
Err(e) => {
tracing::warn!(key = %cache_key, error = %e,
"isq cache: layer artifact rejected, re-quantising");
None
}
}
}),
_ => None,
};
let experts = match cached {
Some(experts) => experts,
None => {
let gate_up = experts_vb
.get((cfg.num_experts, inter * 2, h), "gate_up_proj")
.with_context(|| format!("load '{}/gate_up_proj'", experts_vb.prefix()))?;
let down = experts_vb
.get((cfg.num_experts, h, inter), "down_proj")
.with_context(|| format!("load '{}/down_proj'", experts_vb.prefix()))?;
check_fused_experts(&gate_up, &down, inter)?;
// The fused pair is the peak: 5.03 GB per layer at BF16.
// Quantising here and letting it drop at the end of this
// function is what keeps the whole 241.6 GB from ever
// existing.
match quant {
Some(dtype) => {
let experts = Experts::quantize_banked(&gate_up, &down, inter, dtype)
.with_context(|| format!("quantize experts to {dtype:?}"))?;
if let (Some(cache), Some(entries)) = (isq_cache, experts.cache_entries()) {
let refs: Vec<(&str, &candle_core::quantized::QTensor)> = entries
.iter()
.map(|(n, t)| (n.as_str(), t.as_ref()))
.collect();
cache.store(&cache_key, &refs);
}
experts
}
None => Experts::Banked {
gate_up,
down,
intermediate: inter,
},
}
}
};
let (shared_expert, shared_expert_gate) = if cfg.shared_expert_intermediate_size > 0 {
let shared = Qwen3_5MLP::load_with_dims(
@@ -93,14 +164,17 @@ pub fn load(cfg: &TextConfig, vb: &ShardedVarBuilder) -> Result<Qwen3_5MoeBlock>
))
}
/// Slice `(E, 2I, H)` and `(E, H, I)` into `E` SwiGLU experts.
/// Validate the fused pair before it is banked.
///
/// Gate first, up second — see the module note.
pub(crate) fn split_fused_experts(
/// The block will take views of these per routed expert, so a
/// disagreement here surfaces as a shape error mid-forward on whichever
/// token first routes to the offending expert — which is a worse place
/// to find it than at load.
pub(crate) fn check_fused_experts(
gate_up: &Tensor,
down: &Tensor,
intermediate: usize,
) -> Result<Vec<Qwen3_5MLP>> {
) -> Result<()> {
let (experts, fused, _) = gate_up.dims3()?;
anyhow::ensure!(
fused == intermediate * 2,
@@ -111,78 +185,27 @@ pub(crate) fn split_fused_experts(
"down_proj holds {} experts, gate_up_proj holds {experts}",
down.dims3()?.0
);
let mut out = Vec::with_capacity(experts);
for e in 0..experts {
let fused_e = gate_up.i(e)?;
out.push(Qwen3_5MLP::from_weights(
Linear::new(fused_e.narrow(0, 0, intermediate)?.contiguous()?, None),
Linear::new(
fused_e
.narrow(0, intermediate, intermediate)?
.contiguous()?,
None,
),
Linear::new(down.i(e)?.contiguous()?, None),
));
}
Ok(out)
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use candle_core::{DType, Device, Module};
use candle_core::{DType, Device};
/// One expert, two hidden dims, one intermediate. The gate row picks
/// the first input channel and the up row picks the second, so
/// swapping them swaps which channel is squashed by the SiLU — and
/// the two answers differ by more than any tolerance.
/// The loader validates the fused pair up front. Without this the
/// first token to route to a bad expert fails mid-forward, which is
/// a worse place to learn the checkpoint disagrees with the config.
#[test]
fn the_gate_is_the_first_half_of_the_fused_tensor() {
let dev = Device::Cpu;
// gate_up: [[gate row], [up row]] = [[1, 0], [0, 1]]
let gate_up = Tensor::from_vec(vec![1.0f32, 0.0, 0.0, 1.0], (1, 2, 2), &dev).unwrap();
// down: (E=1, H=2, I=1)
let down = Tensor::from_vec(vec![2.0f32, 3.0], (1, 2, 1), &dev).unwrap();
let experts = split_fused_experts(&gate_up, &down, 1).unwrap();
assert_eq!(experts.len(), 1);
let x = Tensor::from_vec(vec![1.0f32, 2.0], (1, 2), &dev).unwrap();
let got: Vec<f32> = experts[0]
.forward(&x)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
// gate(x) = 1, up(x) = 2 -> silu(1) * 2 -> down
let silu1 = 1.0f32 / (1.0 + (-1.0f32).exp());
let want = [2.0 * silu1 * 2.0, 3.0 * silu1 * 2.0];
for (g, w) in got.iter().zip(want.iter()) {
assert!((g - w).abs() < 1e-5, "got {got:?} want {want:?}");
}
// The swapped reading: silu(2) * 1, which is a different model.
let silu2 = 2.0f32 / (1.0 + (-2.0f32).exp());
assert!(
(got[0] - 2.0 * silu2).abs() > 1e-3,
"this is silu(up) * gate, not silu(gate) * up: {got:?}"
);
}
#[test]
fn a_fused_tensor_of_the_wrong_width_is_rejected() {
fn a_fused_pair_that_disagrees_with_the_config_is_rejected() {
let dev = Device::Cpu;
let gate_up = Tensor::zeros((2, 6, 4), DType::F32, &dev).unwrap();
let down = Tensor::zeros((2, 4, 3), DType::F32, &dev).unwrap();
assert!(split_fused_experts(&gate_up, &down, 3).is_ok());
assert!(check_fused_experts(&gate_up, &down, 3).is_ok());
// 2 x 4 != 6
assert!(split_fused_experts(&gate_up, &down, 4).is_err());
assert!(check_fused_experts(&gate_up, &down, 4).is_err());
// expert counts must agree
let down_short = Tensor::zeros((1, 4, 3), DType::F32, &dev).unwrap();
assert!(split_fused_experts(&gate_up, &down_short, 3).is_err());
assert!(check_fused_experts(&gate_up, &down_short, 3).is_err());
}
}
@@ -46,6 +46,7 @@
//! See `doc/qwen4_exp-port-spec.md` §9 and #313.
use anyhow::{Context, Result};
use candle_core::quantized::GgmlDType;
use candle_core::{D, Module, Tensor};
use candle_nn::Linear;
use candle_nn::var_builder::ShardedVarBuilder;
@@ -87,6 +88,7 @@ impl MtpHead {
pub fn load(
cfg: &TextConfig,
rotary: Arc<RotaryEmbedding>,
quant: Option<GgmlDType>,
vb: &ShardedVarBuilder,
) -> Result<Self> {
let mtp = vb.pp("mtp");
@@ -111,7 +113,12 @@ impl MtpHead {
rotary,
"full_attention",
false,
quant,
&mtp.pp("layers").pp(0),
// The MTP head is not on the serving path (#313), so it
// has nothing to gain from the cache and nothing to
// stale it.
None,
)
.context("load mtp.layers.0")?,
mixer: HyperConnection::load(
+293 -1
View File
@@ -66,7 +66,7 @@
//! reimplement. See `doc/qwen4_exp-port-spec.md` §2.
use anyhow::{Context, Result, ensure};
use candle_core::{D, Module, Tensor};
use candle_core::{D, DType, Device, Module, Tensor};
use candle_nn::Linear;
use candle_nn::var_builder::ShardedVarBuilder;
@@ -337,6 +337,216 @@ impl NGramTable for ShardedNGramTable {
}
}
/// The table read straight out of the checkpoint by mmap (#310).
///
/// The shipping residency. Nothing is made resident anywhere: the 128
/// shard tensors are mapped where they already lie in the checkpoint's
/// safetensors files, and a gather copies the requested rows out of
/// the mapping. The page cache decides what stays warm, which for this
/// access pattern is the right decision-maker — a token needs sixteen
/// rows of 160 values, about 5 KB, out of 102.4 GB, so almost the whole
/// table is cold at any instant and nothing is served by making it
/// otherwise.
///
/// This is what makes the model loadable at all. [`ShardedNGramTable`]
/// puts those 102.4 GB wherever the `VarBuilder` points, which for a
/// CUDA load is a 32 GB card; it stays as the oracle this is diffed
/// against, not as a residency anyone would choose.
///
/// Mapping the checkpoint's own tensors in place — rather than building
/// a separate artifact — is FreeToken's `source_from_safetensors`
/// trick, and the reason it works here is that the shards are already
/// exactly what we want: contiguous, equal-stride, row-major, one
/// extent each. There is nothing to convert and nothing to keep in
/// sync.
///
/// Unlike the sharded gather this needs no sort-and-permute: rows are
/// copied in the order asked for, so the inverse permutation that has
/// to be right there cannot be wrong here.
pub struct MmapNGramTable {
/// One mapping per checkpoint file that holds shards. Kept for the
/// table's lifetime because every `gather` reads through them.
maps: Vec<memmap2::Mmap>,
/// Byte position of each shard's row 0: which mapping, and where in
/// it. Resolved once at load so a gather is arithmetic.
shards: Vec<ShardExtent>,
rows_per_shard: usize,
head_dim: usize,
dtype: DType,
row_bytes: usize,
device: Device,
}
/// Where one shard's bytes live.
struct ShardExtent {
map: usize,
offset: usize,
}
impl MmapNGramTable {
/// `prefix` is the full tensor-name prefix of the shards, e.g.
/// `model.language_model.layers.1.ple.ple_embedding.ngram_embedding`
/// — the shards are then `{prefix}.shard_{s}.weight`.
///
/// `files` is the checkpoint's safetensors set; shards may be
/// spread across any of them (in the real checkpoint, 33 of the
/// 131 files carry them), so each is opened and asked what it has.
pub fn open(
files: &[std::path::PathBuf],
prefix: &str,
parts: usize,
rows: usize,
head_dim: usize,
device: &Device,
) -> Result<Self> {
ensure!(parts > 0, "split_ngram_parts must be > 0");
ensure!(
rows.is_multiple_of(parts),
"{rows} rows do not divide evenly into {parts} shards"
);
let rows_per_shard = rows / parts;
let mut maps: Vec<memmap2::Mmap> = Vec::new();
let mut found: Vec<Option<ShardExtent>> = (0..parts).map(|_| None).collect();
let mut dtype: Option<DType> = None;
for path in files {
let file = std::fs::File::open(path)
.with_context(|| format!("open {} for the n-gram table", path.display()))?;
// SAFETY: the same contract candle's own safetensors
// loading takes — the checkpoint is not modified while we
// serve from it. A truncated or rewritten file underneath a
// live mapping is a corrupt-model problem, not one this can
// defend against.
let map = unsafe { memmap2::Mmap::map(&file) }
.with_context(|| format!("mmap {}", path.display()))?;
let (header_len, meta) =
safetensors::SafeTensors::read_metadata(&map).map_err(|e| {
anyhow::anyhow!("read safetensors metadata from {}: {e}", path.display())
})?;
// `data_offsets` are relative to the start of the data
// section, which follows the 8-byte header length and the
// header itself.
let data_start = 8 + header_len;
let mut used = false;
for (s, slot) in found.iter_mut().enumerate() {
if slot.is_some() {
continue;
}
let name = format!("{prefix}.shard_{s}.weight");
let Some(info) = meta.info(&name) else {
continue;
};
ensure!(
info.shape == [rows_per_shard, head_dim],
"'{name}' is {:?}, expected [{rows_per_shard}, {head_dim}]",
info.shape
);
let dt = st_dtype(info.dtype)
.with_context(|| format!("'{name}' has a dtype the gather cannot read"))?;
match dtype {
None => dtype = Some(dt),
Some(seen) => ensure!(
seen == dt,
"shard dtypes disagree: {seen:?} then {dt:?} at '{name}'"
),
}
*slot = Some(ShardExtent {
map: maps.len(),
offset: data_start + info.data_offsets.0,
});
used = true;
}
if used {
maps.push(map);
}
}
let dtype = dtype.context("no n-gram shard was found in any checkpoint file")?;
let shards = found
.into_iter()
.enumerate()
.map(|(s, e)| e.with_context(|| format!("'{prefix}.shard_{s}.weight' is in no file")))
.collect::<Result<Vec<_>>>()?;
let row_bytes = head_dim * dtype.size_in_bytes();
// Every extent must actually contain its rows: a header that
// disagrees with the file length would otherwise surface as a
// panic inside a gather, mid-request.
for (s, extent) in shards.iter().enumerate() {
let end = extent.offset + rows_per_shard * row_bytes;
ensure!(
end <= maps[extent.map].len(),
"shard {s} claims bytes {}..{end} of a {}-byte file",
extent.offset,
maps[extent.map].len()
);
}
tracing::info!(
parts,
rows,
head_dim,
?dtype,
files = maps.len(),
gib = (rows * row_bytes) as f64 / (1024.0 * 1024.0 * 1024.0),
"ple: n-gram table mapped from the checkpoint, not made resident (#310)"
);
Ok(Self {
maps,
shards,
rows_per_shard,
head_dim,
dtype,
row_bytes,
device: device.clone(),
})
}
}
impl NGramTable for MmapNGramTable {
fn gather(&self, rows: &[i64]) -> Result<Tensor> {
let total = (self.shards.len() * self.rows_per_shard) as i64;
if let Some(bad) = rows.iter().find(|r| **r < 0 || **r >= total) {
anyhow::bail!("row {bad} is outside the table's 0..{total}");
}
let mut out = vec![0u8; rows.len() * self.row_bytes];
for (slot, &row) in rows.iter().enumerate() {
let shard = row as usize / self.rows_per_shard;
let within = row as usize % self.rows_per_shard;
let extent = &self.shards[shard];
let from = extent.offset + within * self.row_bytes;
out[slot * self.row_bytes..(slot + 1) * self.row_bytes]
.copy_from_slice(&self.maps[extent.map][from..from + self.row_bytes]);
}
Ok(Tensor::from_raw_buffer(
&out,
self.dtype,
&[rows.len(), self.head_dim],
&self.device,
)?)
}
fn head_dim(&self) -> usize {
self.head_dim
}
}
/// safetensors' dtype vocabulary is wider than candle's. Only the
/// float widths a checkpoint would store an embedding table in are
/// accepted; anything else is a checkpoint we do not understand rather
/// than one to guess at.
fn st_dtype(dtype: safetensors::Dtype) -> Result<DType> {
Ok(match dtype {
safetensors::Dtype::BF16 => DType::BF16,
safetensors::Dtype::F16 => DType::F16,
safetensors::Dtype::F32 => DType::F32,
other => anyhow::bail!("unsupported n-gram table dtype {other:?}"),
})
}
/// The consumption half of PLE: `Qwen4ExpTextPLE` minus the table.
///
/// Given the gathered n-gram embedding `e` and the four-stream residual
@@ -903,6 +1113,88 @@ mod tests {
// ---- the table ----
/// The mmap table against the device-resident one, row for row.
///
/// This is the differential #310 asks for: [`ShardedNGramTable`] is
/// the oracle — obviously correct, and unusable at 102.4 GB — and
/// [`MmapNGramTable`] is what serves. They must agree on every row
/// of every shard, in whatever order they are asked, or the model
/// quietly reads somebody else's n-gram.
///
/// The row list is chosen to break the things that can break: shard
/// boundaries in both directions, the first and last row of the
/// table, a repeat, and an order that is not sorted (the oracle
/// sorts internally and inverts the permutation; the mmap path does
/// not, so a disagreement here is exactly that inverse being wrong
/// on one side).
#[test]
fn the_mapped_table_agrees_with_the_resident_one() {
let dev = Device::Cpu;
let (parts, rows_per_shard, head_dim) = (4usize, 5usize, 3usize);
let prefix = "model.language_model.layers.1.ple.ple_embedding.ngram_embedding";
// Row r holds [r, r + 0.5, r + 0.25] — distinct per row, and
// not a pattern a stride mistake would reproduce.
let mut tensors = std::collections::HashMap::new();
for s in 0..parts {
let mut v = Vec::with_capacity(rows_per_shard * head_dim);
for r in 0..rows_per_shard {
let global = (s * rows_per_shard + r) as f32;
v.extend_from_slice(&[global, global + 0.5, global + 0.25]);
}
tensors.insert(
format!("{prefix}.shard_{s}.weight"),
Tensor::from_vec(v, (rows_per_shard, head_dim), &dev).unwrap(),
);
}
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("model.safetensors");
candle_core::safetensors::save(&tensors, &path).unwrap();
let vb = unsafe {
candle_nn::var_builder::ShardedSafeTensors::var_builder(&[&path], DType::F32, &dev)
.unwrap()
};
let total = parts * rows_per_shard;
let oracle = ShardedNGramTable::load(&vb.pp(prefix), parts, total, head_dim).unwrap();
let mapped = MmapNGramTable::open(
std::slice::from_ref(&path),
prefix,
parts,
total,
head_dim,
&dev,
)
.unwrap();
assert_eq!(mapped.head_dim(), oracle.head_dim());
let asked: Vec<i64> = vec![19, 0, 5, 4, 9, 10, 19, 1, 15, 14, 7, 7];
let flat = |t: Tensor| -> Vec<f32> { t.flatten_all().unwrap().to_vec1().unwrap() };
let from_oracle = flat(oracle.gather(&asked).unwrap());
let from_mapped = flat(mapped.gather(&asked).unwrap());
assert_eq!(
from_mapped, from_oracle,
"the mapped gather disagrees with the resident one"
);
// And that both are right, rather than wrong together.
assert_eq!(&from_mapped[..3], &[19.0, 19.5, 19.25]);
assert_eq!(&from_mapped[3..6], &[0.0, 0.5, 0.25]);
// Every row of the table, in order, so no shard is untouched.
let all: Vec<i64> = (0..total as i64).collect();
assert_eq!(
flat(mapped.gather(&all).unwrap()),
flat(oracle.gather(&all).unwrap())
);
// The empty gather and the out-of-range guard behave alike.
assert_eq!(mapped.gather(&[]).unwrap().dims(), &[0, head_dim]);
assert!(mapped.gather(&[total as i64]).is_err());
assert!(mapped.gather(&[-1]).is_err());
}
/// Rows come back in the order asked for, not the order they were
/// fetched in. Two shards of three rows, requested out of order and
/// across the boundary: the shard-major gather returns them sorted,
@@ -134,7 +134,13 @@ impl BlockSelector {
/// Returns `[n_blocks]`.
pub fn scores(&self, q: &Tensor, pooled: &Tensor) -> candle_core::Result<Tensor> {
debug_assert_eq!(q.dims2()?, (self.n_heads, self.head_dim));
let per_head = q.matmul(&pooled.t()?)?.relu()?;
// f32 for the same reason `pool` is: this is a *selection*
// function. See the note there and on the batched path in
// `Indexer::positions_from`.
let per_head = q
.to_dtype(DType::F32)?
.matmul(&pooled.to_dtype(DType::F32)?.t()?)?
.relu()?;
per_head.sum(0)? / (self.head_dim as f64).sqrt()
}
@@ -505,15 +511,30 @@ impl Indexer {
// in one batched matmul: every head and query shares the same
// block keys, so fold them into the row axis rather than
// broadcasting the keys across heads.
let q_rows = q.reshape((batch, self.n_heads * seq_len, self.head_dim))?;
let keys = blocks.squeeze(1)?.transpose(1, 2)?.contiguous()?;
//
// In f32, matching upstream, which casts both operands before
// the matmul (`torch.matmul(q.float(), block_key_states.float()
// .transpose(-1, -2))`). It is not a rounding nicety: these
// scores feed a top-k over as many as 65,536 blocks, and the
// output is the *set of positions a query may attend*. Two
// blocks whose f32 scores differ by less than a bf16 tick —
// 0.03125 near 4.0 — compare equal in bf16 and the tie-break
// decides instead, so bf16 does not perturb the answer, it
// changes which past the model sees. Same argument as `pool`,
// which we already got right.
let q_rows = q
.reshape((batch, self.n_heads * seq_len, self.head_dim))?
.to_dtype(DType::F32)?;
let keys = blocks
.squeeze(1)?
.transpose(1, 2)?
.contiguous()?
.to_dtype(DType::F32)?;
let per_head = q_rows
.matmul(&keys)?
.reshape((batch, self.n_heads, seq_len, n_blocks))?
.relu()?;
let scores = (per_head.sum(1)? / (self.head_dim as f64).sqrt())?
.to_dtype(DType::F32)?
.to_vec3::<f32>()?;
let scores = (per_head.sum(1)? / (self.head_dim as f64).sqrt())?.to_vec3::<f32>()?;
let mut out = Vec::with_capacity(batch);
for row in scores.iter().take(batch) {
@@ -649,6 +670,63 @@ mod tests {
/// mean is exactly 0.25 — and so is an f32 mean cast back to bf16,
/// because the true 0.2507 sits below the first bf16 tick above
/// 0.25. Both mistakes land on the same wrong number, which is why
/// Scoring is f32 even when the cache is bf16, because it decides
/// *which past a query sees*, not merely by how much.
///
/// Upstream casts both operands before the matmul
/// (`torch.matmul(q.float(), block_key_states.float().T)`); we did
/// not, and ran the matmul, the relu and the sum over heads in the
/// model dtype.
///
/// The construction is the whole argument. Two blocks, four heads,
/// per-head contributions of 1.0 and — for one head of one block —
/// 1.0078125, which is 1 + 2^-7 and therefore exact in bf16. Their
/// sums are 4.0 and 4.0078125, and bf16's tick near 4.0 is 2^-5 =
/// 0.03125, so the second rounds back onto the first. The inputs
/// survive the dtype; the *comparison* does not. In bf16 the two
/// blocks tie and `select`'s index tie-break picks the wrong one,
/// silently, for as long as the model is served.
#[test]
fn scoring_is_f32_even_when_the_cache_is_bf16() {
let dev = Device::Cpu;
// Budget 4 over 4-wide blocks keeps exactly one block, so the
// comparison decides the answer rather than being absorbed by a
// budget large enough to take both.
let qsa = BlockSelector::new(4, 2, 4, 4).unwrap();
// Block 0 reads dimension 0 of each head, block 1 reads
// dimension 1, so the two scores are independent sums.
let pooled = Tensor::from_vec(vec![1.0f32, 0.0, 0.0, 1.0], (2, 2), &dev)
.unwrap()
.to_dtype(DType::BF16)
.unwrap();
let q = Tensor::from_vec(
vec![
1.0f32, 1.0, // head 0: block 0 <- 1.0, block 1 <- 1.0
1.0, 1.0, // head 1
1.0, 1.0, // head 2
1.0, 1.0078125, // head 3: block 1 gets the extra tick
],
(4, 2),
&dev,
)
.unwrap()
.to_dtype(DType::BF16)
.unwrap();
let scores: Vec<f32> = qsa.scores(&q, &pooled).unwrap().to_vec1().unwrap();
assert!(
scores[1] > scores[0],
"bf16 scoring lost the distinction between the blocks: {scores:?}"
);
// And the selection that rests on it picks the right block.
assert_eq!(
qsa.select(&scores, 8).unwrap(),
vec![4, 5, 6, 7],
"the higher-scoring block should be the one kept"
);
}
/// this asserts the value rather than the dtype.
#[test]
fn pooling_accumulates_in_f32() {
+165 -22
View File
@@ -1349,7 +1349,7 @@ impl LlamaDense {
/// value. New entries land alongside a new `ModelArch` variant + a
/// dispatch branch in `load_arch_dense` (plus, for TP, a parallel
/// pattern in `tp_qwen3.rs`).
const DENSE_SUPPORTED_MODEL_TYPES: &[&str] = &[
pub(crate) const DENSE_SUPPORTED_MODEL_TYPES: &[&str] = &[
"llama",
"qwen3",
"qwen3_5",
@@ -2735,6 +2735,15 @@ impl CandleHarness {
self.resolve_dense_files(spec, source_id).await?;
let device_for_load = device.clone();
let model_id_for_log = spec.model_id.clone();
// Resolved out here: `spec` is borrowed and does not outlive the
// blocking task.
let isq = super::quant::parse_quant_string(spec.quant.as_deref())?;
let isq_cache = self.isq_cache_for(
&spec.model_id,
&source_id.scheme,
spec.quant.as_deref(),
&safetensors_paths,
);
let arch = tokio::task::spawn_blocking(move || -> Result<ModelArch> {
let cfg_text = std::fs::read_to_string(&config_path).context("read config.json")?;
@@ -2845,11 +2854,23 @@ impl CandleHarness {
)
.context("build ShardedVarBuilder for qwen4_exp")?
};
// ISQ is not optional for this architecture in
// practice: without it the routed experts are
// 241.6 GB. `quant` comes from the catalogue, and a
// load without one will fail on memory rather than
// on anything this code can say (#315).
// ISQ is not really optional here: without it the
// routed experts are 241.6 GB and the load fails on
// memory rather than on anything this code can say.
// The catalogue's `quant` is the knob (#315, #252).
let model = super::arch::qwen4_exp::model::Qwen4ExpForCausalLM::load(
&cfg,
dtype,
isq,
&device_for_load,
&sharded_vb,
&safetensors_paths,
isq_cache.as_ref(),
)
.context("build qwen4_exp model")?;
Ok(ModelArch::Qwen4Exp(Box::new(model)))
@@ -2872,6 +2893,99 @@ impl CandleHarness {
Ok((tokenizer_path, arch))
}
/// Whether this spec's `quant` names a GGUF the repo actually has.
///
/// `quant` is overloaded (#320): the TP path reads it as in-situ
/// quantisation of safetensors, while the single-GPU paths read it
/// as "select the GGUF whose filename contains this string". A repo
/// with no GGUF at all cannot mean the second, so rather than fail
/// with an error about a file that was never going to exist, the
/// same string is read the way the TP path reads it.
///
/// This is deliberately narrow. A repo that *does* carry GGUFs keeps
/// the old behaviour exactly, including the old error when none of
/// them match the requested quant — that case is a typo, not an ISQ
/// request, and turning it into a silent 360 GB dense load would be
/// a worse answer than the error.
/// The quantisation cache for this load, or `None` when the quant
/// is unset or the checkpoint's revision cannot be identified
/// (#322).
///
/// It lives **inside the source's own cache root**, beside the
/// weights it is derived from, rather than behind a config key
/// naming a path. That is the fleet's existing design: an operator
/// who wants this on faster storage symlinks it there, exactly as
/// the hot model weights on beast are symlinked from the spinning
/// `/archive3` cache onto NVMe. Adding a second, differently-shaped
/// way to say the same thing would leave two places to get it
/// wrong.
///
/// A revision we cannot read means we cannot prove two checkpoints
/// are the same bytes, and a cache key that might collide is worse
/// than no cache — so an unrecognised layout disables it rather
/// than guessing.
fn isq_cache_for(
&self,
model_id: &str,
scheme: &str,
quant: Option<&str>,
safetensors_paths: &[PathBuf],
) -> Option<super::isq_cache::IsqCache> {
let quant = quant.filter(|q| !q.is_empty())?;
let root = self.hf_cache_for(scheme).ok()?.path().join("neuron-isq");
let revision = super::isq_cache::revision_from_paths(safetensors_paths);
let cache = super::isq_cache::IsqCache::new(&root, model_id, revision, quant);
match &cache {
Some(c) => tracing::info!(model = model_id, quant, dir = %c.dir().display(),
"isq cache: enabled for this load (#322)"),
None => tracing::info!(
model = model_id,
quant,
"isq cache: disabled — the checkpoint revision could not be identified"
),
}
cache
}
async fn quant_selects_a_gguf(
&self,
spec: &ModelSpec,
source_id: &cortex_core::source::ModelSourceId,
) -> Result<bool> {
let Some(quant) = spec.quant.as_deref().filter(|q| !q.is_empty()) else {
return Ok(false);
};
let api = self.hf_api_for(&source_id.scheme)?;
let info = api
.model(source_id.repo_path())
.info()
.await
.with_context(|| format!("fetch HF repo info for {source_id}"))?;
let names: Vec<String> = info
.siblings
.iter()
.map(|s| s.rfilename.to_lowercase())
.collect();
let any_gguf = names.iter().any(|n| n.ends_with(".gguf"));
if any_gguf {
return Ok(true);
}
let any_safetensors = names.iter().any(|n| n.ends_with(".safetensors"));
if any_safetensors {
tracing::info!(
model = %source_id,
quant = %quant,
"no GGUF in this repo; reading `quant` as in-situ quantization of \
the safetensors (#320)"
);
return Ok(false);
}
anyhow::bail!(
"repo {source_id} has neither a GGUF nor safetensors; `quant = {quant:?}` \
cannot be satisfied"
)
}
/// Resolve a model spec to local GGUF and tokenizer file paths via
/// hf-hub. Downloads on first use; subsequent calls are cached.
async fn resolve_files(
@@ -3856,7 +3970,7 @@ impl Harness for CandleHarness {
context_profile,
) = if let Some(w) = &worker {
// CUDA path: resolve, then load in the worker.
if spec.quant.is_some() {
if self.quant_selects_a_gguf(spec, &source_id).await? {
let (gguf_path, tokenizer_path) = self.resolve_files(spec, &source_id).await?;
let handle = w
.load_gguf(gguf_path, spec.model_id.clone())
@@ -3880,36 +3994,53 @@ impl Harness for CandleHarness {
let (config_path, tokenizer_path, safetensors_paths) =
self.resolve_dense_files(spec, &source_id).await?;
let meta = VisionMeta::from_config_path(&config_path);
// Prefix snapshots (#11) exist only for the in-tree
// qwen3_5 arch — which serves BOTH its model_types
// (qwen3_5 and the flat-config qwen3_next MoE family);
// the worker holds the ModelArch so the async side
// decides from config.json instead.
let snapshot_capable = matches!(
config_model_type(&config_path).as_deref(),
Some(super::arch::qwen3_5::MODEL_TYPE)
| Some(super::arch::qwen3_5::MODEL_TYPE_NEXT)
);
// Context-limit physics (#67): single-GPU → world_size 1.
// `None` for non-qwen3_5 dense archs.
// `None` for dense archs with no profile builder — they
// fall back to the static prompt cap.
let context_profile =
super::context_limit::profile_from_qwen3_5_config(&config_path, 1);
let handle = w
.load_dense(config_path, safetensors_paths, spec.model_id.clone())
super::context_limit::profile_from_qwen3_5_config(&config_path, 1).or_else(
|| super::context_limit::profile_from_qwen4_exp_config(&config_path, 1),
);
// In-situ quantisation (#320) reaches the worker load
// too, not just the CPU one — parsed here because
// `spec` does not cross the channel.
let isq = super::quant::parse_quant_string(spec.quant.as_deref())?;
// Built here rather than in the worker: the worker has
// no view of the config, and an `IsqCache` is a path,
// so it crosses the channel freely.
let isq_cache = self.isq_cache_for(
&spec.model_id,
&source_id.scheme,
spec.quant.as_deref(),
&safetensors_paths,
);
let load = w
.load_dense(
config_path,
safetensors_paths,
spec.model_id.clone(),
isq,
isq_cache,
)
.await
.map_err(|e| anyhow::anyhow!("worker load_dense: {e:#}"))?;
// Prefix snapshots (#11): the built model answers for
// itself on the way back, rather than the async side
// re-deriving the answer from `config.json`. See
// `DenseLoad` — the re-derivation is what silently
// dropped batched decode for qwen4_exp.
(
tokenizer_path,
None,
Some(handle),
Some(load.handle),
meta,
snapshot_capable,
load.supports_kv_snapshot,
context_profile,
)
}
} else {
// CPU path: legacy spawn_blocking + Arc<Mutex<ModelArch>>.
let (tokenizer_path, arch) = if spec.quant.is_some() {
let (tokenizer_path, arch) = if self.quant_selects_a_gguf(spec, &source_id).await? {
self.load_arch_gguf(spec, &source_id, &device).await?
} else {
self.load_arch_dense(spec, &source_id, &device).await?
@@ -4727,6 +4858,13 @@ impl CandleHarness {
image_token_id: vision_meta.image_token_id,
image_grid_factor: vision_meta.image_grid_factor,
spec: spec.clone(),
// Snapshot capability re-derived from `config.json`
// rather than asked of the loaded shard. Correct today —
// `TP_SUPPORTED_MODEL_TYPES` is exactly these two — but it
// is the same shape as the bug `DenseLoad` fixed on the
// single-GPU path, so whoever adds an arch here (#319)
// must add it in both places or lose batched decode
// silently.
prefix_cache: self.new_prefix_cache(matches!(
config_model_type(&config_path).as_deref(),
Some(super::arch::qwen3_5::MODEL_TYPE)
@@ -6278,9 +6416,14 @@ struct VisionMeta {
}
/// Peek at `config.json` for its `model_type`. Best-effort — `None`
/// on any read/parse error. The load path uses this to decide prefix-
/// snapshot capability for worker-held models the async side can't
/// inspect directly.
/// on any read/parse error.
///
/// Only the TP load path still asks: the single-GPU path takes the
/// arch's own answer off the load reply (`DenseLoad`) rather than
/// re-deriving it here, which is what stopped the two from drifting.
/// The TP path has the same hazard and not yet the same cure — see
/// the note at its call site.
#[cfg(feature = "cuda")]
fn config_model_type(config_path: &std::path::Path) -> Option<String> {
let text = std::fs::read_to_string(config_path).ok()?;
let v: serde_json::Value = serde_json::from_str(&text).ok()?;
+139 -3
View File
@@ -252,6 +252,85 @@ pub fn profile_from_qwen3_5_config(config_path: &Path, world_size: u32) -> Optio
})
}
/// Bytes of **QSA indexer** cache one token adds, per card (#307).
///
/// `qwen4_exp` keeps a second cache beside the main one: on every
/// full-attention layer the indexer stores the raw index key,
/// `indexer_kv_heads * indexer_head_dim` wide, in the compute dtype
/// (`arch::qwen4_exp::qsa::Indexer::key_cache`). Keys only — the
/// indexer scores blocks, it does not attend them — so there is no
/// `2 *` here.
///
/// It does not shard: `indexer_kv_heads` is 1, so a TP rank holds the
/// whole thing however wide the world is. At the checkpoint's 12
/// full-attention layers x 1 head x 128 dims that is 3 KiB/token on
/// top of the main cache's 24 — the surcharge the epic's original
/// arithmetic omitted, and it grows with context exactly as the main
/// cache does.
pub fn indexer_bytes_per_token(
n_full_attn_layers: usize,
indexer_kv_heads: usize,
indexer_head_dim: usize,
dtype_bytes: usize,
) -> u64 {
(n_full_attn_layers * indexer_kv_heads * indexer_head_dim * dtype_bytes) as u64
}
/// Build a [`ContextProfile`] from a `qwen4_exp` `config.json` — the
/// sibling of [`profile_from_qwen3_5_config`] for the other hybrid
/// arch. Returns `None` for any other `model_type` or an unparseable
/// config.
///
/// Same shape of physics, one term more: the per-token cost is the
/// main KV cache **plus** the QSA indexer's, because both grow per
/// token and both come out of the same card. Advertising against the
/// main cache alone would overstate what is servable by about an
/// eighth, which is the class of error #281 is about.
///
/// `layer_types` is always populated by the time `from_config_json`
/// returns (it fills the array from `full_attention_interval` when the
/// checkpoint omits it), so counting it is authoritative here.
pub fn profile_from_qwen4_exp_config(
config_path: &Path,
world_size: u32,
) -> Option<ContextProfile> {
let text = std::fs::read_to_string(config_path).ok()?;
if !super::arch::qwen4_exp::config::is_qwen4_exp(&text) {
return None;
}
let cfg = super::arch::qwen4_exp::config::Config::from_config_json(&text).ok()?;
let tc = &cfg.text_config;
let n_full_attn_layers = tc
.layer_types
.iter()
.filter(|t| t.as_str() == "full_attention")
.count();
// A model with no full-attention layer has no growing cache, and
// the derivation divides by this. Nothing in the family looks like
// that, so treat it as a config we do not understand rather than
// inventing an unbounded context.
if n_full_attn_layers == 0 {
return None;
}
let kv_bytes_per_token_per_card = kv_bytes_per_token(
n_full_attn_layers,
tc.num_key_value_heads,
tc.head_dim,
KV_CACHE_DTYPE_BYTES,
world_size,
) + indexer_bytes_per_token(
n_full_attn_layers,
tc.indexer_kv_heads,
tc.indexer_head_dim,
KV_CACHE_DTYPE_BYTES,
);
Some(ContextProfile {
max_position_embeddings: tc.max_position_embeddings,
kv_bytes_per_token_per_card,
world_size,
})
}
/// Round a token count down to a clean boundary so the advertised limit
/// doesn't jitter by a handful of tokens as live VRAM / the throughput
/// EMA wobble between polls.
@@ -375,9 +454,15 @@ pub fn derive_limit(
// Observability (#126): every input and intermediate term, so a
// surprising advertised limit is diagnosable from the journal
// instead of re-deriving by hand. DEBUG — this runs on every
// `GET /models` poll (a few lines per poll cycle).
tracing::debug!(
// instead of re-deriving by hand.
//
// TRACE, not debug: this runs on every `GET /models` poll and every
// input is fixed at load, so the line is byte-identical each time —
// about 8,600 a day per model at a 10s poll. It is worth having when
// reasoning about a context ceiling and worth nothing the rest of
// the time, and at debug it buries the lines that are not routine
// (#320).
tracing::trace!(
max_pos = profile.max_position_embeddings,
kv_bytes_per_token_per_card = profile.kv_bytes_per_token_per_card,
free_tightest_mb,
@@ -418,6 +503,57 @@ pub fn derive_limit(
#[cfg(test)]
mod tests {
/// The advertised limit for `qwen4_exp` must price both caches.
///
/// The main KV cache is 12 full-attention layers x 2 kv heads x 256
/// dims x 2 (K and V) x 2 bytes = 24 KiB/token, and the QSA indexer
/// adds 12 x 1 x 128 x 2 bytes = 3 KiB on top. Counting only the
/// first would advertise ~12.5% more context than the card can
/// hold — the failure mode #281 exists to stop, and the one the
/// epic's own arithmetic made before the indexer cache was found.
///
/// Driven from the checkpoint's verbatim `config.json`, so a field
/// that changes shape upstream fails here rather than in
/// production.
#[test]
fn a_qwen4_exp_profile_counts_the_indexer_cache_too() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.json");
std::fs::write(&path, super::super::arch::qwen4_exp::config::SHIPPED).unwrap();
let profile = profile_from_qwen4_exp_config(&path, 1)
.expect("the shipped config.json must yield a profile");
assert_eq!(profile.max_position_embeddings, 262_144);
assert_eq!(
profile.kv_bytes_per_token_per_card,
24 * 1024 + 3 * 1024,
"main KV plus the indexer's second cache"
);
}
/// The two hybrid archs must not answer for each other: whichever
/// builder runs first has to decline a config it does not own, or
/// the load path's `or_else` chain silently prices one model with
/// the other's physics.
#[test]
fn the_profile_builders_decline_each_others_configs() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("config.json");
std::fs::write(&path, super::super::arch::qwen4_exp::config::SHIPPED).unwrap();
assert!(
profile_from_qwen3_5_config(&path, 1).is_none(),
"the qwen3_5 builder must decline a qwen4_exp config"
);
let path = dir.path().join("other.json");
std::fs::write(&path, r#"{"model_type": "qwen3_next"}"#).unwrap();
assert!(
profile_from_qwen4_exp_config(&path, 1).is_none(),
"the qwen4_exp builder must decline everything else"
);
}
/// The deploy's smoke probe must not set the advertised context.
///
/// 19 tokens in 130 ms reads as ~145 tok/s on a host that measures
@@ -17,7 +17,9 @@ use crate::harness::arch::snapshot::KvCacheSnapshot;
use crate::harness::candle::ModelArch;
#[cfg(feature = "cuda")]
use crate::harness::device_worker::jobs::TpHandle;
use crate::harness::device_worker::jobs::{ArchHandle, ImageHandle, ImageInput, Job, KvSnapshotId};
use crate::harness::device_worker::jobs::{
ArchHandle, DenseLoad, ImageHandle, ImageInput, Job, KvSnapshotId,
};
use crate::harness::image::ZImagePipeline;
#[cfg(feature = "cuda")]
use crate::harness::tp::TpLeaderModel;
@@ -131,17 +133,46 @@ pub(crate) fn run(device_index: u32, rx: Receiver<Job>, poisoned: Arc<AtomicBool
} => {
let result = load_gguf_inner(&state.device, &gguf_path, &model_id)
.map(|arch| insert_arch(&mut state, Box::new(arch)));
// Same reasoning as `LoadImage`: release load-time
// transients before serving, and — on the failure
// path — give back whatever the partial load had
// allocated before it gave up. Without this a load
// that OOMs leaves its bytes in the pool, where the
// driver still counts them against the next load.
trim_device_pool(&state);
let _ = reply.send(result);
}
Job::LoadDense {
config_path,
safetensors_paths,
model_id,
quant,
isq_cache,
reply,
} => {
let result =
load_dense_inner(&state.device, &config_path, &safetensors_paths, &model_id)
.map(|arch| insert_arch(&mut state, Box::new(arch)));
let result = load_dense_inner(
&state.device,
&config_path,
&safetensors_paths,
&model_id,
quant,
isq_cache.as_ref(),
)
.map(|arch| {
// Ask the model, once, while it is still in
// hand. See `DenseLoad`.
let supports_kv_snapshot = arch.supports_kv_snapshot();
let handle = insert_arch(&mut state, Box::new(arch));
DenseLoad {
handle,
supports_kv_snapshot,
}
});
// As for `LoadGguf`. It matters more here: a dense
// load that runs out of VRAM part-way has typically
// allocated tens of gigabytes first, and the evicted
// model waiting to be restored needs them back.
trim_device_pool(&state);
let _ = reply.send(result);
}
Job::DropArch { handle, reply } => {
@@ -158,6 +189,14 @@ pub(crate) fn run(device_index: u32, rx: Receiver<Job>, poisoned: Arc<AtomicBool
// the end of the arm; calling drop() explicitly just
// makes the intent visible.
drop(removed);
// Return the freed blocks to the system rather than
// the stream-ordered pool, so the next load actually
// sees the VRAM — `DropImage` has done this since the
// benjy cold-swap, and an unload is an unload whatever
// modality asked for it.
if was_present {
trim_device_pool(&state);
}
tracing::debug!(
device_index,
handle = handle.0,
@@ -374,6 +413,13 @@ pub(crate) fn run(device_index: u32, rx: Receiver<Job>, poisoned: Arc<AtomicBool
let was_present = removed.is_some();
drop(removed);
state.tp_kv_snapshots.retain(|(h, _), _| *h != handle);
// As `DropArch`: this is only the leader's shard —
// the other ranks free theirs in their own processes
// — but the leader's is the one sharing a card with
// whatever loads next.
if was_present {
trim_device_pool(&state);
}
tracing::debug!(
device_index,
tp_handle = handle.0,
@@ -890,11 +936,31 @@ fn load_gguf_inner(
}
/// Load a dense safetensors model on the worker thread.
/// Log the cache summary when the load leaves scope, successful or
/// not. A load that OOMs half way is precisely when the question "how
/// much of that was re-quantised?" is worth answering.
fn scopeguard_summary<'a>(
cache: Option<&'a crate::harness::isq_cache::IsqCache>,
model_id: &'a str,
) -> impl Drop + 'a {
struct S<'a>(Option<&'a crate::harness::isq_cache::IsqCache>, &'a str);
impl Drop for S<'_> {
fn drop(&mut self) {
if let Some(c) = self.0 {
c.log_summary(self.1);
}
}
}
S(cache, model_id)
}
fn load_dense_inner(
device: &candle_core::Device,
config_path: &std::path::Path,
safetensors_paths: &[std::path::PathBuf],
model_id: &str,
quant: Option<candle_core::quantized::GgmlDType>,
isq_cache: Option<&crate::harness::isq_cache::IsqCache>,
) -> anyhow::Result<ModelArch> {
use anyhow::Context;
use candle_core::DType;
@@ -903,6 +969,10 @@ fn load_dense_inner(
use candle_transformers::models::qwen3 as qwen3_dense;
use candle_transformers::models::qwen3_moe as qwen3_moe_dense;
// Report what the cache did, however the load ends: a load that
// fails part-way is exactly when someone wants to know how much of
// it was cheap.
let _summary = scopeguard_summary(isq_cache, model_id);
let cfg_text = std::fs::read_to_string(config_path).context("read config.json")?;
crate::harness::candle::check_dense_config_supported(&cfg_text, model_id)?;
// Peek at model_type to choose the family before the typed
@@ -988,6 +1058,38 @@ fn load_dense_inner(
.context("build Qwen3-Next dense model")?;
Ok(ModelArch::Qwen3_5Dense(model))
}
"qwen4_exp" => {
// Qwen3.8-Flash-Next. Mirrors the CPU path in
// `candle::load_arch_dense` — same config type, same
// sharded backend, same in-situ quantisation.
//
// ISQ is not optional for this architecture: the routed
// experts are 241.6 GB at BF16 (measured from the
// checkpoint's own headers), against 32.6 GB of card. A
// load without a `quant` will fail on memory rather than
// on anything this code can say (#315, #252).
let cfg = crate::harness::arch::qwen4_exp::config::Config::from_config_json(&cfg_text)
.context("parse qwen4_exp config.json")?;
let sharded_vb = unsafe {
candle_nn::var_builder::ShardedSafeTensors::var_builder(
safetensors_paths,
dtype,
device,
)
.context("build ShardedVarBuilder for qwen4_exp")?
};
let model = crate::harness::arch::qwen4_exp::model::Qwen4ExpForCausalLM::load(
&cfg,
dtype,
quant,
device,
&sharded_vb,
safetensors_paths,
isq_cache,
)
.context("build qwen4_exp model")?;
Ok(ModelArch::Qwen4Exp(Box::new(model)))
}
other => anyhow::bail!(
"unrouted supported model_type '{other}' — \
DENSE_SUPPORTED_MODEL_TYPES and load_dense_inner \
@@ -1045,7 +1147,7 @@ fn tp_load_shard_inner(
"qwen3_5" | "qwen3_next" => {
let cfg = crate::harness::tp::tp_qwen3_5::Config::from_config_json(config_json)
.context("parse Qwen3-Next Config JSON for leader load")?;
let quant_dtype = crate::harness::tp::worker::parse_quant_string(quant)?;
let quant_dtype = crate::harness::quant::parse_quant_string(quant)?;
TpLeaderModel::Qwen3_5(crate::harness::tp::tp_qwen3_5::TpQwen3_5ForCausalLM::load(
cfg,
&vb,
@@ -1669,3 +1771,48 @@ fn drain_poisoned(job: Job, device_index: u32) {
}
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Every architecture the preflight admits must have a loader arm
/// on **this** path, not only on the CPU one in
/// `candle::load_arch_dense`.
///
/// There are two dense loaders — the worker's (every CUDA load)
/// and candle's (the CPU fallback) — and one list gating both.
/// `qwen4_exp` was added to the list and to the CPU loader, so it
/// passed preflight and then hit the "unrouted supported
/// model_type" bail here: on a GPU host the model was
/// unloadable, in about 200 ms, having read nothing. The bail's
/// own comment said the two must stay in sync; nothing checked.
///
/// A bare `{"model_type": ...}` reaches the arm and then fails on
/// the typed parse, which is fine — this asserts only that an arm
/// exists to fail in.
#[test]
fn every_supported_dense_model_type_has_a_loader_arm() {
let dir = tempfile::tempdir().unwrap();
for model_type in crate::harness::candle::DENSE_SUPPORTED_MODEL_TYPES {
let path = dir.path().join(format!("{model_type}.json"));
std::fs::write(&path, format!(r#"{{"model_type": "{model_type}"}}"#)).unwrap();
let msg = match load_dense_inner(
&candle_core::Device::Cpu,
&path,
&[],
model_type,
None,
None,
) {
Ok(_) => panic!("a bare config must not build a model"),
Err(e) => format!("{e:#}"),
};
assert!(
!msg.contains("unrouted"),
"`{model_type}` is in DENSE_SUPPORTED_MODEL_TYPES but load_dense_inner \
has no arm for it: {msg}"
);
}
}
}
@@ -19,6 +19,27 @@ use tokio::sync::oneshot;
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct ArchHandle(pub u64);
/// What a dense load hands back: the handle, plus what the arch it
/// built says about itself.
///
/// The async side cannot ask the model anything — it lives in the
/// worker's slab and never crosses the channel — so anything the
/// serving path needs to know has to ride back on the reply. Prefix
/// snapshot support (#11) is the first such fact, and it is here
/// rather than re-derived from `config.json` because that is exactly
/// how it went wrong: the async side matched a list of `model_type`
/// strings that `ModelArch::supports_kv_snapshot` had since grown
/// past, so `qwen4_exp` loaded with batched decode silently off.
/// A capability the arch answers for itself cannot drift from the
/// arch.
#[derive(Debug, Clone, Copy)]
pub struct DenseLoad {
pub handle: ArchHandle,
/// [`crate::harness::candle::ModelArch::supports_kv_snapshot`], as
/// answered by the model that was actually built.
pub supports_kv_snapshot: bool,
}
/// Opaque handle to a `TpLeaderModel` stored in the worker thread's
/// state slab. Same shape as [`ArchHandle`] but in a separate
/// namespace so the two slabs can coexist without ambiguity. Phase 3
@@ -107,7 +128,14 @@ pub enum Job {
config_path: PathBuf,
safetensors_paths: Vec<PathBuf>,
model_id: String,
reply: oneshot::Sender<Result<ArchHandle>>,
/// In-situ quantisation (#320): the catalogue's `quant`,
/// already parsed. `None` loads at the checkpoint dtype.
/// Archs that cannot quantise in situ ignore it.
quant: Option<candle_core::quantized::GgmlDType>,
/// Where to cache the quantised weights (#322). Built on the
/// async side, which is the only side that can see the config.
isq_cache: Option<crate::harness::isq_cache::IsqCache>,
reply: oneshot::Sender<Result<DenseLoad>>,
},
/// Remove the model from the slab and drop it. The `Drop` runs on
/// the worker thread so CUDA tensors release their memory on the
+10 -3
View File
@@ -51,7 +51,7 @@ use tokio::sync::oneshot;
#[cfg(feature = "cuda")]
pub use jobs::TpHandle;
pub use jobs::{ArchHandle, ImageHandle, Job, KvSnapshotId};
pub use jobs::{ArchHandle, DenseLoad, ImageHandle, Job, KvSnapshotId};
/// Errors returned by `DeviceWorkerHandle` submit methods.
#[derive(Debug, thiserror::Error)]
@@ -221,7 +221,9 @@ impl DeviceWorkerHandle {
config_path: std::path::PathBuf,
safetensors_paths: Vec<std::path::PathBuf>,
model_id: String,
) -> Result<ArchHandle, WorkerError> {
quant: Option<candle_core::quantized::GgmlDType>,
isq_cache: Option<crate::harness::isq_cache::IsqCache>,
) -> Result<DenseLoad, WorkerError> {
if self.poisoned.load(Ordering::Acquire) {
return Err(WorkerError::Poisoned {
device_index: self.device_index,
@@ -233,6 +235,8 @@ impl DeviceWorkerHandle {
config_path,
safetensors_paths,
model_id,
quant,
isq_cache,
reply: reply_tx,
})
.map_err(|_| WorkerError::Gone {
@@ -1243,9 +1247,12 @@ mod tests {
fixture.join("config.json"),
vec![fixture.join("model.safetensors")],
"qwen3_next-tiny".into(),
None,
None,
)
.await
.expect("load tiny fixture");
.expect("load tiny fixture")
.handle;
let prompts: [&[u32]; 3] = [&[1, 2, 3], &[4, 5], &[7, 3, 2, 5, 6]];
let steps: [&[u32]; 3] = [&[11, 12, 13], &[9, 8, 7], &[21, 22, 23]];
+4 -1
View File
@@ -1261,9 +1261,12 @@ mod tests {
fixture.join("config.json"),
vec![fixture.join("model.safetensors")],
"qwen3_next-tiny".into(),
None,
None,
)
.await
.expect("load fixture");
.expect("load fixture")
.handle;
let admission_cfg = AdmissionConfig {
max_in_flight: 3,
+451
View File
@@ -0,0 +1,451 @@
//! On-disk cache for in-situ quantisation (#322).
//!
//! Quantising a model's weights at load is a pure function of the
//! checkpoint bytes, the requested type, and the code that does it.
//! None of those change between loads, and yet neuron recomputed the
//! answer every time and threw it away on unload. For `qwen4_exp` that
//! is **~32 s per decoder layer, about 26 minutes for a full load**,
//! measured on beast — paid again on every restart and every eviction
//! that is later reversed.
//!
//! So the quantised tensors are written out once and read back after.
//!
//! ## The key must include our own version
//!
//! `(checkpoint revision, quant name)` is not enough. `bf1e80ed`
//! changed what `quant = "q4k"` *means* for a 640-wide row — k-quants
//! cannot block it, so it now falls back to a 32-block type — and an
//! artifact written before that commit and read after it would be
//! silently wrong in a way no checksum of the inputs could catch.
//! [`QUANTISER_VERSION`] is part of the path, and bumping it orphans
//! every stale artifact rather than misreading one.
//!
//! ## What is cached, and what is not
//!
//! One GGUF per layer, holding that layer's quantised experts. GGUF
//! because it stores a **type per tensor**, which is exactly what the
//! per-row fallback produces — a layer whose `down_proj` is `Q5_0`
//! while its `gate`/`up` are `Q4K` is representable without inventing
//! a container for it — and because neuron already reads the format.
//!
//! Per layer rather than per model because it makes the cache
//! **incrementally useful**: a load that runs out of VRAM at layer 16
//! has still written sixteen layers, and the next attempt starts
//! there. On hardware where the model does not yet fit at all (#318),
//! that is the difference between the cache being useless and being
//! the thing that makes iteration bearable.
//!
//! ## Where it lives
//!
//! Inside the source's own cache root — `<hf cache>/neuron-isq/…` —
//! beside the weights it is derived from, and not behind a config key
//! naming a path.
//!
//! That is deliberate and it is the fleet's existing design. beast
//! already keeps its hot model weights on NVMe by symlinking them out
//! of the spinning `/archive3` cache into `/archive1`, with no config
//! mapping paths to mounts. An operator who wants these artifacts on
//! faster storage does the same thing to one directory. A second,
//! differently-shaped way to express the same intent would be a second
//! place to get it wrong, and the first one already works.
//!
//! Size is the thing to know before symlinking: an artifact is roughly
//! the served size of the model's quantised weights — ~73 GB for
//! `qwen4_exp` at q4k.
//!
//! ## It is an optimisation, and never load-bearing
//!
//! Every failure here is swallowed: a miss, a corrupt file, a full
//! disk, a directory nobody can write. The load proceeds by
//! quantising, exactly as it did before this module existed. A cache
//! that can fail a load is worse than no cache.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
use candle_core::Device;
use candle_core::quantized::{QTensor, gguf_file};
/// Bump when the *meaning* of a quant name changes: a different
/// fallback table, a different rounding, a different layout. Artifacts
/// written by an older quantiser are then ignored rather than
/// misread, and get overwritten in place as layers are re-quantised.
///
/// 1: the initial format. Per-layer GGUF, tensors named
/// `e{index}.{gate,up,down}`, llama.cpp's `tensor_type_fallback`
/// table for rows that do not divide the requested block size.
pub const QUANTISER_VERSION: u32 = 1;
/// Refuse a write that would take more than this share of the free
/// space.
///
/// The rule is about *the write*, not the filesystem. An earlier
/// version required 2% of the volume to remain free, which read
/// sensibly and was wrong in the case that matters least and happens
/// most: on a filesystem already below that line, it refused a
/// six-megabyte write that was in no way responsible. CI found it
/// immediately.
///
/// Bounding each artifact by half the remaining space says the thing
/// actually meant — no single artifact eats the disk — at every size,
/// and needs to know nothing about how big the volume is. An
/// artifact for one 180 B model is ~73 GB, so on a volume with 100 GB
/// left it is refused and on beast's 2 TB of free `/archive3` it is
/// not. A few megabytes is never refused, because a few megabytes is
/// never the problem.
const MAX_SHARE_OF_FREE: f64 = 0.5;
/// A cache scoped to one (model, revision, quant, quantiser version).
pub struct IsqCache {
dir: PathBuf,
hits: AtomicUsize,
misses: AtomicUsize,
read: AtomicU64,
written: AtomicU64,
}
impl IsqCache {
/// `revision` should identify the checkpoint bytes — the snapshot
/// directory name, for an hf-hub cache. `None` disables the cache
/// rather than guessing, because a wrong revision is the one
/// failure mode that produces a confidently wrong model.
pub fn new(base: &Path, model_id: &str, revision: Option<&str>, quant: &str) -> Option<Self> {
let revision = revision?;
let safe = |s: &str| {
s.chars()
.map(|c| {
if c.is_ascii_alphanumeric() || c == '-' || c == '.' {
c
} else {
'-'
}
})
.collect::<String>()
};
let dir = base
.join(safe(model_id))
.join(safe(revision))
.join(format!("{}-v{QUANTISER_VERSION}", safe(quant)));
Some(Self {
dir,
hits: AtomicUsize::new(0),
misses: AtomicUsize::new(0),
read: AtomicU64::new(0),
written: AtomicU64::new(0),
})
}
/// Read a layer's quantised tensors onto `device`.
///
/// `None` for a miss and for anything that goes wrong reading —
/// a truncated artifact from an interrupted write reads as absent,
/// and is rewritten by the load that follows.
pub fn load(&self, key: &str, device: &Device) -> Option<HashMap<String, QTensor>> {
let path = self.dir.join(format!("{key}.gguf"));
let Ok(mut file) = std::fs::File::open(&path) else {
self.misses.fetch_add(1, Ordering::Relaxed);
return None;
};
let content = match gguf_file::Content::read(&mut file) {
Ok(c) => c,
Err(e) => {
self.misses.fetch_add(1, Ordering::Relaxed);
tracing::warn!(path = %path.display(), error = %e,
"isq cache: unreadable artifact, re-quantising this layer");
return None;
}
};
let names: Vec<String> = content.tensor_infos.keys().cloned().collect();
let mut out = HashMap::with_capacity(names.len());
for name in names {
match content.tensor(&mut file, &name, device) {
Ok(t) => {
out.insert(name, t);
}
Err(e) => {
self.misses.fetch_add(1, Ordering::Relaxed);
tracing::warn!(path = %path.display(), tensor = %name, error = %e,
"isq cache: artifact tensor failed to read, re-quantising this layer");
return None;
}
}
}
self.hits.fetch_add(1, Ordering::Relaxed);
self.read.fetch_add(
out.values()
.map(|t| t.storage_size_in_bytes() as u64)
.sum::<u64>(),
Ordering::Relaxed,
);
Some(out)
}
/// Write a layer's quantised tensors.
///
/// Through a temporary file and a rename, so an interrupted write
/// cannot leave a half-artifact that a later load would read as
/// real. Errors are logged and swallowed.
/// Write a layer, refusing one that would take an unreasonable
/// share of the disk. The production entry point: every failure is
/// logged and swallowed.
pub fn store(&self, key: &str, tensors: &[(&str, &QTensor)]) {
let bytes: u64 = tensors
.iter()
.map(|(_, t)| t.storage_size_in_bytes() as u64)
.sum();
if let Some(free) = free_bytes(&self.dir)
&& too_large_for(bytes, free)
{
tracing::warn!(
dir = %self.dir.display(), key,
gib = bytes as f64 / (1024.0 * 1024.0 * 1024.0),
free_gib = free as f64 / (1024.0 * 1024.0 * 1024.0),
"isq cache: refusing a write that would take over {:.0}% of the free space",
MAX_SHARE_OF_FREE * 100.0,
);
return;
}
match self.try_store(key, tensors) {
Ok(()) => {
self.written.fetch_add(bytes, Ordering::Relaxed);
}
Err(e) => tracing::warn!(dir = %self.dir.display(), key, error = %e,
"isq cache: could not write this layer; the load continues uncached"),
}
}
/// Say, once per load, what the cache actually did.
///
/// Without this the journal shows only that the cache was enabled,
/// which is the shape of problem #321 is about: every light green
/// and no way to tell from the outside whether the thing worked.
/// A load that served nothing from cache and a load that served
/// everything looked identical, and the difference between them is
/// eight minutes.
pub fn log_summary(&self, model_id: &str) {
let gib = |b: u64| b as f64 / (1024.0 * 1024.0 * 1024.0);
tracing::info!(
model = model_id,
dir = %self.dir.display(),
layers_from_cache = self.hits.load(Ordering::Relaxed),
layers_quantised = self.misses.load(Ordering::Relaxed),
read_gib = gib(self.read.load(Ordering::Relaxed)),
written_gib = gib(self.written.load(Ordering::Relaxed)),
"isq cache: load complete (#322)"
);
}
/// Write a layer. Nothing else — no policy, and in particular no
/// question asked of the filesystem about its size.
///
/// That separation is the point. The disk check lives in
/// [`Self::store`], so the round-trip tests exercise the write and
/// the read without their result depending on how full the machine
/// running them happens to be. A unit test that consults the free
/// space of whatever host executes it is not testing this code.
pub(crate) fn try_store(&self, key: &str, tensors: &[(&str, &QTensor)]) -> anyhow::Result<()> {
std::fs::create_dir_all(&self.dir)?;
let final_path = self.dir.join(format!("{key}.gguf"));
let tmp = self.dir.join(format!(".{key}.{}.tmp", std::process::id()));
let write = (|| -> anyhow::Result<()> {
let mut f = std::fs::File::create(&tmp)?;
gguf_file::write(&mut f, &[], tensors)?;
Ok(())
})();
if let Err(e) = write {
// A partial artifact is dead weight that nothing will ever
// clean up — and on the failure this is most likely to see,
// a full disk, it is dead weight made of the last free
// bytes.
let _ = std::fs::remove_file(&tmp);
return Err(e);
}
std::fs::rename(&tmp, &final_path)?;
Ok(())
}
pub fn dir(&self) -> &Path {
&self.dir
}
}
/// Whether a write of `bytes` takes an unreasonable share of `free`.
///
/// Split out from the write so the rule can be tested without needing
/// a filesystem in a particular state — which is precisely what made
/// the previous rule's defect invisible until CI ran it.
fn too_large_for(bytes: u64, free: u64) -> bool {
bytes as f64 > free as f64 * MAX_SHARE_OF_FREE
}
/// Free bytes on the filesystem holding `path`, or `None` if it cannot
/// be determined — in which case the caller writes anyway rather than
/// refusing on a guess. `-P` is what guarantees one line per
/// filesystem, so a long device name cannot wrap the fields apart.
fn free_bytes(path: &Path) -> Option<u64> {
let out = std::process::Command::new("df")
.arg("-kP")
.arg(path)
.output()
.ok()?;
let text = String::from_utf8(out.stdout).ok()?;
let kb = text.lines().nth(1)?.split_whitespace().nth(3)?;
kb.parse::<u64>().ok().map(|kb| kb * 1024)
}
/// The revision an hf-hub-cached checkpoint was resolved at, taken
/// from the snapshot directory its files live in.
///
/// `.../models--org--name/snapshots/<revision>/model-00001-of-N.safetensors`
/// — the parent directory *is* the revision, which is what makes it a
/// safe cache key: two checkpoints with different bytes cannot share
/// one. Returns `None` for any layout that is not that, so an
/// unrecognised path disables the cache instead of colliding.
pub fn revision_from_paths(paths: &[PathBuf]) -> Option<&str> {
let first = paths.first()?;
let dir = first.parent()?;
let name = dir.file_name()?.to_str()?;
let looks_like_snapshot = dir.parent()?.file_name()?.to_str()? == "snapshots";
(looks_like_snapshot && !name.is_empty()).then_some(name)
}
#[cfg(test)]
mod tests {
use super::*;
use candle_core::Tensor;
use candle_core::quantized::GgmlDType;
/// A cached layer must dequantise to the same numbers the fresh
/// quantisation produced.
///
/// This is the whole contract. A cache that returns *something*
/// plausible is worse than no cache, because the model loads, it
/// serves, and its weights are quietly not the ones it was asked
/// for. Round-tripping the dequantised values rather than the
/// bytes is deliberate: it is what the model actually consumes.
#[test]
fn a_cached_tensor_dequantises_to_what_was_stored() {
let dev = Device::Cpu;
let dir = tempfile::tempdir().unwrap();
let cache = IsqCache::new(dir.path(), "org/model", Some("rev0"), "q4k").unwrap();
// 256-wide so the k-quant can block it at all.
let src = Tensor::rand(-1f32, 1f32, (8, 256), &dev).unwrap();
let q = QTensor::quantize(&src, GgmlDType::Q4K).unwrap();
let before: Vec<f32> = q
.dequantize(&dev)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
cache
.try_store("layer-0", &[("e0.gate", &q)])
.expect("the write must succeed, and say why if it does not");
let got = cache
.load("layer-0", &dev)
.expect("the artifact just written");
let after: Vec<f32> = got["e0.gate"]
.dequantize(&dev)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
assert_eq!(
before, after,
"a cached tensor changed value on the round trip"
);
assert_eq!(got["e0.gate"].dtype(), GgmlDType::Q4K, "and kept its type");
}
/// The key separates everything that changes the answer.
///
/// The quantiser version is the one that is easy to leave out and
/// impossible to detect: `bf1e80ed` changed what `q4k` means for a
/// 640-wide row, and an artifact written before it is wrong after
/// it in a way no checksum of the *inputs* would reveal.
#[test]
fn the_key_separates_model_revision_quant_and_quantiser() {
let root = std::path::Path::new("/cache");
let base = IsqCache::new(root, "org/model", Some("rev0"), "q4k").unwrap();
let others = [
IsqCache::new(root, "org/other", Some("rev0"), "q4k").unwrap(),
IsqCache::new(root, "org/model", Some("rev1"), "q4k").unwrap(),
IsqCache::new(root, "org/model", Some("rev0"), "q6k").unwrap(),
];
for other in &others {
assert_ne!(base.dir(), other.dir());
}
assert!(
base.dir()
.to_str()
.unwrap()
.contains(&format!("v{QUANTISER_VERSION}")),
"the quantiser version must be in the path: {}",
base.dir().display()
);
}
/// No revision, no cache. A key that might collide across
/// checkpoints is worse than recomputing.
#[test]
fn an_unidentifiable_revision_disables_the_cache() {
assert!(IsqCache::new(std::path::Path::new("/cache"), "org/model", None, "q4k").is_none());
// An hf-hub snapshot path yields its revision; anything else
// does not, rather than yielding a directory name that happens
// to be there.
let hf = PathBuf::from(
"/cache/models--org--model/snapshots/abc123/model-00001-of-00002.safetensors",
);
assert_eq!(revision_from_paths(&[hf]), Some("abc123"));
assert_eq!(
revision_from_paths(&[PathBuf::from("/tmp/model.safetensors")]),
None
);
assert_eq!(revision_from_paths(&[]), None);
}
/// The space guard bounds the *write*, not the filesystem.
///
/// The rule it replaced required a fixed share of the volume to
/// remain free, which refused a six-megabyte write on a disk that
/// was already full — punishing the write least responsible for
/// the state. These cases are the ones that matter: a 73 GB
/// artifact must not land on a volume with 100 GB left, and a few
/// megabytes must always be allowed, however full things are.
#[test]
fn the_space_guard_bounds_the_write_not_the_volume() {
let gib = 1024 * 1024 * 1024;
// A 73 GB artifact against beast's ~2 TB of free /archive3.
assert!(!too_large_for(73 * gib, 2000 * gib));
// The same artifact where it would eat most of what is left.
assert!(too_large_for(73 * gib, 100 * gib));
// Six megabytes on a nearly-full disk: never the problem.
assert!(!too_large_for(6 * 1024 * 1024, 200 * 1024 * 1024));
// Exactly half is allowed; past half is not.
assert!(!too_large_for(50, 100));
assert!(too_large_for(51, 100));
// And nothing is writable onto nothing.
assert!(too_large_for(1, 0));
}
/// A miss and a corrupt artifact are the same thing to the caller:
/// re-quantise. Neither may propagate as an error.
#[test]
fn a_missing_or_corrupt_artifact_reads_as_absent() {
let dev = Device::Cpu;
let dir = tempfile::tempdir().unwrap();
let cache = IsqCache::new(dir.path(), "org/model", Some("rev0"), "q4k").unwrap();
assert!(cache.load("layer-0", &dev).is_none(), "a miss");
std::fs::create_dir_all(cache.dir()).unwrap();
std::fs::write(cache.dir().join("layer-0.gguf"), b"not a gguf file at all").unwrap();
assert!(cache.load("layer-0", &dev).is_none(), "a corrupt artifact");
}
}
+2
View File
@@ -9,9 +9,11 @@ pub mod context_limit;
pub mod device_worker;
pub mod engine;
pub mod image;
pub mod isq_cache;
pub mod prefix_cache;
pub mod preflight;
pub mod preprocess;
pub(crate) mod quant;
pub mod reasoning_budget;
pub mod reasoning_effort;
pub mod sampling;
+49
View File
@@ -0,0 +1,49 @@
//! Catalogue quant names to ggml dtypes.
//!
//! Shared rather than owned by the TP path: `quant = "q4k"` in
//! `models.toml` means the same thing whichever loader reads it. It sat
//! behind `cfg(feature = "cuda")` while only the TP worker used it,
//! which put it out of reach of the single-GPU path — where, for
//! `qwen4_exp`, in-situ quantisation is the difference between a model
//! that loads and 241.6 GB that does not (#315).
/// Parse a `ModelSpec.quant` string into a `GgmlDType`. Accepts the
/// common ggml format names (case-insensitive). `None` and `Some("")`
/// both map to "no quantization".
///
/// Supported: `q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q8_1`,
/// `q2k`/`q2_k`, `q3k`/`q3_k`, `q4k`/`q4_k`, `q5k`/`q5_k`,
/// `q6k`/`q6_k`, `q8k`/`q8_k`, `f16`, `bf16`, `f32`. The underscore
/// is optional and the prefix is case-insensitive.
pub(crate) fn parse_quant_string(
s: Option<&str>,
) -> anyhow::Result<Option<candle_core::quantized::GgmlDType>> {
use candle_core::quantized::GgmlDType;
let s = match s {
Some(s) if !s.is_empty() => s,
_ => return Ok(None),
};
let normalised = s.to_ascii_lowercase().replace('_', "");
let dtype = match normalised.as_str() {
"q40" => GgmlDType::Q4_0,
"q41" => GgmlDType::Q4_1,
"q50" => GgmlDType::Q5_0,
"q51" => GgmlDType::Q5_1,
"q80" => GgmlDType::Q8_0,
"q81" => GgmlDType::Q8_1,
"q2k" => GgmlDType::Q2K,
"q3k" => GgmlDType::Q3K,
"q4k" | "q4km" => GgmlDType::Q4K,
"q5k" | "q5km" => GgmlDType::Q5K,
"q6k" => GgmlDType::Q6K,
"q8k" => GgmlDType::Q8K,
"f16" => GgmlDType::F16,
"bf16" => GgmlDType::BF16,
"f32" => GgmlDType::F32,
other => anyhow::bail!(
"unknown quant '{other}' (expected one of: q4_0, q4_1, q5_0, q5_1, q8_0, \
q8_1, q2k, q3k, q4k, q5k, q6k, q8k, f16, bf16, f32)"
),
};
Ok(Some(dtype))
}
+5 -42
View File
@@ -16,6 +16,11 @@ use anyhow::Result;
use std::collections::HashMap;
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
// Only the CUDA shard-load path calls this; the CPU build of this
// module has no caller.
#[cfg(feature = "cuda")]
use crate::harness::quant::parse_quant_string;
use super::nccl_state::NcclState;
use super::rpc::{WorkerRequest, WorkerResponse};
@@ -1035,45 +1040,3 @@ impl WorkerState {
}
}
}
/// Parse a `ModelSpec.quant` string into a `GgmlDType`. Accepts the
/// common ggml format names (case-insensitive). `None` and `Some("")`
/// both map to "no quantization".
///
/// Supported: `q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q8_1`,
/// `q2k`/`q2_k`, `q3k`/`q3_k`, `q4k`/`q4_k`, `q5k`/`q5_k`,
/// `q6k`/`q6_k`, `q8k`/`q8_k`, `f16`, `bf16`, `f32`. The underscore
/// is optional and the prefix is case-insensitive.
#[cfg(feature = "cuda")]
pub(crate) fn parse_quant_string(
s: Option<&str>,
) -> anyhow::Result<Option<candle_core::quantized::GgmlDType>> {
use candle_core::quantized::GgmlDType;
let s = match s {
Some(s) if !s.is_empty() => s,
_ => return Ok(None),
};
let normalised = s.to_ascii_lowercase().replace('_', "");
let dtype = match normalised.as_str() {
"q40" => GgmlDType::Q4_0,
"q41" => GgmlDType::Q4_1,
"q50" => GgmlDType::Q5_0,
"q51" => GgmlDType::Q5_1,
"q80" => GgmlDType::Q8_0,
"q81" => GgmlDType::Q8_1,
"q2k" => GgmlDType::Q2K,
"q3k" => GgmlDType::Q3K,
"q4k" | "q4km" => GgmlDType::Q4K,
"q5k" | "q5km" => GgmlDType::Q5K,
"q6k" => GgmlDType::Q6K,
"q8k" => GgmlDType::Q8K,
"f16" => GgmlDType::F16,
"bf16" => GgmlDType::BF16,
"f32" => GgmlDType::F32,
other => anyhow::bail!(
"unknown quant '{other}' (expected one of: q4_0, q4_1, q5_0, q5_1, q8_0, \
q8_1, q2k, q3k, q4k, q5k, q6k, q8k, f16, bf16, f32)"
),
};
Ok(Some(dtype))
}
+76
View File
@@ -12,6 +12,82 @@ HuggingFace reference" claim to checked-in numbers.
| `qwen3_5-0.8b-text` | Qwen/Qwen3.5-0.8B | text (>64-token prompt → chunked GDN prefill) | f32 | `text_logits_match_reference` |
| `qwen3_5-0.8b-vision` | Qwen/Qwen3.5-0.8B | 448×448 synthetic image + prompt | f32 | `vision_tower_and_logits_match_reference` |
| `qwen3_6-27b-text` | Qwen/Qwen3.6-27B | text | bf16 | manual (see below) |
| `qwen4_exp-tiny` | random weights, real architecture | text, 12 tokens, no eos | f32 | `qwen4_exp_logits_match_reference` |
| `qwen4_exp-tiny-long` | random weights, real architecture | text, 80 tokens, eos at 37 | f32 | `qwen4_exp_long_prompt_logits_match_reference` |
## `qwen4_exp-tiny` is different, and runs in CI
The others replay a real snapshot and self-skip without
`NEURON_REF_MODEL_PATH`. This one is 44 KB of random weights in the
real architecture — 4 layers (3 linear-attention, 1 QSA), PLE on layer
1, 2 experts — generated *by* `transformers` itself via
[`script/generate_qwen4_exp_fixture.py`](../../../../../script/generate_qwen4_exp_fixture.py).
So it is checked in, needs no weights, and runs on every push.
That matters because the whole `qwen4_exp` architecture was ported from
a written spec (#308) and tested against our own reading of it. This is
the only thing in the tree that compares it to the implementation it
was ported from (#323).
Two properties of the fixture are load-bearing, and both were learned
by getting them wrong:
- **Weights are `uniform(-0.8, 0.8)`, not `uniform(-0.1, 0.1)`.** Near
the origin `silu(x) ~ x/2`, so the SwiGLU is nearly symmetric and
reading the fused expert's gate and up halves the wrong way round
moved the logits by 2.4e-4 — inside any tolerance worth writing. At
the larger scale the same mutation moves them by 2.7.
- **The generator refuses to emit a fixture containing a QSA selection
tie.** A block scores exactly 0.0 whenever relu clamps every head, so
ties are common with random weights; when the k-th and (k+1)-th
scores are equal, which block is dropped is decided by `topk`'s
ordering of equal elements, which torch does not specify and whose
CPU and CUDA kernels need not agree. Seeds 0, 1 and 4 produce ties;
seed 2 (the default) does not. See "the tie-break" below.
Regenerate with the reference from `transformers` **main** — the 5.9.0
release does not carry `qwen4_exp`:
```sh
python -m venv --system-site-packages .venv
.venv/bin/pip install --no-deps "transformers @ git+https://github.com/huggingface/transformers@main"
.venv/bin/pip install "tokenizers>=0.23.1,<0.24" "safetensors>=0.8.0"
.venv/bin/python script/generate_qwen4_exp_fixture.py \
--config crates/neuron/tests/fixtures/numerical/qwen4_exp-tiny/config.json \
--out crates/neuron/tests/fixtures/numerical/qwen4_exp-tiny
```
### Why there are two cases
They are not the same test at two sizes. Each reaches a path the other
structurally cannot, and a mutation demonstrates it:
- **80 tokens** puts the GatedDeltaNet layers on the *chunked* prefill
algorithm (#23), which engages at 64. That is a different
factorisation of the same recurrence, not the same code with more
input — which is also why the long case's bound is 1e-4 against the
short case's 1e-5 (observed 1.3e-5 vs 2e-6). It also gives QSA up to
40 blocks against a budget of 4, so selection discards nearly
everything instead of one block at the boundary.
- **The eos at position 37** is a segment boundary the n-gram window
must refuse to read across. Making `shift_right_ignore_eos` ignore
segments entirely leaves the short fixture at 0.000002 — it contains
no eos, so there is nothing to get wrong — and moves the long one by
**0.67**.
The short case deliberately has no eos: a fixture that happens to
contain one is testing segment handling by accident, and would make the
two cases redundant.
### The tie-break
Where the reference and this port genuinely differ: at an exact tie in
the QSA block scores, `topk` keeps one block and we keep another. Ours
is deterministic (lower block index wins); torch's is unspecified. A
tie means every head's dot product was clamped to zero for both blocks
— the indexer scored them equally worthless — so either choice is
defensible, and there is no stable target to match. The fixture avoids
the situation rather than pretending it is resolved.
## Running the comparison
@@ -0,0 +1,50 @@
{
"model_type": "qwen4_exp",
"tie_word_embeddings": false,
"text_config": {
"vocab_size": 32,
"hidden_size": 8,
"num_hidden_layers": 4,
"num_attention_heads": 2,
"num_key_value_heads": 1,
"head_dim": 4,
"max_position_embeddings": 128,
"rms_norm_eps": 1e-06,
"hidden_act": "silu",
"eos_token_id": 3,
"full_attention_interval": 4,
"rope_parameters": {
"rope_theta": 10000.0,
"partial_rotary_factor": 0.5,
"rope_type": "default"
},
"hc_count": 2,
"hc_lowrank": 4,
"ple_layer_ids": [
2
],
"ple_embed_dim": 8,
"ple_conv_kernel_size": 2,
"ngram_size": 3,
"heads_per_ngram": 2,
"ngram_vocab_size_base": 10,
"make_ngram_vocab_size_divisible_by": 4,
"split_ngram_parts": 2,
"indexer_n_heads": 2,
"indexer_kv_heads": 1,
"indexer_head_dim": 4,
"indexer_budget": 8,
"indexer_compress_ratio": 2,
"linear_num_key_heads": 1,
"linear_key_head_dim": 4,
"linear_num_value_heads": 2,
"linear_value_head_dim": 4,
"linear_conv_kernel_dim": 2,
"mamba_ssm_dtype": "float32",
"output_gate_type": "sigmoid",
"num_experts": 2,
"num_experts_per_tok": 1,
"moe_intermediate_size": 4,
"shared_expert_intermediate_size": 4
}
}
@@ -0,0 +1,50 @@
{
"model_type": "qwen4_exp",
"tie_word_embeddings": false,
"text_config": {
"vocab_size": 32,
"hidden_size": 8,
"num_hidden_layers": 4,
"num_attention_heads": 2,
"num_key_value_heads": 1,
"head_dim": 4,
"max_position_embeddings": 128,
"rms_norm_eps": 1e-06,
"hidden_act": "silu",
"eos_token_id": 3,
"full_attention_interval": 4,
"rope_parameters": {
"rope_theta": 10000.0,
"partial_rotary_factor": 0.5,
"rope_type": "default"
},
"hc_count": 2,
"hc_lowrank": 4,
"ple_layer_ids": [
2
],
"ple_embed_dim": 8,
"ple_conv_kernel_size": 2,
"ngram_size": 3,
"heads_per_ngram": 2,
"ngram_vocab_size_base": 10,
"make_ngram_vocab_size_divisible_by": 4,
"split_ngram_parts": 2,
"indexer_n_heads": 2,
"indexer_kv_heads": 1,
"indexer_head_dim": 4,
"indexer_budget": 8,
"indexer_compress_ratio": 2,
"linear_num_key_heads": 1,
"linear_key_head_dim": 4,
"linear_num_value_heads": 2,
"linear_value_head_dim": 4,
"linear_conv_kernel_dim": 2,
"mamba_ssm_dtype": "float32",
"output_gate_type": "sigmoid",
"num_experts": 2,
"num_experts_per_tok": 1,
"moe_intermediate_size": 4,
"shared_expert_intermediate_size": 4
}
}
+127
View File
@@ -339,3 +339,130 @@ fn vision_tower_and_logits_match_reference() {
);
assert_eq!(cr.argmax_ours, cr.argmax_ref, "chunked argmax vs reference");
}
/// `qwen4_exp` logits against the upstream reference (#323).
///
/// Unlike the fixtures above this one needs no `NEURON_REF_MODEL_PATH`
/// and no snapshot: the model is 44 KB of random weights in the real
/// architecture, generated *by* `transformers` itself
/// (`script/generate_qwen4_exp_fixture.py`), so it is checked in and
/// this runs on every push. A tensor name we cannot find is therefore a
/// finding about our loader rather than a fixture that drifted.
///
/// f32 on both sides, for the reason in this file's header: cross-dtype
/// comparison measures bf16 rounding, not implementation error.
///
/// What it covers that the unit tests cannot: the four novel blocks
/// composed. Hyper-connections carrying four residual streams with no
/// layernorm of the usual kind, the PLE n-gram gather on layer 1, QSA
/// on layer 3, and the final mixer collapse with no `model.norm` before
/// `lm_head` — every one of which was written from prose and tested
/// against our own reading of it.
#[test]
fn qwen4_exp_logits_match_reference() {
// The recurrent GDN step and a 12-token prompt: agreement is
// essentially exact (observed 2e-6), so the bound sits just off the
// floor. At 1e-3 this passed with the fused expert's gate and up
// halves swapped (2.4e-4) — a bound wide enough to feel safe is
// wide enough to hide what it was written for.
qwen4_exp_case("qwen4_exp-tiny", 1e-5);
}
/// The same comparison over a prompt long enough, and shaped
/// deliberately enough, to reach two paths the short case cannot.
///
/// **80 tokens** puts the GatedDeltaNet layers on the *chunked* prefill
/// algorithm (#23), which engages at 64 and is a different
/// implementation from the recurrent step the 12-token case exercises —
/// the same reason the qwen3_5 fixture above is over 64 tokens. It also
/// gives QSA up to 40 blocks against a budget of 4, so selection is
/// discarding almost everything rather than one block at the boundary.
///
/// **An eos at position 37** puts a segment boundary inside the n-gram
/// window. `_shift_right_ignore_eos` refuses to read an n-gram across
/// one, and that logic — a cummax over eos positions, per row — is
/// intricate enough to be worth comparing rather than reasoning about.
/// The short case has no eos at all, on purpose: a fixture that happens
/// to contain one tests segment handling by accident.
#[test]
fn qwen4_exp_long_prompt_logits_match_reference() {
// Looser than the short case, and for a stated reason rather than
// to go green: at 80 tokens the GDN layers take the chunked delta
// rule, which is a different *factorisation* of the same
// recurrence, so its f32 rounding differs from the recurrent step
// by construction. Observed 1.3e-5 against the 2e-6 of the short
// case. The bound is still four orders of magnitude below what the
// mutations produce — gate/up swapped moves this fixture by 2.7 —
// so it discriminates exactly as well.
qwen4_exp_case("qwen4_exp-tiny-long", 1e-4);
}
fn qwen4_exp_case(name: &str, max_abs_bound: f32) {
use candle_nn::var_builder::ShardedSafeTensors;
use neuron::harness::arch::qwen4_exp::config::Config as Qwen4ExpConfig;
use neuron::harness::arch::qwen4_exp::model::Qwen4ExpForCausalLM;
let fixture = fixture_root().join(name);
let weights = fixture.join("model.safetensors");
let device = Device::Cpu;
let cfg = Qwen4ExpConfig::from_config_json(
&std::fs::read_to_string(fixture.join("config.json")).unwrap(),
)
.expect("parse the fixture config");
let vb = unsafe { ShardedSafeTensors::var_builder(&[&weights], DType::F32, &device).unwrap() };
let mut model = Qwen4ExpForCausalLM::load(
&cfg,
DType::F32,
None,
&device,
&vb,
std::slice::from_ref(&weights),
None,
)
.expect("load the fixture through the production loader");
let expected = candle_core::safetensors::load(fixture.join("expected.safetensors"), &device)
.expect("read the reference output");
let ids = expected["input_ids"].to_dtype(DType::U32).unwrap();
let reference: Vec<f32> = expected["logits"].flatten_all().unwrap().to_vec1().unwrap();
let logits = model.forward(&ids, 0).expect("forward");
let ours: Vec<f32> = logits
.to_dtype(DType::F32)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
let c = compare(&ours, &reference);
eprintln!(
"{name}: max_abs={:.6} cosine={:.6} argmax ours={} ref={}",
c.max_abs, c.cosine, c.argmax_ours, c.argmax_ref
);
// On failure, break the error down by position before asserting.
// Whether the divergence is confined to one position or spread
// across all of them is the first thing worth knowing: localised
// means a discrete choice went differently (an expert, a QSA
// block), spread means the arithmetic drifted. Finding that out
// cost an hour the first time.
if c.max_abs >= max_abs_bound {
let v = cfg.text_config.vocab_size;
for t in 0..(ours.len() / v) {
let pc = compare(&ours[t * v..(t + 1) * v], &reference[t * v..(t + 1) * v]);
eprintln!(
" pos {t:2}: max_abs={:.6} cosine={:.6}",
pc.max_abs, pc.cosine
);
}
}
assert_eq!(c.argmax_ours, c.argmax_ref, "argmax token diverged");
assert!(c.cosine > 0.999_999, "cosine {:.6} too low", c.cosine);
assert!(
c.max_abs < max_abs_bound,
"max abs diff {:.6} exceeds {max_abs_bound:e}",
c.max_abs
);
}
+128
View File
@@ -0,0 +1,128 @@
"""Generate a qwen4_exp parity fixture from the upstream reference (#323).
Writes, into --out:
config.json the tiny config, as both implementations parse it
model.safetensors the reference's own weights, with the PLE n-gram
table split into `split_ngram_parts` shards on
dim 0 the way the released checkpoint ships it
expected.safetensors input_ids and the reference's logits
The weights are the reference's, so a name we cannot find is a finding
rather than a fixture bug. Norm weights are perturbed away from their
zero init on purpose: at exactly zero, `(1 + w)` and `w` differ, but
`(1 + w)` and `1` do not, and the whole point is to tell those apart.
"""
import argparse, json, os
import torch
from safetensors.torch import save_file
from transformers.models.qwen4_exp import Qwen4ExpConfig, Qwen4ExpForConditionalGeneration
ap = argparse.ArgumentParser()
ap.add_argument("--config", required=True, help="the tiny config.json both implementations parse")
ap.add_argument("--out", required=True, help="fixture directory to write")
# Seed 2 is tie-free; 0, 1 and 4 are not. See the tie check below.
ap.add_argument("--seed", type=int, default=2)
ap.add_argument("--tokens", type=int, default=12)
ap.add_argument(
"--eos-at",
type=int,
default=None,
help="put an eos at this position, so the n-gram window has a segment "
"boundary to refuse to read across",
)
args = ap.parse_args()
torch.manual_seed(args.seed)
torch.use_deterministic_algorithms(True)
raw = json.load(open(args.config))
cfg = Qwen4ExpConfig(**raw)
tc = cfg.text_config
model = Qwen4ExpForConditionalGeneration(cfg).eval()
# Every parameter gets a non-degenerate value. Zeros hide a transposed
# load. But *small* values hide more than that: silu(x) ~ x/2 near the
# origin, so with weights in +-0.1 the SwiGLU is close enough to
# symmetric that reading gate and up the wrong way round changes the
# logits by 2e-4 — inside any tolerance you would write. The scale here
# is chosen to put activations where silu is actually curved, which is
# what makes the ordering observable, while staying below the point
# where the MoE router saturates and top-k stops discriminating.
with torch.no_grad():
for name, p in model.named_parameters():
if "visual" in name or "vision" in name:
continue
p.copy_(torch.empty_like(p).uniform_(-0.8, 0.8))
ids = torch.randint(0, tc.vocab_size, (1, args.tokens), dtype=torch.long)
# eos is not an ordinary token here: `_shift_right_ignore_eos` treats it
# as a segment boundary, so an n-gram never reads across one. Placing it
# is therefore a deliberate choice rather than something to leave to the
# sampler — a fixture that happens to contain one is testing segment
# handling by accident, and one that happens not to is not testing it at
# all.
ids[ids == tc.eos_token_id] = (tc.eos_token_id + 1) % tc.vocab_size
if args.eos_at is not None:
ids[0, args.eos_at] = tc.eos_token_id
# The QSA selection must not rest on a tie.
#
# A block scores exactly 0.0 whenever relu clamps every head's dot
# product, which is common enough that a random fixture hits it. When
# the k-th and (k+1)-th scores are equal, which block gets dropped is
# decided by `topk`'s ordering of equal elements — and torch does not
# specify that, nor does its CPU kernel agree with its CUDA one. A
# fixture containing such a tie measures that unspecified detail rather
# than our arithmetic, so we refuse to emit one.
orig_topk = torch.Tensor.topk
ties = []
def _topk(self, k, *a, **kw):
if self.dim() == 1 and self.numel() <= 64 and k < self.numel():
ordered = sorted(self.detach().tolist(), reverse=True)
if ordered[k - 1] == ordered[k]:
ties.append((self.detach().tolist(), k))
return orig_topk(self, k, *a, **kw)
torch.Tensor.topk = _topk
with torch.no_grad():
out = model(input_ids=ids, use_cache=False)
torch.Tensor.topk = orig_topk
if ties:
raise SystemExit(
f"seed {args.seed} produces {len(ties)} QSA selection tie(s), e.g. "
f"scores={[round(v, 6) for v in ties[0][0]]} k={ties[0][1]}. "
"Pick another --seed: the fixture must isolate the arithmetic."
)
os.makedirs(args.out, exist_ok=True)
sd = {k: v for k, v in model.state_dict().items() if "visual" not in k and "vision" not in k}
# Split the runtime n-gram embedding back into the checkpoint's shard
# layout: dim 0, ascending, per upstream's own note that "the
# checkpoints for it are sharded on dim0" and loading concatenates them.
parts = tc.split_ngram_parts
for key in [k for k in sd if k.endswith("ngram_embedding.weight")]:
table = sd.pop(key)
rows = table.shape[0]
assert rows % parts == 0, f"{rows} rows do not divide into {parts} shards"
per = rows // parts
base = key[: -len(".weight")]
for s in range(parts):
sd[f"{base}.shard_{s}.weight"] = table[s * per : (s + 1) * per].clone()
save_file({k: v.contiguous() for k, v in sd.items()}, os.path.join(args.out, "model.safetensors"))
save_file(
{"input_ids": ids.to(torch.int64), "logits": out.logits.to(torch.float32).contiguous()},
os.path.join(args.out, "expected.safetensors"),
)
with open(os.path.join(args.out, "config.json"), "w") as f:
json.dump(raw, f, indent=2)
print(f"tokens {args.tokens}")
print(f"tensors {len(sd)}")
print(f"logits {tuple(out.logits.shape)}")
print(f"logits absmax {out.logits.abs().max().item():.6f}")
print(f"finite {bool(torch.isfinite(out.logits).all())}")