Compare commits

...
Author SHA1 Message Date
grenadeandClaude Opus 5 26f26cce8c feat(neuron): host-resident routed experts (#318, slice 1)
CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 17s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 14s
CI / Clippy (push) Successful in 2m39s
CI / CUDA type-check (push) Successful in 19m10s
CI / Test (push) Successful in 7m6s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
`qwen4_exp`'s routed experts are 73.0 GB at q4k against 63.7 GB of VRAM
across both of beast's cards. No quant choice and no tensor-parallel
split makes that fit, so the model has never loaded past decoder layer
15. This is the mechanism that lets the experts live somewhere else.

`ExpertResidency::Host` on the load spec quantises them into host
memory instead of onto the device, and `forward_one` moves the
activations across rather than the weights: the weights are ~2.4 GB a
layer and the activations are the handful of rows routed to one expert.
`to_device` is a clone when the devices already match, so the
device-resident path — every other model on the fleet — pays nothing.

The quantiser already round-tripped through host memory to give the
source ownership of its storage, so host residency is one line at the
bottom of it: build the owned tensor where it will live rather than
where it came from. The bytes are in RAM at that point either way, and
the only question is whether they go back.

Opted into explicitly rather than inferred from a device load that
failed. This issue's premise is that placement is measured per machine,
not decided once, and an automatic fallback would quietly decide for
every host while hiding the measurement it exists to inform. cortex
passes `None`: which side of the PCIe bus a model's experts belong on
is a property of the machine, and cortex routes to three different
ones.

The equivalence test is the contract that makes the choice safe to
take: where the weights live may change what a load costs and must not
change what it computes. On a CPU-only test host both devices are the
same one, so it pins the plumbing — the dtype round trip, the transfer
back, the caller getting its own device and dtype — rather than the
transfer; the transfer is exercised on hardware, and what a unit test
can hold is that residency is not silently part of the arithmetic.

Also bundles the load-time parameters into `LoadOptions`. Threading
them one at a time had reached eight arguments, where clippy's
complaint is a readability point rather than a stylistic one:
`load(cfg, dtype, quant, device, vb, paths, cache, onto)` carries two
devices and two ways to name the checkpoint, and nothing at the call
site says which is which.

Not yet done, and deliberately: no device-side expert cache, no
measured hybrid-vs-offload choice, no interleaved layer selection. This
is the slice that makes the model load.

Refs #307, #318, #322.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 19:35:55 +03:00
grenade 13acfaa58b Merge branch 'fix/322-cache-telemetry'
build-prerelease / Resolve version stamps + change detection (push) Successful in 15s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 2m19s
build-prerelease / Build neuron-ada (push) Successful in 2m30s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m8s
build-prerelease / Test (push) Successful in 7m52s
build-prerelease / Build neuron-blackwell (push) Successful in 14m57s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m7s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m10s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m7s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 26s
2026-09-07 19:35:53 +03:00
grenadeandClaude Opus 5 e04091f37f fix(neuron): say what the isq cache actually did (#322)
CI / Config consistency (push) Successful in 10s
CI / Classify changes (push) Successful in 23s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 13s
CI / Clippy (push) Successful in 2m35s
CI / Test (push) Successful in 8m54s
CI / CUDA type-check (push) Successful in 19m11s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The journal reported only that the cache was enabled for a load. A
load that served every layer from disk and a load that served none of
them looked identical from outside, and the difference between them is
eight minutes. That is the shape of #321 — every light green, no way to
tell whether the thing worked — in code shipped an hour after filing
it.

So one line per load: layers served, layers quantised, bytes read,
bytes written.

It fires from a guard rather than the success path, because a load that
OOMs part way is exactly when the question "how much of that was
re-quantised?" is worth answering — and until #318 lands, every
qwen4_exp load on this hardware is that load.

Measured on beast, same window, same OOM at layer 15, so the only
variable is where the fifteen layers came from:

  quantising                     8m 18.9s
  cache on /archive3 (spinning)  1m 48.4s
  cache on /archive1 (NVMe)      0m 23.1s

The NVMe figure is 21.5x, and it is why the artifact directory is now
symlinked out of the spinning cache onto NVMe the way the hot model
weights already are — one symlink, no config, the mechanism the fleet
already had.

Refs #321, #322.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 19:14:40 +03:00
grenade d0e7f185b6 Merge branch 'feat/322-isq-cache'
build-prerelease / Resolve version stamps + change detection (push) Successful in 20s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m39s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 2m26s
build-prerelease / Build neuron-ada (push) Successful in 2m34s
build-prerelease / Test (push) Successful in 6m53s
build-prerelease / Build neuron-blackwell (push) Successful in 14m47s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m7s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m10s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m10s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 26s
2026-09-07 17:56:07 +03:00
grenadeandClaude Opus 5 bf72ce0ffc fix(neuron): keep the disk check off the path the tests exercise (#322)
CI / Config consistency (push) Successful in 12s
CI / Classify changes (push) Successful in 19s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 11s
CI / Clippy (push) Successful in 2m37s
CI / Test (push) Successful in 6m48s
CI / CUDA type-check (push) Successful in 20m16s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The previous commit made the space rule a pure function and tested it
with literal byte counts, which was right, and then left the two
round-trip tests calling `try_store` — which still ran `df` on the
temporary directory and applied that rule. The tests remained coupled
to how full the machine running them happened to be. Only the margin
had changed: instead of needing 2% of the volume free, they needed
about 24 MB. Same defect, quieter.

So the check moves to `store`, the production entry point, and
`try_store` becomes a write and a rename with no policy in it at all.
The tests call `try_store` and no longer ask the host anything about
itself. A unit test that consults the free space of whatever executes
it is not testing this code.

Also stops leaking the temporary file when a write fails part-way.
There was no cleanup, so a failed write left a partial artifact that
nothing would ever remove — and on the failure most likely to cause
one, a full disk, that dead weight is made of the last free bytes.

Refs #322.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 17:33:50 +03:00
grenadeandClaude Opus 5 22cb84d969 fix(neuron): bound the isq cache write, not the filesystem (#322)
CI / Config consistency (push) Successful in 9s
CI / Classify changes (push) Successful in 17s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 14s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 9m15s
CI / CUDA type-check (push) Successful in 19m4s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
CI failed both cache tests with "the artifact just written" — the load
found nothing where a write had just happened. The write had been
refused and `store` swallows its error, as it must, so the test saw
only the absence.

The rule was wrong. It required 2% of the volume to remain free, which
reads sensibly and fails in the case that matters least and happens
most: on a filesystem already below that line it refused a
six-megabyte write that was in no way responsible for the state. The
runner's /tmp is exactly that filesystem.

Bounding each artifact by half the *remaining* space says the thing
actually meant — no single artifact eats the disk — at every size, and
needs to know nothing about how big the volume is. A 73 GB artifact is
refused onto 100 GB free and allowed onto beast's 2 TB; a few
megabytes is never refused, because a few megabytes is never the
problem.

Two changes so this class of failure is not silent again. The rule is
now a pure function tested against those cases directly, rather than
being reachable only through a filesystem in a particular state — which
is exactly what hid the defect until CI ran it. And the tests call a
fallible `try_store` rather than the swallowing `store`, so a write
that does not happen fails them with its reason instead of surfacing
later as a missing artifact. Production still swallows: a cache that
can fail a load is worse than no cache.

Refs #322.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 17:12:19 +03:00
grenadeandClaude Opus 5 de3f9584e2 feat(neuron): cache in-situ quantisation between loads (#322)
CI / CUDA type-check (push) Blocked by required conditions
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 17s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 13s
CI / Clippy (push) Successful in 2m37s
CI / Test (push) Failing after 9m2s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Quantising at load is a pure function of the checkpoint bytes, the
quant name, and the code that does it. None of those change between
loads, and neuron recomputed the answer every time and dropped it on
unload. For qwen4_exp that is ~32 s per decoder layer — about 26
minutes for a full load — paid again on every restart and every
eviction later reversed.

A hit now skips reading the fused expert tensors at all, which is the
point: the cost avoided is the 5.03 GB read *and* the quantisation,
not just the latter.

The key is (model, revision, quant, quantiser version). The last is not
optional and not paranoia: `bf1e80ed` changed what `quant = "q4k"`
means for a 640-wide row, and an artifact written before it is wrong
after it in a way no checksum of the inputs could catch. Bumping the
version orphans stale artifacts rather than misreading one. A
checkpoint whose revision cannot be identified disables the cache
instead of guessing, because a key that might collide across
checkpoints is worse than recomputing.

One GGUF per layer. GGUF because it records a type per tensor, which is
exactly what the per-row fallback produces — a layer whose down_proj is
Q5_0 while its gate and up are Q4K needs no container invented for it —
and because neuron already reads the format. Per layer because it makes
the cache incrementally useful: a load that runs out of VRAM at layer
16 has still written sixteen layers, and the next attempt starts there.
On hardware where the model does not yet fit at all (#318) that is the
difference between the cache being useless and being the thing that
makes iteration bearable.

It lives inside the source's own cache root, beside the weights it
derives from, rather than behind a config key naming a path. That is
the fleet's existing design — beast already keeps hot weights on NVMe
by symlinking them out of the spinning /archive3 cache, with nothing in
config mapping paths to mounts — and an operator wanting these
artifacts on faster storage does the same to one directory. A second,
differently-shaped way to express that intent would be a second place
to get it wrong.

Every failure is swallowed: a miss, a corrupt artifact, a full disk, an
unwritable directory. The load proceeds by quantising exactly as
before. A cache that can fail a load is worse than no cache. Writes go
through a temp file and a rename so an interrupted one cannot leave a
half-artifact that reads as real, and a write that would take the
filesystem below 2% free is skipped with a warning — proportional
rather than a fixed size, because this lives wherever the weights do
and that ranges from a 7.3 TB array to a developer's checkout.

The round-trip test asserts restored experts are bit-identical in
output, not close: there is no arithmetic between writing and reading
that could legitimately differ. It runs at the checkpoint's real widths
because those decide the types involved — 2560 blocks under Q4K, 640
does not — so it covers a layer whose halves are different quant types,
and asserts that mix is actually present rather than assuming it.

Note the parity fixtures cannot cover this: their expert rows are 4 and
8 wide, far below any block size, so they fall back to F16 and
`QMatMul` keeps non-quantised types as plain tensors with nothing to
cache. The contract is tested where the quantisation is.

Refs #307, #318, #322.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 16:53:14 +03:00
grenade 17394ed8c5 Merge branch 'test/323-parity-long-prompt'
build-prerelease / Resolve version stamps + change detection (push) Successful in 17s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m26s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m37s
build-prerelease / Build neuron-ampere (push) Successful in 2m27s
build-prerelease / Test (push) Successful in 6m39s
build-prerelease / Build neuron-blackwell (push) Successful in 14m44s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m7s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m11s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m17s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 28s
2026-09-07 15:56:38 +03:00
grenadeandClaude Opus 5 b3ed41c744 test(qwen4_exp): a second parity case, for the paths the first cannot reach
CI / Config consistency (push) Successful in 11s
CI / Classify changes (push) Successful in 23s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / CUDA type-check (push) Successful in 19m34s
CI / Clippy (push) Successful in 2m40s
CI / Test (push) Successful in 6m45s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Two fixtures rather than one size of the same test. Each reaches
something the other structurally cannot, and a mutation shows it rather
than the commit message asserting it.

**80 tokens** puts the GatedDeltaNet layers on the chunked prefill
algorithm (#23), which engages at 64 — a different factorisation of the
same recurrence, not the same code with more input. The short case only
ever exercised the recurrent step.

**An eos at position 37** puts a segment boundary inside the n-gram
window, which `shift_right_ignore_eos` must refuse to read across. That
logic is a cummax over eos positions per row, intricate enough to be
worth comparing rather than reasoning about. Making it ignore segments
entirely leaves the short fixture at 0.000002 — it has no eos, so there
is nothing to get wrong — and moves the long one by 0.67. The short
case has no eos deliberately: one that happened to contain an eos would
be testing segment handling by accident and would make the two
redundant.

Both agree with the reference: 2e-6 short, 1.3e-5 long, cosine
1.000000 and matching argmax on each.

The bound is now per case, 1e-5 and 1e-4, with the reason at each call
site. The long case is looser because the chunked factorisation rounds
differently in f32 by construction — a stated reason, not a number
raised until the test passed. It still sits four orders of magnitude
below what the mutations produce, so it discriminates exactly as well:
gate/up swapped moves this fixture by 2.7.

One incidental difference, not a defect: our rotary table is sized by
`max_position_embeddings` and a prompt past it fails inside `narrow`,
where the reference computes cos/sin on the fly and extrapolates. It
cannot arise in production, since `derive_limit` advertises no more
context than the model declares — but it is why the fixture config
declares 128 rather than 64.

Refs #307, #312, #323.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 15:28:23 +03:00
grenade c44dec08d6 Merge branch 'fix/323-layer-type-alias'
build-prerelease / Resolve version stamps + change detection (push) Successful in 18s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m40s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m39s
build-prerelease / Build neuron-ampere (push) Successful in 2m44s
build-prerelease / Test (push) Successful in 6m58s
build-prerelease / Build neuron-blackwell (push) Successful in 14m27s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m12s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m12s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 3m57s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 29s
2026-09-07 15:24:08 +03:00
grenadeandClaude Opus 5 e4961246d2 test(qwen4_exp): logits parity against the upstream reference (#323)
CI / Classify changes (push) Successful in 16s
CI / Config consistency (push) Successful in 8s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m34s
CI / Test (push) Successful in 8m57s
CI / CUDA type-check (push) Successful in 18m59s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The architecture was ported from a written spec and every test asserts
it does what we read that spec to mean. This compares it to the
implementation it was ported from: max_abs 0.000002, cosine 1.000000,
same argmax, through all four novel blocks composed.

Unlike the other fixtures here it needs no snapshot and does not
self-skip. It is 44 KB of random weights in the real architecture — 4
layers, 3 linear-attention and 1 QSA, PLE on layer 1 — generated *by*
transformers, so it is checked in and runs on every push. Generating
the weights from the reference rather than by hand means a tensor name
we cannot find is a finding about our loader, not a fixture that
drifted.

Two properties of the fixture are load-bearing and both were learned by
getting them wrong first.

The weights are uniform(-0.8, 0.8) rather than the (-0.1, 0.1) I
started with. Near the origin silu(x) ~ x/2, so the SwiGLU is nearly
symmetric: with small weights, reading the fused expert's gate and up
halves the wrong way round — the one error moe.rs's own comment warns
about — moved the logits by 2.4e-4 and passed. At this scale the same
mutation moves them by 2.7 and takes the argmax with it. A fixture that
cannot express a defect is not evidence about it.

The tolerance is 1e-5 against an observed 2e-6, rather than a
comfortable 1e-3. The same reasoning: at 1e-3 the gate/up swap passed.
A bound wide enough to feel safe is wide enough to hide the thing it
was written for.

The generator refuses to emit a fixture whose QSA selection rests on a
tie. Blocks score exactly 0.0 whenever relu clamps every head, which is
common with random weights, and when the k-th and (k+1)-th scores are
equal it is `topk`'s ordering of equal elements that picks the survivor
— which torch does not specify and whose CPU and CUDA kernels need not
agree. Seeds 0, 1 and 4 tie; seed 2 does not. This is the one place we
genuinely differ from the reference, and it is benign: a tie means the
indexer scored both blocks equally worthless. The fixture avoids the
situation rather than pretending it is resolved.

On failure the test breaks the error down per position before
asserting. Localised means a discrete choice went differently, spread
means the arithmetic drifted; knowing which cost an hour the first
time.

Also carries the `qwen_sparse_attention` layer-type alias, found while
standing the reference up.

Refs #307, #311, #312, #323.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 15:03:57 +03:00
grenadeandClaude Opus 5 d96f4e8fa1 fix(qwen4_exp): read both of upstream's names for a full-attention layer
CI / Test (push) Blocked by required conditions
CI / CUDA type-check (push) Blocked by required conditions
CI / Config consistency (push) Successful in 9s
CI / Classify changes (push) Successful in 24s
CI / Format (push) Successful in 11s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Clippy (push) Successful in 2m55s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Found while standing up the parity reference (#323). The shipped
`config.json` spells its attention layers `full_attention` — 36 linear
to 12 full, checked against the checkpoint on beast. But
`Qwen4ExpTextConfig` *derives* the array as `qwen_sparse_attention`
when a config omits it, which `transformers` main does for any config
carrying only `full_attention_interval`.

We matched the shipped spelling alone. On a config using the derived
one, `is_full_attention` is false for every layer and the model is
built as 48 linear-attention layers — a 3:1-linear architecture read as
4:1, which is wrong in a way that surfaces somewhere else entirely,
when tensors that should exist do not.

Nothing is broken today: the checkpoint we serve uses the spelling we
read, and the derivation we do ourselves emits it too. This is the case
where someone republishes a config, or a second qwen4_exp checkpoint
appears, and the failure has no obvious relationship to its cause.

Refs #307, #323.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 14:52:48 +03:00
grenade d96e2fcfb7 Merge branch 'fix/323-qsa-scores-in-f32'
build-prerelease / Resolve version stamps + change detection (push) Successful in 20s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m42s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m30s
build-prerelease / Build neuron-ampere (push) Successful in 2m35s
build-prerelease / Test (push) Successful in 7m17s
build-prerelease / Build neuron-blackwell (push) Successful in 14m27s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m4s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m9s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m12s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 27s
2026-09-07 14:29:47 +03:00
grenadeandClaude Opus 5 ea532c24dd fix(qwen4_exp): score QSA blocks in f32, as upstream does (#323)
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 16s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 14s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 8m49s
CI / CUDA type-check (push) Successful in 19m12s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
First finding from the parity work, and found by reading the reference
rather than running it: `modeling_qwen4_exp.py` casts both operands
before the indexer matmul,

    scores = torch.matmul(q.float(), block_key_states.float().transpose(-1, -2))
    scores = torch.relu(scores).sum(dim=-1) / math.sqrt(self.index_head_dim)

and we ran the matmul, the relu and the sum over heads in the model
dtype, casting to f32 only afterwards.

It is not a rounding nicety. These scores feed a top-k over as many as
65,536 blocks and the output is the set of positions a query may attend
— a discrete choice, not a weighting. Two blocks whose f32 scores
differ by less than a bf16 tick compare equal, and `select`'s index
tie-break silently decides which past the model sees.

The same argument is already written down two functions up, on `pool`:
"near 0.25 a bf16 tick is 0.002, so a bf16 accumulation and an f32 one
land on the same wrong number". We applied it to the pooling and not to
the scoring, and the unit tests could not see the difference because
they all pass f32 tensors — the production path is bf16 and is the only
place it mattered.

Both the batched production path and the `BlockSelector::scores` helper
now upcast. They are separate implementations of the same arithmetic,
which is its own hazard, and this is the second time that has cost
something.

The test's construction is the argument: four heads contributing 1.0,
with one head of one block contributing 1.0078125 = 1 + 2^-7, which is
exact in bf16. The sums are 4.0 and 4.0078125, and bf16's tick near 4.0
is 2^-5 = 0.03125, so the second rounds onto the first. The inputs
survive the dtype; the comparison does not. Verified the test fails
without the cast.

Refs #307, #311, #323.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 14:08:35 +03:00
grenade 1ae3ba721e Merge branch 'feat/310-mmap-ngram-table'
build-prerelease / Resolve version stamps + change detection (push) Successful in 17s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m42s
build-prerelease / Build helexa-bench binary (push) Successful in 2m24s
build-prerelease / Build neuron-ampere (push) Successful in 2m44s
build-prerelease / Build cortex binary (push) Successful in 3m31s
build-prerelease / Build helexa-angels binary (push) Successful in 3m30s
build-prerelease / Build helexa-upstream binary (push) Successful in 3m32s
build-prerelease / Test (push) Successful in 9m56s
build-prerelease / Build neuron-blackwell (push) Successful in 15m35s
build-prerelease / Package cortex RPM (push) Successful in 26s
build-prerelease / Package helexa-bench RPM (push) Successful in 28s
build-prerelease / Package helexa-upstream RPM (push) Successful in 26s
build-prerelease / Package helexa-angels RPM (push) Successful in 27s
build-prerelease / Build helexa-router binary (push) Successful in 1m27s
build-prerelease / Build helexa-tools binary (push) Successful in 1m35s
build-prerelease / Package helexa-router RPM (push) Successful in 18s
build-prerelease / Build neuron-ada (push) Successful in 2m13s
build-prerelease / Package helexa-tools RPM (push) Successful in 20s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m6s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m5s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m12s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 30s
2026-09-07 09:22:44 +03:00
grenadeandClaude Opus 5 7dfe18b1f5 feat(qwen4_exp): load the n-gram table first, so its failures are cheap
CI / Config consistency (push) Successful in 10s
CI / Classify changes (push) Successful in 18s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 9s
CI / CUDA type-check (push) Successful in 21m56s
CI / Clippy (push) Successful in 2m57s
CI / Test (push) Successful in 9m35s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
`load_ngram` ran after all 48 decoder layers. Now that the table is
mapped rather than made resident, where it sits in the load order costs
nothing — and everything it can get wrong is a tensor name or an extent
that disagrees with the file it is in.

Those are exactly the failures worth having in the first second. Loading
it last meant a checkpoint whose shard names had moved would say so only
after 48 layers of experts had been quantised — about 26 minutes — and
on the hardware we have, where those experts do not currently fit, it
would never say so at all. The mapping could not be validated against
the real 131-shard checkpoint until #318 lands, which is a poor reason
to not know whether it works.

No behaviour changes: the table is a lookup with no dependency on the
layers, and the layers do not consume it until forward.

Refs #307, #310.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 08:46:14 +03:00
grenade a79b9ae5ad Merge branch 'fix/quant-fallback-match-llamacpp'
build-prerelease / Resolve version stamps + change detection (push) Successful in 22s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 2m23s
build-prerelease / Build neuron-ada (push) Successful in 2m31s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m58s
build-prerelease / Test (push) Successful in 7m25s
build-prerelease / Build neuron-blackwell (push) Successful in 16m38s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m17s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m28s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m39s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 29s
2026-09-07 08:44:13 +03:00
grenadeandClaude Opus 5 8c8d20c248 feat(qwen4_exp): map the n-gram table instead of making it resident (#310)
CI / Test (push) Blocked by required conditions
CI / CUDA type-check (push) Blocked by required conditions
CI / Config consistency (push) Successful in 9s
CI / Classify changes (push) Successful in 22s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 11s
CI / Clippy (push) Successful in 2m37s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
The PLE table is 102.4 GB of BF16 across 128 shards — measured from the
checkpoint's own safetensors headers, 128 x [2500012, 160] at 0.80 GB
each. `ShardedNGramTable` puts all of it wherever the VarBuilder
points, which for a CUDA load is a 32 GB card. It is 28.4% of the
checkpoint and the one component in-situ quantisation never reaches, so
no quant choice shrinks it and no amount of tensor parallelism fits it.

A token needs sixteen rows of 160 values from it. About 5 KB, out of
102.4 GB. Almost the entire table is cold at any instant, so the page
cache is the right thing to decide residency and the right answer is to
decide nothing: map the shards where they already lie in the checkpoint
and copy the requested rows out of the mapping.

Mapping the checkpoint's own tensors in place, rather than building a
separate artifact, is FreeToken's `source_from_safetensors` approach.
It works here because the shards are already exactly the shape a gather
wants — contiguous, equal-stride, row-major, one extent each — so there
is nothing to convert and nothing to keep in sync. Extents are resolved
once at load from the safetensors metadata; a gather is then arithmetic
and a memcpy.

The tensor-name prefix comes from the VarBuilder that would otherwise
have loaded the shards, not from the layer index rebuilt by hand. The
two cannot disagree that way, and disagreeing means reading 102.4 GB of
the wrong thing.

`ShardedNGramTable` stays, as the oracle. The new test is the
differential #310 asks for: both tables over the same fixture, every
row of every shard, plus an unsorted request that crosses shard
boundaries in both directions and repeats a row — the oracle sorts
internally and inverts the permutation, the mapped path does not, so a
disagreement there is precisely that inverse being wrong on one side.
Verified it fails if the data offset drops the 8-byte header length.

The whole qwen4_exp suite now loads through the mapped table rather
than the oracle, so the loader tests exercise the residency that
serves.

This does not by itself make the model loadable: the routed experts are
73.0 GB against 63.7 GB of VRAM, which is #318. It removes the larger
of the two blockers, and the one that no other decision could shrink.

Refs #307, #310, #318.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015v1FQqShiRp9rrJEEcgNER
2026-09-07 08:40:44 +03:00
34 changed files with 1912 additions and 67 deletions
Generated
+1
View File
@@ -3147,6 +3147,7 @@ dependencies = [
"half",
"hf-hub",
"image",
"memmap2",
"minijinja",
"minijinja-contrib",
"rayon",
+33
View File
@@ -172,6 +172,39 @@ pub struct ModelSpec {
/// a belief. A request's own `chat_template_kwargs` still wins.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub preserve_thinking: Option<bool>,
/// Where the routed experts of a sparse MoE live (#318).
///
/// `None` — the default, and every model on the fleet today — keeps
/// them on the device with everything else.
///
/// `Some(ExpertResidency::Host)` quantises them into host memory
/// instead and moves the activations across for the expert matmul.
/// That is the only way `qwen4_exp` loads at all: its routed
/// experts are 73.0 GB at q4k against 63.7 GB of VRAM across both
/// of beast's cards, so no quantisation or tensor-parallel split
/// makes them fit. It is slow — a token's experts are ~1.35 GB of
/// host reads, so decode is bounded near 37 tok/s by DDR5
/// bandwidth before compute — and it is the difference between a
/// model that serves and one that does not.
///
/// Explicitly opted into rather than inferred from a failed device
/// load, because #318's whole point is that placement is measured
/// per machine rather than decided once: an automatic fallback
/// would quietly pick for every host and hide the measurement it
/// exists to inform.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub expert_residency: Option<ExpertResidency>,
}
/// Where a sparse MoE's routed experts are held (#318).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum ExpertResidency {
/// On the device, with the rest of the model. The default.
Device,
/// In host memory, with activations crossing for each expert.
Host,
}
/// Per-model token budget advertised by the catalogue or neuron.
+5
View File
@@ -545,6 +545,11 @@ async fn profile_to_spec(
// this host was configured to hold resident.
sampling: profile.sampling.clone(),
preserve_thinking: profile.preserve_thinking,
// Expert residency (#318) is a property of the machine, not of
// the catalogue: whether a host must hold the experts in RAM
// depends on its VRAM, and cortex routes to several. The neuron
// decides from its own config.
expert_residency: None,
}
}
+4
View File
@@ -218,6 +218,10 @@ impl TargetClient {
// ever used to load a model the host has no config for.
sampling: None,
preserve_thinking: None,
// Same limitation, same reason: `GET /models` does not
// report where the experts were placed (#318), so a
// reconstructed spec cannot carry it either.
expert_residency: None,
})
}
+6
View File
@@ -104,6 +104,12 @@ minijinja-contrib = { version = "2", features = ["pycompat"] }
# tp `fused_load` module to read per-rank slices of fused QKV tensors
# without materialising the full tensor on device.
safetensors = "0.8"
# The qwen4_exp PLE n-gram table is 102.4 GB of BF16 across 128 shards
# (#310). It is a pure gather of ~5 KB per token, so it is read straight
# out of the checkpoint by mmap rather than made resident anywhere —
# the page cache decides what stays warm, and almost all of the table is
# cold at any instant.
memmap2 = "0.9"
# Vision capability for Qwen3.6 (Stage A of the vision plan in
# doc/vision-qwen3_6-spec.md). `image` decodes PNG/JPEG/etc from
# the bytes embedded in `data:image/...;base64,...` content parts;
+232 -6
View File
@@ -30,7 +30,7 @@
use anyhow::{Context, Result};
use candle_core::quantized::{GgmlDType, QMatMul, QTensor};
use candle_core::{DType, IndexOp, Module, Tensor};
use candle_core::{DType, Device, IndexOp, Module, Tensor};
use candle_nn::Linear;
use candle_nn::var_builder::ShardedVarBuilder;
@@ -88,6 +88,20 @@ pub(crate) struct QuantizedExpert {
down: QMatMul,
}
impl QuantizedExpert {
/// Where this expert's weights are, when that is knowable.
///
/// `None` for a `QMatMul` that dequantised on construction — it
/// holds a plain tensor whose device is the load device by
/// construction, so the caller's fallback is right.
fn device(&self) -> Option<Device> {
match &self.gate {
QMatMul::QTensor(t) => Some(t.device()),
_ => None,
}
}
}
/// The quantisation type actually usable for rows of `row` elements.
///
/// ggml quantises along the last dimension, and a row must be a whole
@@ -170,11 +184,31 @@ impl Experts {
/// The two halves may not end up the same dtype: see
/// [`quant_for_row`]. `qwen4_exp`'s `down_proj` reduces over 640,
/// which no k-quant can block.
/// `onto` is where the quantised experts come to rest, which need
/// not be where the fused source tensor was read (#318). Passing
/// `Device::Cpu` for a CUDA load leaves the experts in host memory
/// and is what makes a model whose experts exceed VRAM loadable at
/// all. The source is dropped either way, so the peak is one
/// layer's fused pair regardless.
/// Quantise where the source already is. Only the tests want this
/// now — the loaders name a destination explicitly, because for
/// `qwen4_exp` the answer is not the source's device.
#[cfg(test)]
pub(crate) fn quantize_banked(
gate_up: &Tensor,
down: &Tensor,
intermediate: usize,
dtype: GgmlDType,
) -> candle_core::Result<Self> {
Self::quantize_banked_onto(gate_up, down, intermediate, dtype, gate_up.device())
}
pub(crate) fn quantize_banked_onto(
gate_up: &Tensor,
down: &Tensor,
intermediate: usize,
dtype: GgmlDType,
onto: &Device,
) -> candle_core::Result<Self> {
let n = gate_up.dims()[0];
// Decided once, from the shapes, rather than per expert —
@@ -209,8 +243,14 @@ impl Experts {
// the f32 the quantiser wants comes from.
let q = |t: Tensor, dtype: GgmlDType| -> candle_core::Result<QMatMul> {
let dims = t.dims().to_vec();
// This copy to host was always here — it is how the
// source comes to own its storage. Building the owned
// tensor on `onto` rather than on `t.device()` is the
// whole of host residency: the bytes are already in
// RAM at this point, and the only question is whether
// they go back.
let vals = t.to_dtype(DType::F32)?.flatten_all()?.to_vec1::<f32>()?;
let owned = Tensor::from_vec(vals, dims, t.device())?;
let owned = Tensor::from_vec(vals, dims, onto)?;
QMatMul::from_qtensor(QTensor::quantize(&owned, dtype)?)
};
out.push(QuantizedExpert {
@@ -222,6 +262,57 @@ impl Experts {
Ok(Experts::Quantized(out))
}
/// The quantised tensors, named for the on-disk cache (#322).
///
/// `None` for any layout that is not quantised — there is nothing
/// to cache about weights that were never transformed.
pub(crate) fn cache_entries(&self) -> Option<Vec<(String, std::sync::Arc<QTensor>)>> {
let Experts::Quantized(v) = self else {
return None;
};
let of = |m: &QMatMul| match m {
QMatMul::QTensor(t) => Some(t.clone()),
// A dequantised-on-load QMatMul has no quantised tensor to
// write. `CANDLE_DEQUANTIZE_ALL` produces these; caching
// half a layer would be worse than caching none of it.
_ => None,
};
let mut out = Vec::with_capacity(v.len() * 3);
for (i, q) in v.iter().enumerate() {
out.push((format!("e{i}.gate"), of(&q.gate)?));
out.push((format!("e{i}.up"), of(&q.up)?));
out.push((format!("e{i}.down"), of(&q.down)?));
}
Some(out)
}
/// Rebuild from what [`Self::cache_entries`] wrote.
///
/// Errors rather than filling gaps: a layer missing an expert is a
/// truncated artifact, and serving it would mean one expert of 512
/// quietly doing nothing.
pub(crate) fn from_cache_entries(
mut entries: std::collections::HashMap<String, QTensor>,
num_experts: usize,
) -> candle_core::Result<Self> {
let mut v = Vec::with_capacity(num_experts);
for i in 0..num_experts {
let mut take = |part: &str| -> candle_core::Result<QMatMul> {
let name = format!("e{i}.{part}");
match entries.remove(&name) {
Some(t) => QMatMul::from_qtensor(t),
None => candle_core::bail!("cached expert layer is missing '{name}'"),
}
};
v.push(QuantizedExpert {
gate: take("gate")?,
up: take("up")?,
down: take("down")?,
});
}
Ok(Experts::Quantized(v))
}
/// Expert `e` applied to the rows routed to it.
pub(crate) fn forward_one(&self, e: usize, xs: &Tensor) -> candle_core::Result<Tensor> {
match self {
@@ -243,13 +334,25 @@ impl Experts {
// The quantised kernels accumulate in f32 whatever the
// activation dtype, so the cast is explicit here rather
// than implied — and the result comes back in the
// caller's dtype so the routing arithmetic upstream
// does not have to know which layout it got.
// caller's dtype, and on the caller's device, so the
// routing arithmetic upstream does not have to know
// which layout or residency it got.
let dtype = xs.dtype();
let xs32 = xs.to_dtype(DType::F32)?;
let home = q.device().unwrap_or_else(|| xs.device().clone());
// Host-resident experts (#318): the activations go to
// the weights rather than the weights to the
// activations, because the weights are ~2.4 GB a layer
// and the activations are the rows routed to one
// expert. `to_device` is a clone when the devices
// already match, so the device-resident path pays
// nothing for this.
let xs32 = xs.to_dtype(DType::F32)?.to_device(&home)?;
let lhs = candle_nn::ops::silu(&q.gate.forward(&xs32)?)?;
let rhs = q.up.forward(&xs32)?;
q.down.forward(&(lhs * rhs)?)?.to_dtype(dtype)
q.down
.forward(&(lhs * rhs)?)?
.to_device(xs.device())?
.to_dtype(dtype)
}
}
}
@@ -707,6 +810,129 @@ mod tests {
/// a tiny fixture silently cannot be quantised at all, which is the
/// first thing that bites when this is tried on a toy model.
///
/// Host-resident experts compute the same function as device-
/// resident ones (#318).
///
/// This is the contract that makes the whole placement decision
/// safe to take: *where* the weights live may change what a load
/// costs, and must not change what it computes. The activations
/// cross to the weights and the result crosses back, and nothing
/// else about the expert is different.
///
/// On a CPU-only test host both "devices" are the same one, so this
/// pins the plumbing — the dtype round trip, the `to_device` on the
/// way out, the caller getting its own device and dtype back —
/// rather than the transfer itself. The transfer is exercised on
/// hardware, where the two devices genuinely differ; what a unit
/// test can hold is that residency is not silently part of the
/// arithmetic.
#[test]
fn host_resident_experts_compute_the_same_function() {
let (n_experts, hidden, inter) = (2usize, 2560usize, 640usize);
let gate_up = randn(&[n_experts, inter * 2, hidden]);
let down = randn(&[n_experts, hidden, inter]);
let on_device =
Experts::quantize_banked_onto(&gate_up, &down, inter, GgmlDType::Q4K, &Device::Cpu)
.unwrap();
let on_host =
Experts::quantize_banked_onto(&gate_up, &down, inter, GgmlDType::Q4K, &Device::Cpu)
.unwrap();
let xs = randn(&[3, hidden]);
for e in 0..n_experts {
let a: Vec<f32> = on_device
.forward_one(e, &xs)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
let b = on_host.forward_one(e, &xs).unwrap();
// The caller's device and dtype come back unchanged,
// whatever the weights did in between.
assert_eq!(b.dtype(), xs.dtype());
assert!(b.device().same_device(xs.device()));
let b: Vec<f32> = b.flatten_all().unwrap().to_vec1().unwrap();
assert_eq!(a, b, "expert {e} computed differently off-device");
}
}
/// Experts restored from the on-disk cache compute the same
/// function as the ones that filled it (#322).
///
/// The cache's whole risk is that it returns something plausible:
/// the model loads, it serves, and its weights are quietly not the
/// ones it was asked for. So this asserts the restored experts are
/// *bit-identical* in output, not close — there is no arithmetic
/// between writing and reading that could legitimately differ.
///
/// At the checkpoint's real widths, because they are what decide
/// the types involved: 2560 blocks under Q4K, 640 does not and
/// falls back to Q5_0, so this exercises a layer whose halves are
/// *different* quantisation types — exactly the case a container
/// storing one type per file could not represent, and the reason
/// the artifact is GGUF.
#[test]
fn cached_experts_compute_the_same_function() {
let dev = Device::Cpu;
let (n_experts, hidden, inter) = (2usize, 2560usize, 640usize);
let gate_up = randn(&[n_experts, inter * 2, hidden]);
let down = randn(&[n_experts, hidden, inter]);
let fresh = Experts::quantize_banked(&gate_up, &down, inter, GgmlDType::Q4K).unwrap();
let entries = fresh
.cache_entries()
.expect("quantised experts must be cacheable");
// The two halves really did take different types; if they had
// not, this test would not be covering what it claims to.
let dtypes: std::collections::HashSet<GgmlDType> =
entries.iter().map(|(_, t)| t.dtype()).collect();
assert!(
dtypes.len() > 1,
"expected a mix of quant types across gate/up and down, got {dtypes:?}"
);
let dir = tempfile::tempdir().unwrap();
let cache =
crate::harness::isq_cache::IsqCache::new(dir.path(), "test/model", Some("rev0"), "q4k")
.unwrap();
let refs: Vec<(&str, &QTensor)> = entries
.iter()
.map(|(n, t)| (n.as_str(), t.as_ref()))
.collect();
cache
.try_store("layer-0", &refs)
.expect("the write must succeed, and say why if it does not");
let restored = Experts::from_cache_entries(
cache
.load("layer-0", &dev)
.expect("the artifact just written"),
n_experts,
)
.unwrap();
let xs = randn(&[3, hidden]);
for e in 0..n_experts {
let a: Vec<f32> = fresh
.forward_one(e, &xs)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
let b: Vec<f32> = restored
.forward_one(e, &xs)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
assert_eq!(a, b, "expert {e} changed value on the cache round trip");
}
}
/// `qwen4_exp`'s real expert widths, which no k-quant can block.
///
/// `moe_intermediate_size` is 640, so `down_proj` is `[2560, 640]`
@@ -197,10 +197,18 @@ impl TextConfig {
.map(|d| d.eq_ignore_ascii_case("float32") || d.eq_ignore_ascii_case("f32"))
}
/// Both spellings upstream uses. The shipped checkpoint's
/// `layer_types` says `full_attention` (36 linear, 12 full), but
/// `Qwen4ExpTextConfig` *derives* the array with
/// `qwen_sparse_attention` when the checkpoint omits it. Matching
/// only the first is a silent misclassification: every layer would
/// read as linear attention, and a model that is 3:1 linear would
/// look almost right until its full-attention tensors were not
/// found.
pub fn is_full_attention(&self, layer: usize) -> bool {
self.layer_types
.get(layer)
.is_some_and(|t| t == "full_attention")
.is_some_and(|t| t == "full_attention" || t == "qwen_sparse_attention")
}
fn fill_layer_types(&mut self) -> Result<()> {
@@ -474,6 +482,35 @@ mod tests {
}
/// A one-indexed field that reads 0 is a config written against the
/// Upstream has two names for the same layer, and we must read both.
///
/// The shipped `config.json` spells the attention layers
/// `full_attention`. `Qwen4ExpTextConfig` derives them as
/// `qwen_sparse_attention` when the checkpoint omits the array —
/// observed directly from `transformers` `main`, which reports
/// `['linear_attention', ..., 'qwen_sparse_attention']` for a config
/// carrying only `full_attention_interval`.
///
/// Reading only the shipped spelling would classify every layer as
/// linear on such a config. That is not a load error at the point it
/// happens — it is a model built with the wrong 12 layers, which
/// fails later and elsewhere.
#[test]
fn both_upstream_spellings_of_a_full_attention_layer_are_read() {
let mut cfg = Config::from_config_json(SHIPPED).unwrap().text_config;
cfg.layer_types = vec!["linear_attention".into(), "full_attention".into()];
assert!(!cfg.is_full_attention(0));
assert!(cfg.is_full_attention(1), "the shipped spelling");
cfg.layer_types = vec!["linear_attention".into(), "qwen_sparse_attention".into()];
assert!(!cfg.is_full_attention(0));
assert!(
cfg.is_full_attention(1),
"the derived spelling must classify as full attention too"
);
}
/// wrong convention, not a layer.
#[test]
fn rejects_a_zero_ple_layer_id() {
@@ -41,8 +41,6 @@ use crate::harness::arch::qwen3_5::linear_attn::GatedDeltaNet;
use crate::harness::arch::qwen3_5::moe::Qwen3_5MoeBlock;
use crate::harness::arch::qwen3_5::rope::RotaryEmbedding;
use candle_core::quantized::GgmlDType;
use super::config::TextConfig;
use super::full_attn::Attention;
use super::hyper::HyperConnection;
@@ -74,8 +72,8 @@ impl DecoderLayer {
cfg: &TextConfig,
rotary: Arc<RotaryEmbedding>,
layer_idx: usize,
quant: Option<GgmlDType>,
vb: &ShardedVarBuilder,
opts: &super::LoadOptions<'_>,
) -> Result<Self> {
let layer_type = cfg
.layer_types
@@ -92,8 +90,8 @@ impl DecoderLayer {
rotary,
layer_type,
cfg.ple_layers().contains(&layer_idx),
quant,
vb,
opts,
)
}
@@ -108,8 +106,8 @@ impl DecoderLayer {
rotary: Arc<RotaryEmbedding>,
layer_type: &str,
with_ple: bool,
quant: Option<GgmlDType>,
vb: &ShardedVarBuilder,
opts: &super::LoadOptions<'_>,
) -> Result<Self> {
let mixer = match layer_type {
"full_attention" => {
@@ -155,7 +153,7 @@ impl DecoderLayer {
attn_hc: hc("attn_hyper_connection")?,
mlp_hc: hc("mlp_hyper_connection")?,
mixer,
mlp: moe::load(cfg, quant, &vb.pp("mlp"))?,
mlp: moe::load(cfg, &vb.pp("mlp"), opts)?,
ple,
})
}
@@ -44,3 +44,31 @@ pub mod moe;
pub mod mtp;
pub mod ple;
pub mod qsa;
/// What a load needs to know beyond the config and the weights.
///
/// Threading these one at a time got the loader signatures to eight
/// arguments, at which point clippy is making a readability point
/// rather than a stylistic one: `load(cfg, dtype, quant, device, vb,
/// paths, cache, onto)` has two devices and two ways to name the
/// checkpoint in it, and nothing at the call site says which is which.
///
/// They also belong together. Every one of them is a decision about
/// *this load* rather than about the architecture — what precision, on
/// what hardware, reading from where, reusing what.
pub struct LoadOptions<'a> {
/// In-situ quantisation type (#315). `None` keeps the checkpoint's
/// precision, which for this architecture fits nowhere.
pub quant: Option<candle_core::quantized::GgmlDType>,
/// The checkpoint's own files, for mmapping the PLE n-gram table
/// where it already lies (#310). Empty falls back to loading it
/// through the VarBuilder, which puts 102.4 GB wherever that
/// points.
pub safetensors_paths: &'a [std::path::PathBuf],
/// Reuse of a previous load's quantisation (#322).
pub isq_cache: Option<&'a crate::harness::isq_cache::IsqCache>,
/// Where the routed experts come to rest (#318) — the load device
/// for an ordinary model, host memory for one whose experts exceed
/// VRAM.
pub experts_onto: &'a candle_core::Device,
}
@@ -34,7 +34,6 @@
//! See `doc/qwen4_exp-port-spec.md`.
use anyhow::{Context, Result, ensure};
use candle_core::quantized::GgmlDType;
use candle_core::{DType, Device, IndexOp, Module, Tensor};
use candle_nn::var_builder::ShardedVarBuilder;
use candle_nn::{Embedding, Linear};
@@ -45,7 +44,7 @@ use crate::harness::arch::qwen3_5::rope::RotaryEmbedding;
use super::config::{Config, TextConfig};
use super::decoder::DecoderLayer;
use super::hyper::HyperConnection;
use super::ple::{NGramHasher, NGramTable, ShardedNGramTable};
use super::ple::{MmapNGramTable, NGramHasher, NGramTable, ShardedNGramTable};
use crate::harness::arch::snapshot::{KvCacheSnapshot, PleSnapshot};
/// The hashed n-gram lookup, hoisted to the model because it is a
@@ -129,12 +128,17 @@ impl Qwen4ExpForCausalLM {
/// keeps the checkpoint's precision, which for this architecture is
/// useful only on a model small enough to hold: 241.6 GB of BF16
/// experts fit neither 64 GB of VRAM nor 123 GB of host RAM.
/// `safetensors_paths` is the checkpoint's own file set, used only
/// to mmap the PLE n-gram table where it already lies (#310). Empty
/// falls back to [`ShardedNGramTable`], which puts 102.4 GB
/// wherever `vb` points — the oracle, not a residency to serve
/// from.
pub fn load(
cfg: &Config,
dtype: DType,
quant: Option<GgmlDType>,
device: &Device,
vb: &ShardedVarBuilder,
opts: &super::LoadOptions<'_>,
) -> Result<Self> {
let text = &cfg.text_config;
// `RotaryEmbedding` reads a qwen3_5 config; the rope block is the
@@ -150,10 +154,29 @@ impl Qwen4ExpForCausalLM {
let embed_tokens = Embedding::new(embed_weight, text.hidden_size);
let layers_vb = root.pp("layers");
// Before the layers, not after them. The table is mapped rather
// than made resident (#310), so its position in the load order
// costs nothing — but everything it can get wrong is a name or
// an extent, and those are worth finding in the second before
// the expert loop rather than the half-hour after it. Loading
// it last meant a checkpoint whose shard names had moved would
// report that only once 48 layers of experts had been
// quantised, or — since they do not currently fit — never.
let ngram = match text.ple_layers().first() {
Some(layer) => Some(load_ngram(
text,
&layers_vb.pp(*layer).pp("ple"),
opts.safetensors_paths,
device,
)?),
None => None,
};
let mut layers = Vec::with_capacity(text.num_hidden_layers);
for i in 0..text.num_hidden_layers {
layers.push(
DecoderLayer::load(text, rotary.clone(), i, quant, &layers_vb.pp(i))
DecoderLayer::load(text, rotary.clone(), i, &layers_vb.pp(i), opts)
.with_context(|| format!("load decoder layer {i}"))?,
);
}
@@ -175,11 +198,6 @@ impl Qwen4ExpForCausalLM {
.get((text.vocab_size, text.hidden_size), "weight")
.context("load 'lm_head.weight'")?;
let ngram = match text.ple_layers().first() {
Some(layer) => Some(load_ngram(text, &layers_vb.pp(*layer).pp("ple"))?),
None => None,
};
Ok(Self {
embed_tokens,
layers,
@@ -356,7 +374,12 @@ impl Qwen4ExpForCausalLM {
}
}
fn load_ngram(cfg: &TextConfig, ple_vb: &ShardedVarBuilder) -> Result<NGramEmbedding> {
fn load_ngram(
cfg: &TextConfig,
ple_vb: &ShardedVarBuilder,
safetensors_paths: &[std::path::PathBuf],
device: &Device,
) -> Result<NGramEmbedding> {
let vb = ple_vb.pp("ple_embedding");
let heads = cfg.ngram_heads();
let head_dim = cfg.ngram_head_dim();
@@ -382,12 +405,28 @@ fn load_ngram(cfg: &TextConfig, ple_vb: &ShardedVarBuilder) -> Result<NGramEmbed
cfg.split_ngram_parts
);
let table = ShardedNGramTable::load(
&vb.pp("ngram_embedding"),
cfg.split_ngram_parts,
rows,
head_dim,
)?;
// The shard names are whatever this VarBuilder would have asked
// for, so the prefix is taken from it rather than rebuilt from the
// layer index — the two cannot disagree that way, and getting it
// wrong is a 102.4 GB read of the wrong thing.
let ngram_vb = vb.pp("ngram_embedding");
let table: Box<dyn NGramTable> = if safetensors_paths.is_empty() {
Box::new(ShardedNGramTable::load(
&ngram_vb,
cfg.split_ngram_parts,
rows,
head_dim,
)?)
} else {
Box::new(MmapNGramTable::open(
safetensors_paths,
&ngram_vb.prefix(),
cfg.split_ngram_parts,
rows,
head_dim,
device,
)?)
};
Ok(NGramEmbedding {
hasher: NGramHasher::new(
@@ -398,7 +437,7 @@ fn load_ngram(cfg: &TextConfig, ple_vb: &ShardedVarBuilder) -> Result<NGramEmbed
offsets,
cfg.eos_token_id,
)?,
table: Box::new(table),
table,
heads,
head_dim,
carried: Vec::new(),
@@ -757,7 +796,22 @@ mod tests {
candle_core::safetensors::save(&tensors(&cfg.text_config), &path).unwrap();
let vb =
unsafe { ShardedSafeTensors::var_builder(&[&path], DType::F32, &Device::Cpu).unwrap() };
let model = Qwen4ExpForCausalLM::load(&cfg, DType::F32, None, &Device::Cpu, &vb).unwrap();
// Passing the path exercises the mmap table (#310), which is
// the residency production uses — the whole suite then runs
// against the gather that actually serves, not the oracle.
let model = Qwen4ExpForCausalLM::load(
&cfg,
DType::F32,
&Device::Cpu,
&vb,
&super::super::LoadOptions {
quant: None,
safetensors_paths: std::slice::from_ref(&path),
isq_cache: None,
experts_onto: &Device::Cpu,
},
)
.unwrap();
(dir, model)
}
+60 -20
View File
@@ -40,7 +40,6 @@
use anyhow::{Context, Result};
use candle_core::Tensor;
use candle_core::quantized::GgmlDType;
use candle_nn::Linear;
use candle_nn::var_builder::ShardedVarBuilder;
@@ -58,9 +57,10 @@ use super::config::TextConfig;
/// load of `Qwen3.8-Flash-Next` always passes one.
pub fn load(
cfg: &TextConfig,
quant: Option<GgmlDType>,
vb: &ShardedVarBuilder,
opts: &super::LoadOptions<'_>,
) -> Result<Qwen3_5MoeBlock> {
let (quant, isq_cache, experts_onto) = (opts.quant, opts.isq_cache, opts.experts_onto);
let (h, inter) = (cfg.hidden_size, cfg.moe_intermediate_size);
anyhow::ensure!(
cfg.num_experts > 0 && cfg.num_experts_per_tok > 0 && inter > 0,
@@ -78,24 +78,64 @@ pub fn load(
);
let experts_vb = vb.pp("experts");
let gate_up = experts_vb
.get((cfg.num_experts, inter * 2, h), "gate_up_proj")
.with_context(|| format!("load '{}/gate_up_proj'", experts_vb.prefix()))?;
let down = experts_vb
.get((cfg.num_experts, h, inter), "down_proj")
.with_context(|| format!("load '{}/down_proj'", experts_vb.prefix()))?;
check_fused_experts(&gate_up, &down, inter)?;
// The fused pair is the peak: 5.03 GB per layer at BF16. Quantising
// here and letting it drop at the end of this function is what keeps
// the whole 241.6 GB from ever existing.
let experts = match quant {
Some(dtype) => Experts::quantize_banked(&gate_up, &down, inter, dtype)
.with_context(|| format!("quantize experts to {dtype:?}"))?,
None => Experts::Banked {
gate_up,
down,
intermediate: inter,
},
// The cache key is the layer's own tensor prefix, so it cannot
// collide across layers and needs no separate index to be passed
// down alongside it.
let cache_key = experts_vb.prefix().replace(['.', '/'], "-");
// A hit skips reading the fused tensors at all, which is the point:
// the cost being avoided is the 5.03 GB read *and* the ~32 s of
// quantisation, not just the latter.
let cached =
match (quant, isq_cache) {
(Some(_), Some(cache)) => cache.load(&cache_key, vb.device()).and_then(|entries| {
match Experts::from_cache_entries(entries, cfg.num_experts) {
Ok(e) => Some(e),
Err(e) => {
tracing::warn!(key = %cache_key, error = %e,
"isq cache: layer artifact rejected, re-quantising");
None
}
}
}),
_ => None,
};
let experts = match cached {
Some(experts) => experts,
None => {
let gate_up = experts_vb
.get((cfg.num_experts, inter * 2, h), "gate_up_proj")
.with_context(|| format!("load '{}/gate_up_proj'", experts_vb.prefix()))?;
let down = experts_vb
.get((cfg.num_experts, h, inter), "down_proj")
.with_context(|| format!("load '{}/down_proj'", experts_vb.prefix()))?;
check_fused_experts(&gate_up, &down, inter)?;
// The fused pair is the peak: 5.03 GB per layer at BF16.
// Quantising here and letting it drop at the end of this
// function is what keeps the whole 241.6 GB from ever
// existing.
match quant {
Some(dtype) => {
let experts =
Experts::quantize_banked_onto(&gate_up, &down, inter, dtype, experts_onto)
.with_context(|| format!("quantize experts to {dtype:?}"))?;
if let (Some(cache), Some(entries)) = (isq_cache, experts.cache_entries()) {
let refs: Vec<(&str, &candle_core::quantized::QTensor)> = entries
.iter()
.map(|(n, t)| (n.as_str(), t.as_ref()))
.collect();
cache.store(&cache_key, &refs);
}
experts
}
None => Experts::Banked {
gate_up,
down,
intermediate: inter,
},
}
}
};
let (shared_expert, shared_expert_gate) = if cfg.shared_expert_intermediate_size > 0 {
@@ -113,8 +113,17 @@ impl MtpHead {
rotary,
"full_attention",
false,
quant,
&mtp.pp("layers").pp(0),
// The MTP head is not on the serving path (#313), so it
// has nothing to gain from the cache and nothing to
// stale it — and its experts go wherever the rest of it
// does.
&super::LoadOptions {
quant,
safetensors_paths: &[],
isq_cache: None,
experts_onto: vb.device(),
},
)
.context("load mtp.layers.0")?,
mixer: HyperConnection::load(
+293 -1
View File
@@ -66,7 +66,7 @@
//! reimplement. See `doc/qwen4_exp-port-spec.md` §2.
use anyhow::{Context, Result, ensure};
use candle_core::{D, Module, Tensor};
use candle_core::{D, DType, Device, Module, Tensor};
use candle_nn::Linear;
use candle_nn::var_builder::ShardedVarBuilder;
@@ -337,6 +337,216 @@ impl NGramTable for ShardedNGramTable {
}
}
/// The table read straight out of the checkpoint by mmap (#310).
///
/// The shipping residency. Nothing is made resident anywhere: the 128
/// shard tensors are mapped where they already lie in the checkpoint's
/// safetensors files, and a gather copies the requested rows out of
/// the mapping. The page cache decides what stays warm, which for this
/// access pattern is the right decision-maker — a token needs sixteen
/// rows of 160 values, about 5 KB, out of 102.4 GB, so almost the whole
/// table is cold at any instant and nothing is served by making it
/// otherwise.
///
/// This is what makes the model loadable at all. [`ShardedNGramTable`]
/// puts those 102.4 GB wherever the `VarBuilder` points, which for a
/// CUDA load is a 32 GB card; it stays as the oracle this is diffed
/// against, not as a residency anyone would choose.
///
/// Mapping the checkpoint's own tensors in place — rather than building
/// a separate artifact — is FreeToken's `source_from_safetensors`
/// trick, and the reason it works here is that the shards are already
/// exactly what we want: contiguous, equal-stride, row-major, one
/// extent each. There is nothing to convert and nothing to keep in
/// sync.
///
/// Unlike the sharded gather this needs no sort-and-permute: rows are
/// copied in the order asked for, so the inverse permutation that has
/// to be right there cannot be wrong here.
pub struct MmapNGramTable {
/// One mapping per checkpoint file that holds shards. Kept for the
/// table's lifetime because every `gather` reads through them.
maps: Vec<memmap2::Mmap>,
/// Byte position of each shard's row 0: which mapping, and where in
/// it. Resolved once at load so a gather is arithmetic.
shards: Vec<ShardExtent>,
rows_per_shard: usize,
head_dim: usize,
dtype: DType,
row_bytes: usize,
device: Device,
}
/// Where one shard's bytes live.
struct ShardExtent {
map: usize,
offset: usize,
}
impl MmapNGramTable {
/// `prefix` is the full tensor-name prefix of the shards, e.g.
/// `model.language_model.layers.1.ple.ple_embedding.ngram_embedding`
/// — the shards are then `{prefix}.shard_{s}.weight`.
///
/// `files` is the checkpoint's safetensors set; shards may be
/// spread across any of them (in the real checkpoint, 33 of the
/// 131 files carry them), so each is opened and asked what it has.
pub fn open(
files: &[std::path::PathBuf],
prefix: &str,
parts: usize,
rows: usize,
head_dim: usize,
device: &Device,
) -> Result<Self> {
ensure!(parts > 0, "split_ngram_parts must be > 0");
ensure!(
rows.is_multiple_of(parts),
"{rows} rows do not divide evenly into {parts} shards"
);
let rows_per_shard = rows / parts;
let mut maps: Vec<memmap2::Mmap> = Vec::new();
let mut found: Vec<Option<ShardExtent>> = (0..parts).map(|_| None).collect();
let mut dtype: Option<DType> = None;
for path in files {
let file = std::fs::File::open(path)
.with_context(|| format!("open {} for the n-gram table", path.display()))?;
// SAFETY: the same contract candle's own safetensors
// loading takes — the checkpoint is not modified while we
// serve from it. A truncated or rewritten file underneath a
// live mapping is a corrupt-model problem, not one this can
// defend against.
let map = unsafe { memmap2::Mmap::map(&file) }
.with_context(|| format!("mmap {}", path.display()))?;
let (header_len, meta) =
safetensors::SafeTensors::read_metadata(&map).map_err(|e| {
anyhow::anyhow!("read safetensors metadata from {}: {e}", path.display())
})?;
// `data_offsets` are relative to the start of the data
// section, which follows the 8-byte header length and the
// header itself.
let data_start = 8 + header_len;
let mut used = false;
for (s, slot) in found.iter_mut().enumerate() {
if slot.is_some() {
continue;
}
let name = format!("{prefix}.shard_{s}.weight");
let Some(info) = meta.info(&name) else {
continue;
};
ensure!(
info.shape == [rows_per_shard, head_dim],
"'{name}' is {:?}, expected [{rows_per_shard}, {head_dim}]",
info.shape
);
let dt = st_dtype(info.dtype)
.with_context(|| format!("'{name}' has a dtype the gather cannot read"))?;
match dtype {
None => dtype = Some(dt),
Some(seen) => ensure!(
seen == dt,
"shard dtypes disagree: {seen:?} then {dt:?} at '{name}'"
),
}
*slot = Some(ShardExtent {
map: maps.len(),
offset: data_start + info.data_offsets.0,
});
used = true;
}
if used {
maps.push(map);
}
}
let dtype = dtype.context("no n-gram shard was found in any checkpoint file")?;
let shards = found
.into_iter()
.enumerate()
.map(|(s, e)| e.with_context(|| format!("'{prefix}.shard_{s}.weight' is in no file")))
.collect::<Result<Vec<_>>>()?;
let row_bytes = head_dim * dtype.size_in_bytes();
// Every extent must actually contain its rows: a header that
// disagrees with the file length would otherwise surface as a
// panic inside a gather, mid-request.
for (s, extent) in shards.iter().enumerate() {
let end = extent.offset + rows_per_shard * row_bytes;
ensure!(
end <= maps[extent.map].len(),
"shard {s} claims bytes {}..{end} of a {}-byte file",
extent.offset,
maps[extent.map].len()
);
}
tracing::info!(
parts,
rows,
head_dim,
?dtype,
files = maps.len(),
gib = (rows * row_bytes) as f64 / (1024.0 * 1024.0 * 1024.0),
"ple: n-gram table mapped from the checkpoint, not made resident (#310)"
);
Ok(Self {
maps,
shards,
rows_per_shard,
head_dim,
dtype,
row_bytes,
device: device.clone(),
})
}
}
impl NGramTable for MmapNGramTable {
fn gather(&self, rows: &[i64]) -> Result<Tensor> {
let total = (self.shards.len() * self.rows_per_shard) as i64;
if let Some(bad) = rows.iter().find(|r| **r < 0 || **r >= total) {
anyhow::bail!("row {bad} is outside the table's 0..{total}");
}
let mut out = vec![0u8; rows.len() * self.row_bytes];
for (slot, &row) in rows.iter().enumerate() {
let shard = row as usize / self.rows_per_shard;
let within = row as usize % self.rows_per_shard;
let extent = &self.shards[shard];
let from = extent.offset + within * self.row_bytes;
out[slot * self.row_bytes..(slot + 1) * self.row_bytes]
.copy_from_slice(&self.maps[extent.map][from..from + self.row_bytes]);
}
Ok(Tensor::from_raw_buffer(
&out,
self.dtype,
&[rows.len(), self.head_dim],
&self.device,
)?)
}
fn head_dim(&self) -> usize {
self.head_dim
}
}
/// safetensors' dtype vocabulary is wider than candle's. Only the
/// float widths a checkpoint would store an embedding table in are
/// accepted; anything else is a checkpoint we do not understand rather
/// than one to guess at.
fn st_dtype(dtype: safetensors::Dtype) -> Result<DType> {
Ok(match dtype {
safetensors::Dtype::BF16 => DType::BF16,
safetensors::Dtype::F16 => DType::F16,
safetensors::Dtype::F32 => DType::F32,
other => anyhow::bail!("unsupported n-gram table dtype {other:?}"),
})
}
/// The consumption half of PLE: `Qwen4ExpTextPLE` minus the table.
///
/// Given the gathered n-gram embedding `e` and the four-stream residual
@@ -903,6 +1113,88 @@ mod tests {
// ---- the table ----
/// The mmap table against the device-resident one, row for row.
///
/// This is the differential #310 asks for: [`ShardedNGramTable`] is
/// the oracle — obviously correct, and unusable at 102.4 GB — and
/// [`MmapNGramTable`] is what serves. They must agree on every row
/// of every shard, in whatever order they are asked, or the model
/// quietly reads somebody else's n-gram.
///
/// The row list is chosen to break the things that can break: shard
/// boundaries in both directions, the first and last row of the
/// table, a repeat, and an order that is not sorted (the oracle
/// sorts internally and inverts the permutation; the mmap path does
/// not, so a disagreement here is exactly that inverse being wrong
/// on one side).
#[test]
fn the_mapped_table_agrees_with_the_resident_one() {
let dev = Device::Cpu;
let (parts, rows_per_shard, head_dim) = (4usize, 5usize, 3usize);
let prefix = "model.language_model.layers.1.ple.ple_embedding.ngram_embedding";
// Row r holds [r, r + 0.5, r + 0.25] — distinct per row, and
// not a pattern a stride mistake would reproduce.
let mut tensors = std::collections::HashMap::new();
for s in 0..parts {
let mut v = Vec::with_capacity(rows_per_shard * head_dim);
for r in 0..rows_per_shard {
let global = (s * rows_per_shard + r) as f32;
v.extend_from_slice(&[global, global + 0.5, global + 0.25]);
}
tensors.insert(
format!("{prefix}.shard_{s}.weight"),
Tensor::from_vec(v, (rows_per_shard, head_dim), &dev).unwrap(),
);
}
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("model.safetensors");
candle_core::safetensors::save(&tensors, &path).unwrap();
let vb = unsafe {
candle_nn::var_builder::ShardedSafeTensors::var_builder(&[&path], DType::F32, &dev)
.unwrap()
};
let total = parts * rows_per_shard;
let oracle = ShardedNGramTable::load(&vb.pp(prefix), parts, total, head_dim).unwrap();
let mapped = MmapNGramTable::open(
std::slice::from_ref(&path),
prefix,
parts,
total,
head_dim,
&dev,
)
.unwrap();
assert_eq!(mapped.head_dim(), oracle.head_dim());
let asked: Vec<i64> = vec![19, 0, 5, 4, 9, 10, 19, 1, 15, 14, 7, 7];
let flat = |t: Tensor| -> Vec<f32> { t.flatten_all().unwrap().to_vec1().unwrap() };
let from_oracle = flat(oracle.gather(&asked).unwrap());
let from_mapped = flat(mapped.gather(&asked).unwrap());
assert_eq!(
from_mapped, from_oracle,
"the mapped gather disagrees with the resident one"
);
// And that both are right, rather than wrong together.
assert_eq!(&from_mapped[..3], &[19.0, 19.5, 19.25]);
assert_eq!(&from_mapped[3..6], &[0.0, 0.5, 0.25]);
// Every row of the table, in order, so no shard is untouched.
let all: Vec<i64> = (0..total as i64).collect();
assert_eq!(
flat(mapped.gather(&all).unwrap()),
flat(oracle.gather(&all).unwrap())
);
// The empty gather and the out-of-range guard behave alike.
assert_eq!(mapped.gather(&[]).unwrap().dims(), &[0, head_dim]);
assert!(mapped.gather(&[total as i64]).is_err());
assert!(mapped.gather(&[-1]).is_err());
}
/// Rows come back in the order asked for, not the order they were
/// fetched in. Two shards of three rows, requested out of order and
/// across the boundary: the shard-major gather returns them sorted,
@@ -134,7 +134,13 @@ impl BlockSelector {
/// Returns `[n_blocks]`.
pub fn scores(&self, q: &Tensor, pooled: &Tensor) -> candle_core::Result<Tensor> {
debug_assert_eq!(q.dims2()?, (self.n_heads, self.head_dim));
let per_head = q.matmul(&pooled.t()?)?.relu()?;
// f32 for the same reason `pool` is: this is a *selection*
// function. See the note there and on the batched path in
// `Indexer::positions_from`.
let per_head = q
.to_dtype(DType::F32)?
.matmul(&pooled.to_dtype(DType::F32)?.t()?)?
.relu()?;
per_head.sum(0)? / (self.head_dim as f64).sqrt()
}
@@ -505,15 +511,30 @@ impl Indexer {
// in one batched matmul: every head and query shares the same
// block keys, so fold them into the row axis rather than
// broadcasting the keys across heads.
let q_rows = q.reshape((batch, self.n_heads * seq_len, self.head_dim))?;
let keys = blocks.squeeze(1)?.transpose(1, 2)?.contiguous()?;
//
// In f32, matching upstream, which casts both operands before
// the matmul (`torch.matmul(q.float(), block_key_states.float()
// .transpose(-1, -2))`). It is not a rounding nicety: these
// scores feed a top-k over as many as 65,536 blocks, and the
// output is the *set of positions a query may attend*. Two
// blocks whose f32 scores differ by less than a bf16 tick —
// 0.03125 near 4.0 — compare equal in bf16 and the tie-break
// decides instead, so bf16 does not perturb the answer, it
// changes which past the model sees. Same argument as `pool`,
// which we already got right.
let q_rows = q
.reshape((batch, self.n_heads * seq_len, self.head_dim))?
.to_dtype(DType::F32)?;
let keys = blocks
.squeeze(1)?
.transpose(1, 2)?
.contiguous()?
.to_dtype(DType::F32)?;
let per_head = q_rows
.matmul(&keys)?
.reshape((batch, self.n_heads, seq_len, n_blocks))?
.relu()?;
let scores = (per_head.sum(1)? / (self.head_dim as f64).sqrt())?
.to_dtype(DType::F32)?
.to_vec3::<f32>()?;
let scores = (per_head.sum(1)? / (self.head_dim as f64).sqrt())?.to_vec3::<f32>()?;
let mut out = Vec::with_capacity(batch);
for row in scores.iter().take(batch) {
@@ -649,6 +670,63 @@ mod tests {
/// mean is exactly 0.25 — and so is an f32 mean cast back to bf16,
/// because the true 0.2507 sits below the first bf16 tick above
/// 0.25. Both mistakes land on the same wrong number, which is why
/// Scoring is f32 even when the cache is bf16, because it decides
/// *which past a query sees*, not merely by how much.
///
/// Upstream casts both operands before the matmul
/// (`torch.matmul(q.float(), block_key_states.float().T)`); we did
/// not, and ran the matmul, the relu and the sum over heads in the
/// model dtype.
///
/// The construction is the whole argument. Two blocks, four heads,
/// per-head contributions of 1.0 and — for one head of one block —
/// 1.0078125, which is 1 + 2^-7 and therefore exact in bf16. Their
/// sums are 4.0 and 4.0078125, and bf16's tick near 4.0 is 2^-5 =
/// 0.03125, so the second rounds back onto the first. The inputs
/// survive the dtype; the *comparison* does not. In bf16 the two
/// blocks tie and `select`'s index tie-break picks the wrong one,
/// silently, for as long as the model is served.
#[test]
fn scoring_is_f32_even_when_the_cache_is_bf16() {
let dev = Device::Cpu;
// Budget 4 over 4-wide blocks keeps exactly one block, so the
// comparison decides the answer rather than being absorbed by a
// budget large enough to take both.
let qsa = BlockSelector::new(4, 2, 4, 4).unwrap();
// Block 0 reads dimension 0 of each head, block 1 reads
// dimension 1, so the two scores are independent sums.
let pooled = Tensor::from_vec(vec![1.0f32, 0.0, 0.0, 1.0], (2, 2), &dev)
.unwrap()
.to_dtype(DType::BF16)
.unwrap();
let q = Tensor::from_vec(
vec![
1.0f32, 1.0, // head 0: block 0 <- 1.0, block 1 <- 1.0
1.0, 1.0, // head 1
1.0, 1.0, // head 2
1.0, 1.0078125, // head 3: block 1 gets the extra tick
],
(4, 2),
&dev,
)
.unwrap()
.to_dtype(DType::BF16)
.unwrap();
let scores: Vec<f32> = qsa.scores(&q, &pooled).unwrap().to_vec1().unwrap();
assert!(
scores[1] > scores[0],
"bf16 scoring lost the distinction between the blocks: {scores:?}"
);
// And the selection that rests on it picks the right block.
assert_eq!(
qsa.select(&scores, 8).unwrap(),
vec![4, 5, 6, 7],
"the higher-scoring block should be the one kept"
);
}
/// this asserts the value rather than the dtype.
#[test]
fn pooling_accumulates_in_f32() {
+83 -2
View File
@@ -2738,6 +2738,12 @@ impl CandleHarness {
// Resolved out here: `spec` is borrowed and does not outlive the
// blocking task.
let isq = super::quant::parse_quant_string(spec.quant.as_deref())?;
let isq_cache = self.isq_cache_for(
&spec.model_id,
&source_id.scheme,
spec.quant.as_deref(),
&safetensors_paths,
);
let arch = tokio::task::spawn_blocking(move || -> Result<ModelArch> {
let cfg_text = std::fs::read_to_string(&config_path).context("read config.json")?;
@@ -2860,9 +2866,17 @@ impl CandleHarness {
let model = super::arch::qwen4_exp::model::Qwen4ExpForCausalLM::load(
&cfg,
dtype,
isq,
&device_for_load,
&sharded_vb,
&super::arch::qwen4_exp::LoadOptions {
quant: isq,
safetensors_paths: &safetensors_paths,
isq_cache: isq_cache.as_ref(),
// The CPU path already has everything in
// host memory, so residency is not a
// choice here.
experts_onto: &device_for_load,
},
)
.context("build qwen4_exp model")?;
Ok(ModelArch::Qwen4Exp(Box::new(model)))
@@ -2899,6 +2913,46 @@ impl CandleHarness {
/// them match the requested quant — that case is a typo, not an ISQ
/// request, and turning it into a silent 360 GB dense load would be
/// a worse answer than the error.
/// The quantisation cache for this load, or `None` when the quant
/// is unset or the checkpoint's revision cannot be identified
/// (#322).
///
/// It lives **inside the source's own cache root**, beside the
/// weights it is derived from, rather than behind a config key
/// naming a path. That is the fleet's existing design: an operator
/// who wants this on faster storage symlinks it there, exactly as
/// the hot model weights on beast are symlinked from the spinning
/// `/archive3` cache onto NVMe. Adding a second, differently-shaped
/// way to say the same thing would leave two places to get it
/// wrong.
///
/// A revision we cannot read means we cannot prove two checkpoints
/// are the same bytes, and a cache key that might collide is worse
/// than no cache — so an unrecognised layout disables it rather
/// than guessing.
fn isq_cache_for(
&self,
model_id: &str,
scheme: &str,
quant: Option<&str>,
safetensors_paths: &[PathBuf],
) -> Option<super::isq_cache::IsqCache> {
let quant = quant.filter(|q| !q.is_empty())?;
let root = self.hf_cache_for(scheme).ok()?.path().join("neuron-isq");
let revision = super::isq_cache::revision_from_paths(safetensors_paths);
let cache = super::isq_cache::IsqCache::new(&root, model_id, revision, quant);
match &cache {
Some(c) => tracing::info!(model = model_id, quant, dir = %c.dir().display(),
"isq cache: enabled for this load (#322)"),
None => tracing::info!(
model = model_id,
quant,
"isq cache: disabled — the checkpoint revision could not be identified"
),
}
cache
}
async fn quant_selects_a_gguf(
&self,
spec: &ModelSpec,
@@ -3957,8 +4011,35 @@ impl Harness for CandleHarness {
// too, not just the CPU one — parsed here because
// `spec` does not cross the channel.
let isq = super::quant::parse_quant_string(spec.quant.as_deref())?;
// Built here rather than in the worker: the worker has
// no view of the config, and an `IsqCache` is a path,
// so it crosses the channel freely.
let isq_cache = self.isq_cache_for(
&spec.model_id,
&source_id.scheme,
spec.quant.as_deref(),
&safetensors_paths,
);
let experts_on_host = matches!(
spec.expert_residency,
Some(cortex_core::harness::ExpertResidency::Host)
);
if experts_on_host {
tracing::info!(
model = %spec.model_id,
"loading routed experts into host memory (#318); the device \
holds the dense path and the activations cross per expert"
);
}
let load = w
.load_dense(config_path, safetensors_paths, spec.model_id.clone(), isq)
.load_dense(
config_path,
safetensors_paths,
spec.model_id.clone(),
isq,
isq_cache,
experts_on_host,
)
.await
.map_err(|e| anyhow::anyhow!("worker load_dense: {e:#}"))?;
// Prefix snapshots (#11): the built model answers for
@@ -147,6 +147,8 @@ pub(crate) fn run(device_index: u32, rx: Receiver<Job>, poisoned: Arc<AtomicBool
safetensors_paths,
model_id,
quant,
isq_cache,
experts_on_host,
reply,
} => {
let result = load_dense_inner(
@@ -155,6 +157,8 @@ pub(crate) fn run(device_index: u32, rx: Receiver<Job>, poisoned: Arc<AtomicBool
&safetensors_paths,
&model_id,
quant,
isq_cache.as_ref(),
experts_on_host,
)
.map(|arch| {
// Ask the model, once, while it is still in
@@ -934,12 +938,32 @@ fn load_gguf_inner(
}
/// Load a dense safetensors model on the worker thread.
/// Log the cache summary when the load leaves scope, successful or
/// not. A load that OOMs half way is precisely when the question "how
/// much of that was re-quantised?" is worth answering.
fn scopeguard_summary<'a>(
cache: Option<&'a crate::harness::isq_cache::IsqCache>,
model_id: &'a str,
) -> impl Drop + 'a {
struct S<'a>(Option<&'a crate::harness::isq_cache::IsqCache>, &'a str);
impl Drop for S<'_> {
fn drop(&mut self) {
if let Some(c) = self.0 {
c.log_summary(self.1);
}
}
}
S(cache, model_id)
}
fn load_dense_inner(
device: &candle_core::Device,
config_path: &std::path::Path,
safetensors_paths: &[std::path::PathBuf],
model_id: &str,
quant: Option<candle_core::quantized::GgmlDType>,
isq_cache: Option<&crate::harness::isq_cache::IsqCache>,
experts_on_host: bool,
) -> anyhow::Result<ModelArch> {
use anyhow::Context;
use candle_core::DType;
@@ -948,6 +972,10 @@ fn load_dense_inner(
use candle_transformers::models::qwen3 as qwen3_dense;
use candle_transformers::models::qwen3_moe as qwen3_moe_dense;
// Report what the cache did, however the load ends: a load that
// fails part-way is exactly when someone wants to know how much of
// it was cheap.
let _summary = scopeguard_summary(isq_cache, model_id);
let cfg_text = std::fs::read_to_string(config_path).context("read config.json")?;
crate::harness::candle::check_dense_config_supported(&cfg_text, model_id)?;
// Peek at model_type to choose the family before the typed
@@ -1056,9 +1084,21 @@ fn load_dense_inner(
let model = crate::harness::arch::qwen4_exp::model::Qwen4ExpForCausalLM::load(
&cfg,
dtype,
quant,
device,
&sharded_vb,
&crate::harness::arch::qwen4_exp::LoadOptions {
quant,
safetensors_paths,
isq_cache,
// Host residency (#318) is what makes a model whose
// experts exceed VRAM loadable; everything else in
// the model still lands on the device.
experts_onto: if experts_on_host {
&candle_core::Device::Cpu
} else {
device
},
},
)
.context("build qwen4_exp model")?;
Ok(ModelArch::Qwen4Exp(Box::new(model)))
@@ -1770,11 +1810,18 @@ mod tests {
for model_type in crate::harness::candle::DENSE_SUPPORTED_MODEL_TYPES {
let path = dir.path().join(format!("{model_type}.json"));
std::fs::write(&path, format!(r#"{{"model_type": "{model_type}"}}"#)).unwrap();
let msg =
match load_dense_inner(&candle_core::Device::Cpu, &path, &[], model_type, None) {
Ok(_) => panic!("a bare config must not build a model"),
Err(e) => format!("{e:#}"),
};
let msg = match load_dense_inner(
&candle_core::Device::Cpu,
&path,
&[],
model_type,
None,
None,
false,
) {
Ok(_) => panic!("a bare config must not build a model"),
Err(e) => format!("{e:#}"),
};
assert!(
!msg.contains("unrouted"),
"`{model_type}` is in DENSE_SUPPORTED_MODEL_TYPES but load_dense_inner \
@@ -132,6 +132,11 @@ pub enum Job {
/// already parsed. `None` loads at the checkpoint dtype.
/// Archs that cannot quantise in situ ignore it.
quant: Option<candle_core::quantized::GgmlDType>,
/// Where to cache the quantised weights (#322). Built on the
/// async side, which is the only side that can see the config.
isq_cache: Option<crate::harness::isq_cache::IsqCache>,
/// Keep the routed experts in host memory (#318).
experts_on_host: bool,
reply: oneshot::Sender<Result<DenseLoad>>,
},
/// Remove the model from the slab and drop it. The `Drop` runs on
@@ -222,6 +222,8 @@ impl DeviceWorkerHandle {
safetensors_paths: Vec<std::path::PathBuf>,
model_id: String,
quant: Option<candle_core::quantized::GgmlDType>,
isq_cache: Option<crate::harness::isq_cache::IsqCache>,
experts_on_host: bool,
) -> Result<DenseLoad, WorkerError> {
if self.poisoned.load(Ordering::Acquire) {
return Err(WorkerError::Poisoned {
@@ -235,6 +237,8 @@ impl DeviceWorkerHandle {
safetensors_paths,
model_id,
quant,
isq_cache,
experts_on_host,
reply: reply_tx,
})
.map_err(|_| WorkerError::Gone {
@@ -1246,6 +1250,8 @@ mod tests {
vec![fixture.join("model.safetensors")],
"qwen3_next-tiny".into(),
None,
None,
false,
)
.await
.expect("load tiny fixture")
+2
View File
@@ -1262,6 +1262,8 @@ mod tests {
vec![fixture.join("model.safetensors")],
"qwen3_next-tiny".into(),
None,
None,
false,
)
.await
.expect("load fixture")
+451
View File
@@ -0,0 +1,451 @@
//! On-disk cache for in-situ quantisation (#322).
//!
//! Quantising a model's weights at load is a pure function of the
//! checkpoint bytes, the requested type, and the code that does it.
//! None of those change between loads, and yet neuron recomputed the
//! answer every time and threw it away on unload. For `qwen4_exp` that
//! is **~32 s per decoder layer, about 26 minutes for a full load**,
//! measured on beast — paid again on every restart and every eviction
//! that is later reversed.
//!
//! So the quantised tensors are written out once and read back after.
//!
//! ## The key must include our own version
//!
//! `(checkpoint revision, quant name)` is not enough. `bf1e80ed`
//! changed what `quant = "q4k"` *means* for a 640-wide row — k-quants
//! cannot block it, so it now falls back to a 32-block type — and an
//! artifact written before that commit and read after it would be
//! silently wrong in a way no checksum of the inputs could catch.
//! [`QUANTISER_VERSION`] is part of the path, and bumping it orphans
//! every stale artifact rather than misreading one.
//!
//! ## What is cached, and what is not
//!
//! One GGUF per layer, holding that layer's quantised experts. GGUF
//! because it stores a **type per tensor**, which is exactly what the
//! per-row fallback produces — a layer whose `down_proj` is `Q5_0`
//! while its `gate`/`up` are `Q4K` is representable without inventing
//! a container for it — and because neuron already reads the format.
//!
//! Per layer rather than per model because it makes the cache
//! **incrementally useful**: a load that runs out of VRAM at layer 16
//! has still written sixteen layers, and the next attempt starts
//! there. On hardware where the model does not yet fit at all (#318),
//! that is the difference between the cache being useless and being
//! the thing that makes iteration bearable.
//!
//! ## Where it lives
//!
//! Inside the source's own cache root — `<hf cache>/neuron-isq/…` —
//! beside the weights it is derived from, and not behind a config key
//! naming a path.
//!
//! That is deliberate and it is the fleet's existing design. beast
//! already keeps its hot model weights on NVMe by symlinking them out
//! of the spinning `/archive3` cache into `/archive1`, with no config
//! mapping paths to mounts. An operator who wants these artifacts on
//! faster storage does the same thing to one directory. A second,
//! differently-shaped way to express the same intent would be a second
//! place to get it wrong, and the first one already works.
//!
//! Size is the thing to know before symlinking: an artifact is roughly
//! the served size of the model's quantised weights — ~73 GB for
//! `qwen4_exp` at q4k.
//!
//! ## It is an optimisation, and never load-bearing
//!
//! Every failure here is swallowed: a miss, a corrupt file, a full
//! disk, a directory nobody can write. The load proceeds by
//! quantising, exactly as it did before this module existed. A cache
//! that can fail a load is worse than no cache.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
use candle_core::Device;
use candle_core::quantized::{QTensor, gguf_file};
/// Bump when the *meaning* of a quant name changes: a different
/// fallback table, a different rounding, a different layout. Artifacts
/// written by an older quantiser are then ignored rather than
/// misread, and get overwritten in place as layers are re-quantised.
///
/// 1: the initial format. Per-layer GGUF, tensors named
/// `e{index}.{gate,up,down}`, llama.cpp's `tensor_type_fallback`
/// table for rows that do not divide the requested block size.
pub const QUANTISER_VERSION: u32 = 1;
/// Refuse a write that would take more than this share of the free
/// space.
///
/// The rule is about *the write*, not the filesystem. An earlier
/// version required 2% of the volume to remain free, which read
/// sensibly and was wrong in the case that matters least and happens
/// most: on a filesystem already below that line, it refused a
/// six-megabyte write that was in no way responsible. CI found it
/// immediately.
///
/// Bounding each artifact by half the remaining space says the thing
/// actually meant — no single artifact eats the disk — at every size,
/// and needs to know nothing about how big the volume is. An
/// artifact for one 180 B model is ~73 GB, so on a volume with 100 GB
/// left it is refused and on beast's 2 TB of free `/archive3` it is
/// not. A few megabytes is never refused, because a few megabytes is
/// never the problem.
const MAX_SHARE_OF_FREE: f64 = 0.5;
/// A cache scoped to one (model, revision, quant, quantiser version).
pub struct IsqCache {
dir: PathBuf,
hits: AtomicUsize,
misses: AtomicUsize,
read: AtomicU64,
written: AtomicU64,
}
impl IsqCache {
/// `revision` should identify the checkpoint bytes — the snapshot
/// directory name, for an hf-hub cache. `None` disables the cache
/// rather than guessing, because a wrong revision is the one
/// failure mode that produces a confidently wrong model.
pub fn new(base: &Path, model_id: &str, revision: Option<&str>, quant: &str) -> Option<Self> {
let revision = revision?;
let safe = |s: &str| {
s.chars()
.map(|c| {
if c.is_ascii_alphanumeric() || c == '-' || c == '.' {
c
} else {
'-'
}
})
.collect::<String>()
};
let dir = base
.join(safe(model_id))
.join(safe(revision))
.join(format!("{}-v{QUANTISER_VERSION}", safe(quant)));
Some(Self {
dir,
hits: AtomicUsize::new(0),
misses: AtomicUsize::new(0),
read: AtomicU64::new(0),
written: AtomicU64::new(0),
})
}
/// Read a layer's quantised tensors onto `device`.
///
/// `None` for a miss and for anything that goes wrong reading —
/// a truncated artifact from an interrupted write reads as absent,
/// and is rewritten by the load that follows.
pub fn load(&self, key: &str, device: &Device) -> Option<HashMap<String, QTensor>> {
let path = self.dir.join(format!("{key}.gguf"));
let Ok(mut file) = std::fs::File::open(&path) else {
self.misses.fetch_add(1, Ordering::Relaxed);
return None;
};
let content = match gguf_file::Content::read(&mut file) {
Ok(c) => c,
Err(e) => {
self.misses.fetch_add(1, Ordering::Relaxed);
tracing::warn!(path = %path.display(), error = %e,
"isq cache: unreadable artifact, re-quantising this layer");
return None;
}
};
let names: Vec<String> = content.tensor_infos.keys().cloned().collect();
let mut out = HashMap::with_capacity(names.len());
for name in names {
match content.tensor(&mut file, &name, device) {
Ok(t) => {
out.insert(name, t);
}
Err(e) => {
self.misses.fetch_add(1, Ordering::Relaxed);
tracing::warn!(path = %path.display(), tensor = %name, error = %e,
"isq cache: artifact tensor failed to read, re-quantising this layer");
return None;
}
}
}
self.hits.fetch_add(1, Ordering::Relaxed);
self.read.fetch_add(
out.values()
.map(|t| t.storage_size_in_bytes() as u64)
.sum::<u64>(),
Ordering::Relaxed,
);
Some(out)
}
/// Write a layer's quantised tensors.
///
/// Through a temporary file and a rename, so an interrupted write
/// cannot leave a half-artifact that a later load would read as
/// real. Errors are logged and swallowed.
/// Write a layer, refusing one that would take an unreasonable
/// share of the disk. The production entry point: every failure is
/// logged and swallowed.
pub fn store(&self, key: &str, tensors: &[(&str, &QTensor)]) {
let bytes: u64 = tensors
.iter()
.map(|(_, t)| t.storage_size_in_bytes() as u64)
.sum();
if let Some(free) = free_bytes(&self.dir)
&& too_large_for(bytes, free)
{
tracing::warn!(
dir = %self.dir.display(), key,
gib = bytes as f64 / (1024.0 * 1024.0 * 1024.0),
free_gib = free as f64 / (1024.0 * 1024.0 * 1024.0),
"isq cache: refusing a write that would take over {:.0}% of the free space",
MAX_SHARE_OF_FREE * 100.0,
);
return;
}
match self.try_store(key, tensors) {
Ok(()) => {
self.written.fetch_add(bytes, Ordering::Relaxed);
}
Err(e) => tracing::warn!(dir = %self.dir.display(), key, error = %e,
"isq cache: could not write this layer; the load continues uncached"),
}
}
/// Say, once per load, what the cache actually did.
///
/// Without this the journal shows only that the cache was enabled,
/// which is the shape of problem #321 is about: every light green
/// and no way to tell from the outside whether the thing worked.
/// A load that served nothing from cache and a load that served
/// everything looked identical, and the difference between them is
/// eight minutes.
pub fn log_summary(&self, model_id: &str) {
let gib = |b: u64| b as f64 / (1024.0 * 1024.0 * 1024.0);
tracing::info!(
model = model_id,
dir = %self.dir.display(),
layers_from_cache = self.hits.load(Ordering::Relaxed),
layers_quantised = self.misses.load(Ordering::Relaxed),
read_gib = gib(self.read.load(Ordering::Relaxed)),
written_gib = gib(self.written.load(Ordering::Relaxed)),
"isq cache: load complete (#322)"
);
}
/// Write a layer. Nothing else — no policy, and in particular no
/// question asked of the filesystem about its size.
///
/// That separation is the point. The disk check lives in
/// [`Self::store`], so the round-trip tests exercise the write and
/// the read without their result depending on how full the machine
/// running them happens to be. A unit test that consults the free
/// space of whatever host executes it is not testing this code.
pub(crate) fn try_store(&self, key: &str, tensors: &[(&str, &QTensor)]) -> anyhow::Result<()> {
std::fs::create_dir_all(&self.dir)?;
let final_path = self.dir.join(format!("{key}.gguf"));
let tmp = self.dir.join(format!(".{key}.{}.tmp", std::process::id()));
let write = (|| -> anyhow::Result<()> {
let mut f = std::fs::File::create(&tmp)?;
gguf_file::write(&mut f, &[], tensors)?;
Ok(())
})();
if let Err(e) = write {
// A partial artifact is dead weight that nothing will ever
// clean up — and on the failure this is most likely to see,
// a full disk, it is dead weight made of the last free
// bytes.
let _ = std::fs::remove_file(&tmp);
return Err(e);
}
std::fs::rename(&tmp, &final_path)?;
Ok(())
}
pub fn dir(&self) -> &Path {
&self.dir
}
}
/// Whether a write of `bytes` takes an unreasonable share of `free`.
///
/// Split out from the write so the rule can be tested without needing
/// a filesystem in a particular state — which is precisely what made
/// the previous rule's defect invisible until CI ran it.
fn too_large_for(bytes: u64, free: u64) -> bool {
bytes as f64 > free as f64 * MAX_SHARE_OF_FREE
}
/// Free bytes on the filesystem holding `path`, or `None` if it cannot
/// be determined — in which case the caller writes anyway rather than
/// refusing on a guess. `-P` is what guarantees one line per
/// filesystem, so a long device name cannot wrap the fields apart.
fn free_bytes(path: &Path) -> Option<u64> {
let out = std::process::Command::new("df")
.arg("-kP")
.arg(path)
.output()
.ok()?;
let text = String::from_utf8(out.stdout).ok()?;
let kb = text.lines().nth(1)?.split_whitespace().nth(3)?;
kb.parse::<u64>().ok().map(|kb| kb * 1024)
}
/// The revision an hf-hub-cached checkpoint was resolved at, taken
/// from the snapshot directory its files live in.
///
/// `.../models--org--name/snapshots/<revision>/model-00001-of-N.safetensors`
/// — the parent directory *is* the revision, which is what makes it a
/// safe cache key: two checkpoints with different bytes cannot share
/// one. Returns `None` for any layout that is not that, so an
/// unrecognised path disables the cache instead of colliding.
pub fn revision_from_paths(paths: &[PathBuf]) -> Option<&str> {
let first = paths.first()?;
let dir = first.parent()?;
let name = dir.file_name()?.to_str()?;
let looks_like_snapshot = dir.parent()?.file_name()?.to_str()? == "snapshots";
(looks_like_snapshot && !name.is_empty()).then_some(name)
}
#[cfg(test)]
mod tests {
use super::*;
use candle_core::Tensor;
use candle_core::quantized::GgmlDType;
/// A cached layer must dequantise to the same numbers the fresh
/// quantisation produced.
///
/// This is the whole contract. A cache that returns *something*
/// plausible is worse than no cache, because the model loads, it
/// serves, and its weights are quietly not the ones it was asked
/// for. Round-tripping the dequantised values rather than the
/// bytes is deliberate: it is what the model actually consumes.
#[test]
fn a_cached_tensor_dequantises_to_what_was_stored() {
let dev = Device::Cpu;
let dir = tempfile::tempdir().unwrap();
let cache = IsqCache::new(dir.path(), "org/model", Some("rev0"), "q4k").unwrap();
// 256-wide so the k-quant can block it at all.
let src = Tensor::rand(-1f32, 1f32, (8, 256), &dev).unwrap();
let q = QTensor::quantize(&src, GgmlDType::Q4K).unwrap();
let before: Vec<f32> = q
.dequantize(&dev)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
cache
.try_store("layer-0", &[("e0.gate", &q)])
.expect("the write must succeed, and say why if it does not");
let got = cache
.load("layer-0", &dev)
.expect("the artifact just written");
let after: Vec<f32> = got["e0.gate"]
.dequantize(&dev)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
assert_eq!(
before, after,
"a cached tensor changed value on the round trip"
);
assert_eq!(got["e0.gate"].dtype(), GgmlDType::Q4K, "and kept its type");
}
/// The key separates everything that changes the answer.
///
/// The quantiser version is the one that is easy to leave out and
/// impossible to detect: `bf1e80ed` changed what `q4k` means for a
/// 640-wide row, and an artifact written before it is wrong after
/// it in a way no checksum of the *inputs* would reveal.
#[test]
fn the_key_separates_model_revision_quant_and_quantiser() {
let root = std::path::Path::new("/cache");
let base = IsqCache::new(root, "org/model", Some("rev0"), "q4k").unwrap();
let others = [
IsqCache::new(root, "org/other", Some("rev0"), "q4k").unwrap(),
IsqCache::new(root, "org/model", Some("rev1"), "q4k").unwrap(),
IsqCache::new(root, "org/model", Some("rev0"), "q6k").unwrap(),
];
for other in &others {
assert_ne!(base.dir(), other.dir());
}
assert!(
base.dir()
.to_str()
.unwrap()
.contains(&format!("v{QUANTISER_VERSION}")),
"the quantiser version must be in the path: {}",
base.dir().display()
);
}
/// No revision, no cache. A key that might collide across
/// checkpoints is worse than recomputing.
#[test]
fn an_unidentifiable_revision_disables_the_cache() {
assert!(IsqCache::new(std::path::Path::new("/cache"), "org/model", None, "q4k").is_none());
// An hf-hub snapshot path yields its revision; anything else
// does not, rather than yielding a directory name that happens
// to be there.
let hf = PathBuf::from(
"/cache/models--org--model/snapshots/abc123/model-00001-of-00002.safetensors",
);
assert_eq!(revision_from_paths(&[hf]), Some("abc123"));
assert_eq!(
revision_from_paths(&[PathBuf::from("/tmp/model.safetensors")]),
None
);
assert_eq!(revision_from_paths(&[]), None);
}
/// The space guard bounds the *write*, not the filesystem.
///
/// The rule it replaced required a fixed share of the volume to
/// remain free, which refused a six-megabyte write on a disk that
/// was already full — punishing the write least responsible for
/// the state. These cases are the ones that matter: a 73 GB
/// artifact must not land on a volume with 100 GB left, and a few
/// megabytes must always be allowed, however full things are.
#[test]
fn the_space_guard_bounds_the_write_not_the_volume() {
let gib = 1024 * 1024 * 1024;
// A 73 GB artifact against beast's ~2 TB of free /archive3.
assert!(!too_large_for(73 * gib, 2000 * gib));
// The same artifact where it would eat most of what is left.
assert!(too_large_for(73 * gib, 100 * gib));
// Six megabytes on a nearly-full disk: never the problem.
assert!(!too_large_for(6 * 1024 * 1024, 200 * 1024 * 1024));
// Exactly half is allowed; past half is not.
assert!(!too_large_for(50, 100));
assert!(too_large_for(51, 100));
// And nothing is writable onto nothing.
assert!(too_large_for(1, 0));
}
/// A miss and a corrupt artifact are the same thing to the caller:
/// re-quantise. Neither may propagate as an error.
#[test]
fn a_missing_or_corrupt_artifact_reads_as_absent() {
let dev = Device::Cpu;
let dir = tempfile::tempdir().unwrap();
let cache = IsqCache::new(dir.path(), "org/model", Some("rev0"), "q4k").unwrap();
assert!(cache.load("layer-0", &dev).is_none(), "a miss");
std::fs::create_dir_all(cache.dir()).unwrap();
std::fs::write(cache.dir().join("layer-0.gguf"), b"not a gguf file at all").unwrap();
assert!(cache.load("layer-0", &dev).is_none(), "a corrupt artifact");
}
}
+1
View File
@@ -9,6 +9,7 @@ pub mod context_limit;
pub mod device_worker;
pub mod engine;
pub mod image;
pub mod isq_cache;
pub mod prefix_cache;
pub mod preflight;
pub mod preprocess;
+1
View File
@@ -403,6 +403,7 @@ mod tests {
devices: None,
sampling: None,
preserve_thinking: None,
expert_residency: None,
}
}
+4
View File
@@ -31,6 +31,7 @@ async fn test_load_default_models_skips_unknown_harness() {
devices: None,
sampling: None,
preserve_thinking: None,
expert_residency: None,
},
ModelSpec {
model_id: "model-b".into(),
@@ -40,6 +41,7 @@ async fn test_load_default_models_skips_unknown_harness() {
devices: None,
sampling: None,
preserve_thinking: None,
expert_residency: None,
},
];
@@ -101,6 +103,7 @@ async fn test_load_default_models_skipped_on_driver_mismatch() {
devices: None,
sampling: None,
preserve_thinking: None,
expert_residency: None,
}];
let activation = ActivationTracker::new(&specs);
let reason = "host NVIDIA driver/library mismatch (userspace NVML 580.159 vs loaded \
@@ -204,6 +207,7 @@ fn qwen_spec() -> ModelSpec {
devices: None,
sampling: None,
preserve_thinking: None,
expert_residency: None,
}
}
+1
View File
@@ -55,6 +55,7 @@ async fn test_candle_qwen3_load_unload_lifecycle() {
devices: Some(vec![0]),
sampling: None,
preserve_thinking: None,
expert_residency: None,
};
registry
+76
View File
@@ -12,6 +12,82 @@ HuggingFace reference" claim to checked-in numbers.
| `qwen3_5-0.8b-text` | Qwen/Qwen3.5-0.8B | text (>64-token prompt → chunked GDN prefill) | f32 | `text_logits_match_reference` |
| `qwen3_5-0.8b-vision` | Qwen/Qwen3.5-0.8B | 448×448 synthetic image + prompt | f32 | `vision_tower_and_logits_match_reference` |
| `qwen3_6-27b-text` | Qwen/Qwen3.6-27B | text | bf16 | manual (see below) |
| `qwen4_exp-tiny` | random weights, real architecture | text, 12 tokens, no eos | f32 | `qwen4_exp_logits_match_reference` |
| `qwen4_exp-tiny-long` | random weights, real architecture | text, 80 tokens, eos at 37 | f32 | `qwen4_exp_long_prompt_logits_match_reference` |
## `qwen4_exp-tiny` is different, and runs in CI
The others replay a real snapshot and self-skip without
`NEURON_REF_MODEL_PATH`. This one is 44 KB of random weights in the
real architecture — 4 layers (3 linear-attention, 1 QSA), PLE on layer
1, 2 experts — generated *by* `transformers` itself via
[`script/generate_qwen4_exp_fixture.py`](../../../../../script/generate_qwen4_exp_fixture.py).
So it is checked in, needs no weights, and runs on every push.
That matters because the whole `qwen4_exp` architecture was ported from
a written spec (#308) and tested against our own reading of it. This is
the only thing in the tree that compares it to the implementation it
was ported from (#323).
Two properties of the fixture are load-bearing, and both were learned
by getting them wrong:
- **Weights are `uniform(-0.8, 0.8)`, not `uniform(-0.1, 0.1)`.** Near
the origin `silu(x) ~ x/2`, so the SwiGLU is nearly symmetric and
reading the fused expert's gate and up halves the wrong way round
moved the logits by 2.4e-4 — inside any tolerance worth writing. At
the larger scale the same mutation moves them by 2.7.
- **The generator refuses to emit a fixture containing a QSA selection
tie.** A block scores exactly 0.0 whenever relu clamps every head, so
ties are common with random weights; when the k-th and (k+1)-th
scores are equal, which block is dropped is decided by `topk`'s
ordering of equal elements, which torch does not specify and whose
CPU and CUDA kernels need not agree. Seeds 0, 1 and 4 produce ties;
seed 2 (the default) does not. See "the tie-break" below.
Regenerate with the reference from `transformers` **main** — the 5.9.0
release does not carry `qwen4_exp`:
```sh
python -m venv --system-site-packages .venv
.venv/bin/pip install --no-deps "transformers @ git+https://github.com/huggingface/transformers@main"
.venv/bin/pip install "tokenizers>=0.23.1,<0.24" "safetensors>=0.8.0"
.venv/bin/python script/generate_qwen4_exp_fixture.py \
--config crates/neuron/tests/fixtures/numerical/qwen4_exp-tiny/config.json \
--out crates/neuron/tests/fixtures/numerical/qwen4_exp-tiny
```
### Why there are two cases
They are not the same test at two sizes. Each reaches a path the other
structurally cannot, and a mutation demonstrates it:
- **80 tokens** puts the GatedDeltaNet layers on the *chunked* prefill
algorithm (#23), which engages at 64. That is a different
factorisation of the same recurrence, not the same code with more
input — which is also why the long case's bound is 1e-4 against the
short case's 1e-5 (observed 1.3e-5 vs 2e-6). It also gives QSA up to
40 blocks against a budget of 4, so selection discards nearly
everything instead of one block at the boundary.
- **The eos at position 37** is a segment boundary the n-gram window
must refuse to read across. Making `shift_right_ignore_eos` ignore
segments entirely leaves the short fixture at 0.000002 — it contains
no eos, so there is nothing to get wrong — and moves the long one by
**0.67**.
The short case deliberately has no eos: a fixture that happens to
contain one is testing segment handling by accident, and would make the
two cases redundant.
### The tie-break
Where the reference and this port genuinely differ: at an exact tie in
the QSA block scores, `topk` keeps one block and we keep another. Ours
is deterministic (lower block index wins); torch's is unspecified. A
tie means every head's dot product was clamped to zero for both blocks
— the indexer scored them equally worthless — so either choice is
defensible, and there is no stable target to match. The fixture avoids
the situation rather than pretending it is resolved.
## Running the comparison
@@ -0,0 +1,50 @@
{
"model_type": "qwen4_exp",
"tie_word_embeddings": false,
"text_config": {
"vocab_size": 32,
"hidden_size": 8,
"num_hidden_layers": 4,
"num_attention_heads": 2,
"num_key_value_heads": 1,
"head_dim": 4,
"max_position_embeddings": 128,
"rms_norm_eps": 1e-06,
"hidden_act": "silu",
"eos_token_id": 3,
"full_attention_interval": 4,
"rope_parameters": {
"rope_theta": 10000.0,
"partial_rotary_factor": 0.5,
"rope_type": "default"
},
"hc_count": 2,
"hc_lowrank": 4,
"ple_layer_ids": [
2
],
"ple_embed_dim": 8,
"ple_conv_kernel_size": 2,
"ngram_size": 3,
"heads_per_ngram": 2,
"ngram_vocab_size_base": 10,
"make_ngram_vocab_size_divisible_by": 4,
"split_ngram_parts": 2,
"indexer_n_heads": 2,
"indexer_kv_heads": 1,
"indexer_head_dim": 4,
"indexer_budget": 8,
"indexer_compress_ratio": 2,
"linear_num_key_heads": 1,
"linear_key_head_dim": 4,
"linear_num_value_heads": 2,
"linear_value_head_dim": 4,
"linear_conv_kernel_dim": 2,
"mamba_ssm_dtype": "float32",
"output_gate_type": "sigmoid",
"num_experts": 2,
"num_experts_per_tok": 1,
"moe_intermediate_size": 4,
"shared_expert_intermediate_size": 4
}
}
@@ -0,0 +1,50 @@
{
"model_type": "qwen4_exp",
"tie_word_embeddings": false,
"text_config": {
"vocab_size": 32,
"hidden_size": 8,
"num_hidden_layers": 4,
"num_attention_heads": 2,
"num_key_value_heads": 1,
"head_dim": 4,
"max_position_embeddings": 128,
"rms_norm_eps": 1e-06,
"hidden_act": "silu",
"eos_token_id": 3,
"full_attention_interval": 4,
"rope_parameters": {
"rope_theta": 10000.0,
"partial_rotary_factor": 0.5,
"rope_type": "default"
},
"hc_count": 2,
"hc_lowrank": 4,
"ple_layer_ids": [
2
],
"ple_embed_dim": 8,
"ple_conv_kernel_size": 2,
"ngram_size": 3,
"heads_per_ngram": 2,
"ngram_vocab_size_base": 10,
"make_ngram_vocab_size_divisible_by": 4,
"split_ngram_parts": 2,
"indexer_n_heads": 2,
"indexer_kv_heads": 1,
"indexer_head_dim": 4,
"indexer_budget": 8,
"indexer_compress_ratio": 2,
"linear_num_key_heads": 1,
"linear_key_head_dim": 4,
"linear_num_value_heads": 2,
"linear_value_head_dim": 4,
"linear_conv_kernel_dim": 2,
"mamba_ssm_dtype": "float32",
"output_gate_type": "sigmoid",
"num_experts": 2,
"num_experts_per_tok": 1,
"moe_intermediate_size": 4,
"shared_expert_intermediate_size": 4
}
}
+130
View File
@@ -339,3 +339,133 @@ fn vision_tower_and_logits_match_reference() {
);
assert_eq!(cr.argmax_ours, cr.argmax_ref, "chunked argmax vs reference");
}
/// `qwen4_exp` logits against the upstream reference (#323).
///
/// Unlike the fixtures above this one needs no `NEURON_REF_MODEL_PATH`
/// and no snapshot: the model is 44 KB of random weights in the real
/// architecture, generated *by* `transformers` itself
/// (`script/generate_qwen4_exp_fixture.py`), so it is checked in and
/// this runs on every push. A tensor name we cannot find is therefore a
/// finding about our loader rather than a fixture that drifted.
///
/// f32 on both sides, for the reason in this file's header: cross-dtype
/// comparison measures bf16 rounding, not implementation error.
///
/// What it covers that the unit tests cannot: the four novel blocks
/// composed. Hyper-connections carrying four residual streams with no
/// layernorm of the usual kind, the PLE n-gram gather on layer 1, QSA
/// on layer 3, and the final mixer collapse with no `model.norm` before
/// `lm_head` — every one of which was written from prose and tested
/// against our own reading of it.
#[test]
fn qwen4_exp_logits_match_reference() {
// The recurrent GDN step and a 12-token prompt: agreement is
// essentially exact (observed 2e-6), so the bound sits just off the
// floor. At 1e-3 this passed with the fused expert's gate and up
// halves swapped (2.4e-4) — a bound wide enough to feel safe is
// wide enough to hide what it was written for.
qwen4_exp_case("qwen4_exp-tiny", 1e-5);
}
/// The same comparison over a prompt long enough, and shaped
/// deliberately enough, to reach two paths the short case cannot.
///
/// **80 tokens** puts the GatedDeltaNet layers on the *chunked* prefill
/// algorithm (#23), which engages at 64 and is a different
/// implementation from the recurrent step the 12-token case exercises —
/// the same reason the qwen3_5 fixture above is over 64 tokens. It also
/// gives QSA up to 40 blocks against a budget of 4, so selection is
/// discarding almost everything rather than one block at the boundary.
///
/// **An eos at position 37** puts a segment boundary inside the n-gram
/// window. `_shift_right_ignore_eos` refuses to read an n-gram across
/// one, and that logic — a cummax over eos positions, per row — is
/// intricate enough to be worth comparing rather than reasoning about.
/// The short case has no eos at all, on purpose: a fixture that happens
/// to contain one tests segment handling by accident.
#[test]
fn qwen4_exp_long_prompt_logits_match_reference() {
// Looser than the short case, and for a stated reason rather than
// to go green: at 80 tokens the GDN layers take the chunked delta
// rule, which is a different *factorisation* of the same
// recurrence, so its f32 rounding differs from the recurrent step
// by construction. Observed 1.3e-5 against the 2e-6 of the short
// case. The bound is still four orders of magnitude below what the
// mutations produce — gate/up swapped moves this fixture by 2.7 —
// so it discriminates exactly as well.
qwen4_exp_case("qwen4_exp-tiny-long", 1e-4);
}
fn qwen4_exp_case(name: &str, max_abs_bound: f32) {
use candle_nn::var_builder::ShardedSafeTensors;
use neuron::harness::arch::qwen4_exp::config::Config as Qwen4ExpConfig;
use neuron::harness::arch::qwen4_exp::model::Qwen4ExpForCausalLM;
let fixture = fixture_root().join(name);
let weights = fixture.join("model.safetensors");
let device = Device::Cpu;
let cfg = Qwen4ExpConfig::from_config_json(
&std::fs::read_to_string(fixture.join("config.json")).unwrap(),
)
.expect("parse the fixture config");
let vb = unsafe { ShardedSafeTensors::var_builder(&[&weights], DType::F32, &device).unwrap() };
let mut model = Qwen4ExpForCausalLM::load(
&cfg,
DType::F32,
&device,
&vb,
&neuron::harness::arch::qwen4_exp::LoadOptions {
quant: None,
safetensors_paths: std::slice::from_ref(&weights),
isq_cache: None,
experts_onto: &device,
},
)
.expect("load the fixture through the production loader");
let expected = candle_core::safetensors::load(fixture.join("expected.safetensors"), &device)
.expect("read the reference output");
let ids = expected["input_ids"].to_dtype(DType::U32).unwrap();
let reference: Vec<f32> = expected["logits"].flatten_all().unwrap().to_vec1().unwrap();
let logits = model.forward(&ids, 0).expect("forward");
let ours: Vec<f32> = logits
.to_dtype(DType::F32)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
let c = compare(&ours, &reference);
eprintln!(
"{name}: max_abs={:.6} cosine={:.6} argmax ours={} ref={}",
c.max_abs, c.cosine, c.argmax_ours, c.argmax_ref
);
// On failure, break the error down by position before asserting.
// Whether the divergence is confined to one position or spread
// across all of them is the first thing worth knowing: localised
// means a discrete choice went differently (an expert, a QSA
// block), spread means the arithmetic drifted. Finding that out
// cost an hour the first time.
if c.max_abs >= max_abs_bound {
let v = cfg.text_config.vocab_size;
for t in 0..(ours.len() / v) {
let pc = compare(&ours[t * v..(t + 1) * v], &reference[t * v..(t + 1) * v]);
eprintln!(
" pos {t:2}: max_abs={:.6} cosine={:.6}",
pc.max_abs, pc.cosine
);
}
}
assert_eq!(c.argmax_ours, c.argmax_ref, "argmax token diverged");
assert!(c.cosine > 0.999_999, "cosine {:.6} too low", c.cosine);
assert!(
c.max_abs < max_abs_bound,
"max abs diff {:.6} exceeds {max_abs_bound:e}",
c.max_abs
);
}
+1
View File
@@ -108,6 +108,7 @@ fn spec(model_id: &str, tp: Option<u32>, quant: Option<&str>) -> ModelSpec {
devices: None,
sampling: None,
preserve_thinking: None,
expert_residency: None,
}
}
+128
View File
@@ -0,0 +1,128 @@
"""Generate a qwen4_exp parity fixture from the upstream reference (#323).
Writes, into --out:
config.json the tiny config, as both implementations parse it
model.safetensors the reference's own weights, with the PLE n-gram
table split into `split_ngram_parts` shards on
dim 0 the way the released checkpoint ships it
expected.safetensors input_ids and the reference's logits
The weights are the reference's, so a name we cannot find is a finding
rather than a fixture bug. Norm weights are perturbed away from their
zero init on purpose: at exactly zero, `(1 + w)` and `w` differ, but
`(1 + w)` and `1` do not, and the whole point is to tell those apart.
"""
import argparse, json, os
import torch
from safetensors.torch import save_file
from transformers.models.qwen4_exp import Qwen4ExpConfig, Qwen4ExpForConditionalGeneration
ap = argparse.ArgumentParser()
ap.add_argument("--config", required=True, help="the tiny config.json both implementations parse")
ap.add_argument("--out", required=True, help="fixture directory to write")
# Seed 2 is tie-free; 0, 1 and 4 are not. See the tie check below.
ap.add_argument("--seed", type=int, default=2)
ap.add_argument("--tokens", type=int, default=12)
ap.add_argument(
"--eos-at",
type=int,
default=None,
help="put an eos at this position, so the n-gram window has a segment "
"boundary to refuse to read across",
)
args = ap.parse_args()
torch.manual_seed(args.seed)
torch.use_deterministic_algorithms(True)
raw = json.load(open(args.config))
cfg = Qwen4ExpConfig(**raw)
tc = cfg.text_config
model = Qwen4ExpForConditionalGeneration(cfg).eval()
# Every parameter gets a non-degenerate value. Zeros hide a transposed
# load. But *small* values hide more than that: silu(x) ~ x/2 near the
# origin, so with weights in +-0.1 the SwiGLU is close enough to
# symmetric that reading gate and up the wrong way round changes the
# logits by 2e-4 — inside any tolerance you would write. The scale here
# is chosen to put activations where silu is actually curved, which is
# what makes the ordering observable, while staying below the point
# where the MoE router saturates and top-k stops discriminating.
with torch.no_grad():
for name, p in model.named_parameters():
if "visual" in name or "vision" in name:
continue
p.copy_(torch.empty_like(p).uniform_(-0.8, 0.8))
ids = torch.randint(0, tc.vocab_size, (1, args.tokens), dtype=torch.long)
# eos is not an ordinary token here: `_shift_right_ignore_eos` treats it
# as a segment boundary, so an n-gram never reads across one. Placing it
# is therefore a deliberate choice rather than something to leave to the
# sampler — a fixture that happens to contain one is testing segment
# handling by accident, and one that happens not to is not testing it at
# all.
ids[ids == tc.eos_token_id] = (tc.eos_token_id + 1) % tc.vocab_size
if args.eos_at is not None:
ids[0, args.eos_at] = tc.eos_token_id
# The QSA selection must not rest on a tie.
#
# A block scores exactly 0.0 whenever relu clamps every head's dot
# product, which is common enough that a random fixture hits it. When
# the k-th and (k+1)-th scores are equal, which block gets dropped is
# decided by `topk`'s ordering of equal elements — and torch does not
# specify that, nor does its CPU kernel agree with its CUDA one. A
# fixture containing such a tie measures that unspecified detail rather
# than our arithmetic, so we refuse to emit one.
orig_topk = torch.Tensor.topk
ties = []
def _topk(self, k, *a, **kw):
if self.dim() == 1 and self.numel() <= 64 and k < self.numel():
ordered = sorted(self.detach().tolist(), reverse=True)
if ordered[k - 1] == ordered[k]:
ties.append((self.detach().tolist(), k))
return orig_topk(self, k, *a, **kw)
torch.Tensor.topk = _topk
with torch.no_grad():
out = model(input_ids=ids, use_cache=False)
torch.Tensor.topk = orig_topk
if ties:
raise SystemExit(
f"seed {args.seed} produces {len(ties)} QSA selection tie(s), e.g. "
f"scores={[round(v, 6) for v in ties[0][0]]} k={ties[0][1]}. "
"Pick another --seed: the fixture must isolate the arithmetic."
)
os.makedirs(args.out, exist_ok=True)
sd = {k: v for k, v in model.state_dict().items() if "visual" not in k and "vision" not in k}
# Split the runtime n-gram embedding back into the checkpoint's shard
# layout: dim 0, ascending, per upstream's own note that "the
# checkpoints for it are sharded on dim0" and loading concatenates them.
parts = tc.split_ngram_parts
for key in [k for k in sd if k.endswith("ngram_embedding.weight")]:
table = sd.pop(key)
rows = table.shape[0]
assert rows % parts == 0, f"{rows} rows do not divide into {parts} shards"
per = rows // parts
base = key[: -len(".weight")]
for s in range(parts):
sd[f"{base}.shard_{s}.weight"] = table[s * per : (s + 1) * per].clone()
save_file({k: v.contiguous() for k, v in sd.items()}, os.path.join(args.out, "model.safetensors"))
save_file(
{"input_ids": ids.to(torch.int64), "logits": out.logits.to(torch.float32).contiguous()},
os.path.join(args.out, "expected.safetensors"),
)
with open(os.path.join(args.out, "config.json"), "w") as f:
json.dump(raw, f, indent=2)
print(f"tokens {args.tokens}")
print(f"tensors {len(sd)}")
print(f"logits {tuple(out.logits.shape)}")
print(f"logits absmax {out.logits.abs().max().item():.6f}")
print(f"finite {bool(torch.isfinite(out.logits).all())}")