Compare commits

..
Author SHA1 Message Date
grenade 3403fbed37 feat(qwen4_exp): grouped RMSNorm and a configurable output gate
CI / Config consistency (push) Successful in 6s
CI / Classify changes (push) Successful in 15s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m32s
CI / Test (push) Successful in 9m30s
CI / CUDA type-check (push) Successful in 18m54s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
First piece of the qwen4_exp port. Both additions extend the Qwen3.6
norm primitives rather than forking them, because the conventions —
(1+w) scaling on the plain norm, raw weight on the gated one — are
already exactly right for the new arch.

`group_size` splits the last axis into independently normalised slices.
qwen4_exp needs it for every 10240-wide norm: hc_norm on both
hyper-connections of all 48 layers, PLE's three norms, and the MTP
head's hidden pre-norm, all with group_size = hidden_size. Four
independent normalisations, not one over 10240 — and a single norm over
the full axis compiles, runs, and is quietly wrong, which is why it
arrives with hand-computed parity tests rather than a shape assertion.

`OutputGate` makes the gated norm's nonlinearity explicit. Qwen3.6
leaves output_gate_type unset and falls back to hidden_act (SiLU);
qwen4_exp sets it to sigmoid. Inheriting SiLU there would be wrong on
all 36 linear-attention layers, in the fluent-but-worse way that is
expensive to detect later.

Both default to the Qwen3.6 behaviour, so the decoder, the attention
q/k norms and the TP analogues are untouched.

Refs #307, #312.
2026-08-31 14:46:56 +03:00
grenade 8fbd935ea1 Merge branch 'docs/308-qwen4-exp-port-spec'
build-prerelease / Resolve version stamps + change detection (push) Successful in 16s
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-08-31 09:41:22 +03:00
grenade 3d6aa57eb4 doc(qwen4_exp): port spec read from the reference and the checkpoint
Child A of the qwen4_exp epic. A written specification of
Qwen3.8-Flash-Next precise enough to implement the decoder from,
so nobody has to re-read 131 KB of upstream Python.

Everything dimensional is measured rather than estimated: the
parameter split comes from all 131 safetensors headers (routed
experts 67.11%, PLE n-gram tables 28.44%, always-resident text path
2.74%), which confirms the shape of the epic's arithmetic and
corrects its residual bucket.

Settles the questions the study set out to answer. ple_layer_ids is
one-indexed, so PLE sits on zero-indexed layer 1. The n-gram table
has 16 heads of 160 dims, not 8, and its three derived buffers ship
in the checkpoint, so the prime search and splitmix64 multipliers
need not be reimplemented. QSA's budget is in tokens (512 blocks of
4), the full KV is still materialised, and there is a *second*
indexer cache the epic's KV arithmetic omitted. Hyper-connections
replace both layernorms and widen the inter-layer residual 4x, and
there is no final model norm at all.

Also draws the reuse boundary against arch/qwen3_5: rope carries
over unchanged, rmsnorm and linear_attn need small deltas (grouped
norms, a sigmoid output gate, f32 SSM state), moe needs a loader
branch for fused 3D experts, and only decoder.rs is a rewrite.

Refs #308, #307.
2026-08-31 09:41:11 +03:00
grenade e067d596ae Merge branch 'fix/alias-test-temp-catalogue-collision'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m35s
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build cortex binary (push) Successful in 1m54s
build-prerelease / Build neuron-ada (push) Successful in 2m39s
build-prerelease / Build neuron-ampere (push) Successful in 2m45s
build-prerelease / Test (push) Successful in 6m36s
build-prerelease / Build neuron-blackwell (push) Successful in 14m43s
build-prerelease / Package cortex RPM (push) Successful in 22s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m3s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m6s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 3m54s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 26s
2026-08-30 22:35:43 +03:00
grenadeandClaude Opus 5 33a62d6aa5 fix(test): give each alias fixture its own temp directory
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 15s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m35s
CI / Test (push) Successful in 8m30s
CI / CUDA type-check (push) Successful in 18m56s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
`aliases.rs` wrote every fixture as a distinct filename in the shared
system temp dir. Its three tests run concurrently in one binary and
write *different* alias tables, so a collision means a test loads
another's catalogue.

The failure is silent, which is what makes it expensive.
`ModelCatalogue::load` maps missing, unreadable, unparseable and empty
alike onto an empty catalogue — correct for production, where an absent
catalogue is legitimate — and `resolve_alias` then returns the alias
unchanged. Nothing errors. The mismatch surfaces at whatever assertion
first touches the alias, tens of lines from the cause, as

    assertion `left == right` failed: alias should resolve to target id
      left: "helexa/small"
     right: "test-model"

which is precisely what CI reported on `main` at d4f74910 — a tree
byte-identical to the branch tip that had just passed all four
validation jobs, and one that passes 60/60 locally under parallelism.

So: a directory per call plus an atomic counter, which removes the
collision rather than making it less likely. And a read-back before
returning, because a test may not assume its fixture reached the disk
when the loader cannot tell an empty catalogue from one that never
loaded.

The doc-comment also claimed an "$XDG_RUNTIME_DIR fallback ... keeps the
file off shared /tmp on CI". The code never did that — it called
`std::env::temp_dir()`. Removed; the comment now describes what the
function does and why.

Not a fix for the underlying non-determinism if the trigger turns out to
be something other than collision — the exact CI trigger was not
reproduced locally. It removes one real mechanism and converts the rest
into a loud failure at the point of cause.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVtMbW8YwVmuRmJ38mHbTE
2026-08-30 22:15:11 +03:00
grenade d4f7491008 Merge branch 'chore/retire-vestigial-reasoning-ladder'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m39s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m56s
build-prerelease / Build neuron-ampere (push) Successful in 2m58s
build-prerelease / Test (push) Failing after 6m12s
build-prerelease / Build neuron-blackwell (push) Successful in 14m58s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m4s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m7s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m7s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-08-30 20:20:03 +03:00
grenade cc5b99f91a Merge branch 'fix/context-limit-honours-kv-budget'
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Package helexa-neuron-ada RPM (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 14s
build-prerelease / Build neuron-blackwell (push) Successful in 16m40s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m34s
build-prerelease / Build neuron-ampere (push) Successful in 2m37s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m35s
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-08-30 20:00:01 +03:00
grenadeandClaude Opus 5 687753767f chore(reasoning): retire the effort→token ladder that nothing read
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 17s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m30s
CI / Test (push) Successful in 9m3s
CI / CUDA type-check (push) Successful in 22m8s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
`ReasoningBudgetLadder` carried a per-effort token table — `minimal`,
`low`, `medium`, `high`, `xhigh`, `max` — plus `for_effort()` to resolve
against it and `advertised()` to publish it. #290 moved effort onto the
model's own chat template, where it is a trained control, and rewired
the budget to `declared_max_tokens - answer_reserve_tokens`. Neither
method has had a production caller since; the only call sites were their
own unit tests.

Dead code that describes a mechanism is worse than dead code that
doesn't. Reading this struct, it is natural to conclude that asking for
`xhigh` buys a different token budget than `medium`, or that the rungs
named here are what a client sees on `/v1/models`. Both are false:
effort changes the rendered system prompt and nothing else, and the
advertised rungs come from `reasoning_effort::probe` rendering the
template. That misreading has now cost two people an afternoon.

So the table goes, and the struct is renamed to `ReasoningBudgetConfig`
for what it now is: one field, `answer_reserve_tokens`, which is live
and load-bearing. The TOML key `[harness.candle.reasoning_budget]` is
unchanged and the struct does not deny unknown fields, so an operator
config carrying a stale `medium = 12288` keeps working and simply
ignores it.

The module and function docs now say where effort *does* act, and point
at `apply_effort_kwarg` rather than leaving the reader to infer a
mapping that is not there. `effort_no_longer_selects_a_rungs_token_budget`
survives with 12,288 as a literal — if a per-effort budget is ever
reintroduced, that is the number it lands on, and the test should fail.

No behaviour change: 47/47 reasoning tests pass, workspace clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVtMbW8YwVmuRmJ38mHbTE
2026-08-30 19:56:49 +03:00
grenadeandClaude Opus 5 2e48533b4d fix(context-limit): advertise only what admission will actually grant
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 14s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m30s
CI / Test (push) Successful in 9m3s
CI / CUDA type-check (push) Successful in 19m13s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
`derive_limit` computed the context ceiling from the model's position
limit, live free VRAM and the measured prefill rate, and never consulted
the KV budget that `AdmissionController` hands out. Two independent
answers to "how long a prompt fits" — the drift `check_vram`'s own
doc-comment warns about — and they disagreed.

Measured on a 27B at TP=2, by probing until `prompt_too_long_for_vram`
and solving for the constant: 32,768 B of KV per token per card, so a
3689 MiB budget holds 118,048 tokens. The node advertised the model's
full 131,072. A client that believed the advertisement filled a session
toward it and got a hard 413 with no compaction path — the failure is
unrecoverable mid-session because the number it planned against was
cached at startup.

`context` now takes `min(..., kv_ceiling)`. A zero budget is the
"not published yet" sentinel (pre-load, CPU) and imposes no ceiling,
matching how `free_tightest_mb == 0` already behaves, so models still
advertise normally during startup.

Note this bounds the whole sequence: a caller intending to request
`output_ceiling` must plan against `context - its own declared output`.
`input` stays `context - output_reserve`, honest for the default.

Also corrects `long_rung_available`'s doc-comment. A controlled A/B/C
reproduced its target defect at `max_in_flight = 2` with one request in
flight, while `medium` at 8 was clean, so concurrency does not predict
it; `NEURON_GDN_STATE_F32=1` did not change it either. The constant
stays as a precaution but is no longer allowed to read as a diagnosis.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MVtMbW8YwVmuRmJ38mHbTE
2026-08-30 19:39:08 +03:00
grenade b142bea782 Merge branch 'feat/angels-internal-vhost'
build-prerelease / Resolve version stamps + change detection (push) Successful in 22s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m5s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Successful in 1m38s
build-prerelease / Build cortex binary (push) Successful in 2m24s
build-prerelease / Build neuron-ampere (push) Successful in 2m49s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m48s
build-prerelease / Build neuron-ada (push) Successful in 3m7s
build-prerelease / Test (push) Successful in 9m29s
build-prerelease / Build neuron-blackwell (push) Successful in 15m1s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Successful in 1m29s
build-prerelease / Package helexa-bench RPM (push) Successful in 22s
build-prerelease / Package helexa-upstream RPM (push) Successful in 24s
build-prerelease / Package helexa-router RPM (push) Successful in 19s
build-prerelease / Package cortex RPM (push) Successful in 27s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m5s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m8s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m13s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 31s
2026-08-29 16:06:11 +03:00
grenade 943570895d Merge branch 'fix/replay-only-the-latest-reasoning'
Add/add conflict in `openai_responses.rs`: both sides appended a
function to the same region, and both function bodies ended at the
shared closing brace. Resolved by keeping both — they are independent
and compose in the right order.

`merge_tool_call_into` (#296) runs during message assembly, folding a
bare function_call into the assistant turn it belongs to.
`prune_stale_reasoning` runs afterwards on the assembled list, keeping
reasoning only on the newest assistant turn. So a merged turn is a
single turn before pruning decides whether its thinking is context,
which is the order both changes want.

All 47 openai_responses tests from both branches pass together.
2026-08-29 16:05:58 +03:00
grenade f48035f2ce Merge branch 'fix/docs-directory-index' 2026-08-29 16:05:46 +03:00
grenade eecb039875 Merge branch 'fix/upstream-test-fixture-collisions' 2026-08-29 16:05:46 +03:00
grenade 5502cb2893 Merge branch 'feat/server-advertised-reasoning-ladder'
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 17s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m35s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Successful in 1m56s
build-prerelease / Build helexa-router binary (push) Successful in 1m52s
build-prerelease / Build cortex binary (push) Successful in 2m28s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m48s
build-prerelease / Build helexa-tools binary (push) Has been cancelled
build-prerelease / Build neuron-blackwell (push) Has been cancelled
build-prerelease / Build neuron-ampere (push) Has been cancelled
build-prerelease / Build neuron-ada (push) Has been cancelled
build-prerelease / Package cortex RPM (push) Has been cancelled
build-prerelease / Package helexa-bench RPM (push) Has been cancelled
build-prerelease / Package helexa-upstream RPM (push) Has been cancelled
build-prerelease / Package helexa-router RPM (push) Has been cancelled
build-prerelease / Package helexa-tools RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-08-29 15:58:44 +03:00
grenade 2f358264c6 feat(reasoning): advertise which rungs a node can currently serve
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 18s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m54s
CI / Format (push) Successful in 15s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 10m48s
CI / CUDA type-check (push) Successful in 19m30s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Replaces the on/off checkbox with a picker driven by what the server
publishes, and teaches the server to publish it.

Operator report: `xhigh` "hits issues" at `max_in_flight = 8`. The
mechanism is not isolated, so this is written down as a precaution rather
than a diagnosis — calling it a fix would stop anyone looking. The
plausible account is in `long_rung_available`: xhigh instructs the model
to deliberate exhaustively, KV is a shared pool (#291), and a slot holds
its reservation for the request's whole life, so concurrent xhigh turns
can hold most of the pool for minutes. Consistent with "issues"; not
evidence of them.

**Advertised, never enforced.** The request path does not reject a
withheld rung — a caller that insists still gets it. This keeps a
capacity hint from becoming a new refusal path for every existing API
client, which would be a much larger decision than a dropdown.

Three defects surfaced on the way:

- **The router discarded the ladder.** `reasoning_budget` was cleared in
  the block that clears fields *derived from `limit`*, which it is not.
  cortex published it and the public endpoint returned null, so no
  browser client could have read it. Restored, with a federated merge: a
  rung is offered when **any** operator can serve it, since the router
  can route there — marking it unavailable because one busy operator
  withholds it would hide capacity the federation has.
- **The ladder was held per node, not per model**, on the reasoning that
  "every model on a neuron reports the same rungs". True while a rung was
  only a name from a template; false once availability depends on the
  model's own `max_in_flight`. Two models on one host can differ, and the
  node-level copy would attribute one's withheld rung to the other. Now
  per model, with the node copy kept only as a fallback for a neuron too
  old to report one.
- **The SPA never fetched `/v1/models` at all.** It does now, and the
  rungs are the model's own (#290) rather than a hardcoded
  `["low","medium","xhigh"]` that rots the first time the catalogue
  changes.

A remembered rung that has since been withdrawn resolves to off, and
deliberately not to "the next best rung" — that chooses on the user's
behalf silently, and the model's own default is `xhigh`, the rung most
likely to be withheld.

That resolution is derived at render rather than written back by an
effect. The first version used an effect and `npm test`'s neighbour
`react-hooks/set-state-in-effect` rejected it — the same rule the web CI
comment records having once let a defect reach main.

13 new tests: the availability rule and longest-rung selection in neuron,
the federated merge in the router, and the ladder fetch in the SPA
including every quiet-degradation path (non-2xx, network failure,
unexpected body) — the control is an enhancement, and a chat that works
without it beats an error banner about a capability nobody asked for.

Rung labels translated into all 42 locales, with `defaultValue` falling
back to the raw wire name so a model shipping an unfamiliar rung renders
it rather than a missing-key placeholder.
2026-08-29 14:41:43 +03:00
grenade f5c9e17ab6 Merge branch 'feat/web-chat-collapsible-working'
build-prerelease / Resolve version stamps + change detection (push) Successful in 18s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m32s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ada (push) Successful in 2m56s
build-prerelease / Build neuron-ampere (push) Successful in 2m58s
build-prerelease / Test (push) Successful in 10m12s
build-prerelease / Build neuron-blackwell (push) Successful in 15m1s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m11s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m20s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m51s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 26s
2026-08-29 14:25:18 +03:00
grenade 37afba40ab fix(toolcall): recover a call that tagged its function as a parameter
CI / Config consistency (push) Successful in 9s
CI / Classify changes (push) Successful in 21s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m10s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m29s
CI / Test (push) Successful in 9m32s
CI / CUDA type-check (push) Successful in 18m51s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Reported from helexa.ai on 2026-08-29. Asked "what's the best hardware to
run qwen 3.8 flash next on?", the user's answer was a screenful of raw
markup:

    <tool_call> <parameter=web_search> <parameter=query>
    "Qwen3.8 Flash Next" hardware GPU </parameter> </function> </tool_call>

The model tagged the function itself as a parameter — there is no
`<function=` anywhere in that block. `parse_qwen_xml_tool_call` opens
with `body.split("<function=").nth(1)?`, so it returned None, and the
caller did what it does with an unparseable block: re-emitted it as
text. That text was the whole reply.

The journal shows the rest of the shape. Rounds one to four each finished
`ToolCalls` and ran real searches — 17 sources were collected — and the
fifth, tool-less round produced 35 tokens which were this malformed call.
So the turn spent five model passes and never produced a sentence.

Now a leading `<parameter=X>` is promoted to the function name when X is
a tool the request actually offered, and that pseudo-parameter is
excluded from the arguments.

The known-tool guard is the whole safety of it: without it any
`<parameter=x>` mints a function call, and a model rambling in angle
brackets starts invoking tools. `does_not_invent_a_function_from_an_unknown_parameter`
pins it, and was checked by deleting the guard — it fails, as it should.

Tested against the verbatim string from the report, plus a well-formed
call to prove the recovery does not disturb the normal path.

This makes the call parse, which is what lets the UI render it as a
search block rather than markup. It does not fix the deeper problem that
an unparseable block still reaches the user as prose — filed separately.
2026-08-29 13:47:42 +03:00
grenade 8d32f4e9aa feat(chat): a reasoning toggle, because the server only thinks when asked
The reasoning UI in 940124b0 would have displayed nothing. neuron
couples reasoning *generation* to reasoning *surfacing* (api.rs:359): a
caller that does not send `x-include-thinking: true` gets
`enable_thinking: false`, and the model produces no reasoning at all.
The web chat sent no such header, which is why every request in the
reported session logged `reasoning_tokens=0` — including an earlier
1,866-token answer.

That coupling is right and stays: an undisplayed think block still eats
the output budget, which is the failure it was added to prevent. The
client just has to opt in.

A checkbox rather than always-on, per the operator: the chat interface is
a showcase for integrators, and both paths are worth demonstrating — the
reasoning path, and the clean-output path every vanilla OpenAI client
gets. It switches real server behaviour, not a rendering preference.

Verified through the live public router before building anything on it:
the same request with the header returns `reasoning_tokens=30` where it
returned 0 without. `x-include-thinking` survives helexa-stream's
forwarder (only `host` and `content-length` are dropped) and the router's
`CorsLayer::permissive()` admits it through preflight — a custom header
from a browser needs that, and helexa-acp never exercised it.

Preference persists in localStorage, wrapped in try/catch on both sides:
private windows and blocked site data throw on access, and losing a
convenience must not take the page down.

Translated into all 42 locales.

Refs #304.
2026-08-29 13:45:23 +03:00
grenade 22ce46cb47 test(web): add vitest and gate on it, and fix the bug that found
CI / CUDA type-check (push) Blocked by required conditions
CI / Config consistency (push) Successful in 10s
CI / Classify changes (push) Successful in 21s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m22s
CI / Format (push) Successful in 15s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 10m46s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
helexa.ai had no test runner: lint, typecheck, i18n parity and build all
gated, but nothing executed a line of application logic. Closes the first
half of #304.

Gating from the first commit was the operator's call, and the reasoning
is better than warn-first: with no existing suite there is nothing to
stabilise, so a warning period only creates a window in which a vacuous
test lands and gets normalised. A suite nobody trusts is worse than none,
because it gets cited.

**It immediately earned its keep.** Extracting `useChat`'s tool-result
matching into a testable function exposed that it was backwards. Results
arrive in call order, and the inline version scanned from the newest end
— so with two `web_search` calls in one turn, the first query's results
were filed under the second query's heading. Nothing would have reported
it: no error, no empty block, just a confident answer under the wrong
question. Now FIFO, with `pairs repeated calls to one tool in order`
pinning it.

That is the argument for this issue in one defect. The bug was written,
reviewed and shipped by me hours earlier; it became visible only when the
rule had to be stated somewhere it could be run.

25 tests across three modules:

- `turnBlockState` — the eight visibility behaviours, ported from the
  throwaway harness and now importing the real module rather than
  re-declaring the rule, which is what the harness's first version did
  and which would pass against any implementation.
- `turnBlocks` — reasoning coalescing, think → act → think ordering, and
  the result-matching cases above.
- `chatClient` — SSE framing against a real `ReadableStream`: a frame cut
  mid-JSON across two reads, several frames in one read, `[DONE]`
  terminating, a malformed frame not killing the stream, comments and
  `event:` lines ignored, and reasoning never leaking into content.

Each of the three core behaviours was mutation-checked — the buggy
newest-first scan, blocks staying open past the first answer token, and
reasoning routed to `onDelta` — and every mutation is caught. A test that
cannot fail is not a test.

Also gated in deploy.yml's website build, alongside i18n:check and for
the same reason: pushes to main skip ci.yml, so that is the only check a
merge itself passes through.

No jsdom yet and no coverage threshold. The pure modules are most of the
value; a threshold picked against an empty codebase is arbitrary and
rewards tests written to satisfy it.

Refs #304.
2026-08-29 13:27:38 +03:00
grenade 940124b06f feat(chat): show the assistant's working, collapsed until it matters
CI / Config consistency (push) Successful in 9s
CI / Classify changes (push) Successful in 19s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m6s
CI / CUDA type-check (push) Has been skipped
CI / Format (push) Has been skipped
CI / Clippy (push) Has been skipped
CI / Test (push) Has been skipped
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The web chat threw reasoning away. neuron emits it as
`choice.delta.reasoning_content` — deliberately not as `content`, so
naïve clients still render clean output — and this client had no handler,
so on a hard prompt the UI sat silent for however long the model thought.
Tool calls were equally invisible: the status line said "searching…" and
nothing recorded what was searched or what came back.

Both are now captured as an ordered `blocks` list on the message, and
rendered above the answer.

Ordered rather than "the reasoning" plus "the tool calls" because the
order is the story: a turn can reason, call a tool, reason about what
came back, and call another. Two flat fields lose which thought preceded
which call.

Visibility follows activity:

  * the live block — still streaming, no answer text yet — is open,
    because that span is otherwise dead air, and this says what the model
    is working on rather than merely that it is working;
  * the moment answer text starts, everything collapses to one-line
    headers, so the thing that was asked for takes the space;
  * completed turns load collapsed;
  * an explicit click outranks all of it, in both directions. Closing a
    noisy reasoning stream must not spring back open on the next token.

That rule lives in `lib/turnBlockState.ts`, free of React imports, so it
can be read and exercised on its own — as an inline expression the only
way to check it was to click around. Verified against the built module
across all eight behaviours above, including the two override cases.

Collapsed rows carry a tail-anchored peek rather than a head-anchored
one: while a block streams, the end is where the model currently is, and
a head preview freezes on the first few words. Bodies are capped and
scrollable, since a long think block would otherwise push the answer
off-screen and defeat the point.

Storage needed no Dexie version bump — `blocks` is not indexed, and Dexie
stores whole objects. Older messages simply have none, which renders as
an answer with no working shown: correct, since none was captured.

All three strings are translated into all 42 locales; `i18n:check`
enforces parity and would have failed on English-only, which I had
assumed `fallbackLng` covered. `blockTool` is pure interpolation and is
identical everywhere.
2026-08-29 13:14:59 +03:00
grenade 5208b18784 Merge branch 'fix/router-sync-restarts-only-on-settings-change'
build-prerelease / Resolve version stamps + change detection (push) Successful in 19s
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-08-29 10:34:48 +03:00
grenade 4b3bf433bb fix(deploy): restart the router for a settings change, not a comment
CI / Classify changes (push) Successful in 21s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Config consistency (push) Successful in 9s
CI / Format (push) Successful in 14s
CI / Clippy (push) Successful in 2m30s
CI / Test (push) Successful in 11m16s
CI / CUDA type-check (push) Successful in 27m27s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The sync step's first real run restarted helexa-router, and the only
difference was the MANAGED BY CI header I had just added to the tracked
copy. Parsed settings were identical. So as written, editing a comment
drops whatever federation requests are in flight.

Byte equality and settings equality answer different questions. The bytes
decide whether to SYNC; the parsed settings decide whether to RESTART.
The router reads its aliases at startup, so a comment is not a reason to
interrupt anyone.

`script/toml-settings-equal.py` makes the second comparison, as a script
rather than an inline heredoc — a heredoc inside a YAML block scalar
needs its terminator at column 0, which breaks out of the block, and the
first attempt at this did exactly that. Verified on all three cases: the
comment-only diff that caused the restart returns "settings identical";
Wednesday's alias diff returns "settings differ in: aliases"; a missing
or unparseable file returns changed, so being wrong costs one restart
rather than leaving a service on settings nobody deployed.

Needs a scoped `cat` grant to read the deployed copy back. It is
deliberately excluded from the step's fail-hard preflight: without it the
read yields nothing, the comparison says "changed", and the step restarts
on any byte difference — the previous behaviour. A host part-way through
a sudoers rollout degrades rather than fails.
2026-08-29 10:05:28 +03:00
grenade 98df7de7e5 Merge branch 'feat/ci-owns-the-router-config'
build-prerelease / Resolve version stamps + change detection (push) Successful in 22s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m45s
build-prerelease / Build neuron-blackwell (push) Successful in 17m35s
build-prerelease / Build helexa-bench binary (push) Successful in 1m53s
build-prerelease / Build cortex binary (push) Successful in 2m23s
build-prerelease / Build neuron-ampere (push) Successful in 2m41s
build-prerelease / Build helexa-angels binary (push) Successful in 2m48s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m51s
build-prerelease / Test (push) Successful in 9m31s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package cortex RPM (push) Successful in 26s
build-prerelease / Package helexa-upstream RPM (push) Successful in 27s
build-prerelease / Package helexa-bench RPM (push) Successful in 31s
build-prerelease / Package helexa-angels RPM (push) Successful in 28s
build-prerelease / Build helexa-router binary (push) Successful in 1m34s
build-prerelease / Package helexa-router RPM (push) Successful in 20s
build-prerelease / Build neuron-ada (push) Successful in 2m34s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m10s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m10s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 3m56s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 29s
2026-08-29 09:20:31 +03:00
grenade b89ab1fafb feat(config): an operator-owned secrets layer, so CI can own the rest
CI / Classify changes (push) Successful in 18s
CI / Config consistency (push) Successful in 8s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m27s
CI / Format (push) Successful in 12s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 15m51s
CI / CUDA type-check (push) Successful in 21m37s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The remaining #287 Gap 1 files could not be version-controlled because
they mix structure with credentials, and the proposed fix — render them
in CI from Gitea secrets — makes the pipeline a credential store. It also
is not needed: the secrets already exist on the hosts and work. What CI
needs is to own the structure without touching them.

cortex, helexa-upstream and helexa-angels now merge a `secrets.toml`
sibling after the main config and before env overrides. The main file
holds structure and can be deployed by CI; `secrets.toml` holds
credentials, is written only by the operator, and is never read, written
or diffed by the pipeline.

Merging last is the safety property, not an implementation detail: the
operator's value wins, so a deploy can never replace a live credential
with a placeholder. Pinned by `the_operators_value_wins_over_the_deployed_one`.

An absent secrets.toml is not an error — figment treats a missing
`Toml::file` as an empty source — so every existing single-file
deployment keeps working untouched and adoption is per host, per service.
That is the property most worth having: if it regressed, the change would
break the fleet on rollout rather than on adoption, so
`an_absent_secrets_file_is_not_an_error` asserts it directly.

A second TOML layer rather than env vars because
`[[entitlements.keys]]` is an array of tables, which `CORTEX_*`
`__`-split env overrides cannot express. The layer then carries every
secret uniformly instead of splitting the mechanism by field shape.

Docs cover the migration as four steps that are safe to stop between,
since a value present in both files resolves to the operator's.

This is the general form of the lesson from 2026-08-27: config CI cannot
write is config no check can defend. `helexa-router.toml` was excluded
from git for sitting beside secret-bearing files, and its stale alias
took beast down. The fix is to stop mixing the two, not to hand the
pipeline the credentials.

Refs #287.
2026-08-29 08:55:06 +03:00
grenade 2956e3843c feat(deploy): CI owns the router config, and aliases must name real models
CI / Test (push) Blocked by required conditions
CI / CUDA type-check (push) Blocked by required conditions
CI / Classify changes (push) Successful in 17s
CI / Config consistency (push) Successful in 9s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m31s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
`helexa-router.toml` was gitignored beside the secret-bearing service
configs, and it holds no secrets: a listen address, poll interval, region,
the product-tier aliases, and a mesh endpoint. It was excluded by
association.

That exclusion cost an outage. On 2026-08-27 the router still aliased
`helexa/balanced` to Qwen/Qwen3.6-27B after that model had been retired
from the catalogue. Both 27Bs sat at equal `residency_priority` — the
right design, so an A/B could displace either way — so every
authenticated web-chat turn evicted beast's resident flagship and
cold-loaded the retired one, ~70 s of TP layer loading each way. Four
overlapped and left the node holding 32 GB on one card with no model
registered and nothing serving, while `activation.state` said "ready".

No bad code was involved. Two files described one routing decision and
nothing could diff them, because one of them was not in git.

So:

- `helexa-router.toml` is tracked and carries the MANAGED BY CI banner.
- `deploy-gallumbits` syncs it, digest-gated, restarting the router only
  on change (it reads aliases at startup; a needless restart drops
  in-flight federation requests). The step sits after the package work
  and preflights its own sudoers grants, per the two ways that went
  wrong this week.
- `script/infra-setup.sh` no longer writes it. Two writers for one file
  is how a fleet ends up running something no commit describes.
- `asset/sudoers.d/gallumbits-host.conf` gains three scoped grants. That
  host had no rsync grant at all, which is why #287 lists this as
  blocked; the grants must be installed by a hand-run infra-setup before
  the step can pass, and it says so when they are not.
- `script/check-alias-targets.py` fails CI when any alias names a model
  the catalogue does not define, in both ci.yml and deploy.yml. Verified
  against Wednesday's exact config: it fails, and names Qwen/Qwen3.8-27B
  as the intended target.

The check is deliberately narrow — it asks whether a target exists, never
which model a tier ought to point at. That is a product decision, and a
check that guessed would be wrong more often than the operator.

The genuinely secret-bearing configs (cortex.toml, helexa-upstream.toml,
helexa-angels.toml) are untouched and still hand-synced. They need
rendering from Gitea secrets, which is the rest of #287 Gap 1.

Refs #287.
2026-08-29 08:47:26 +03:00
grenade c43f4bc74f Merge branch 'fix/infra-setup-ships-ratelimit-to-gateway'
build-prerelease / Resolve version stamps + change detection (push) Successful in 19s
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-08-29 08:42:03 +03:00
grenade cd0211a864 Merge branch 'fix/nginx-preflight-checks-every-grant' 2026-08-29 08:42:03 +03:00
grenade 65011311f2 fix(infra-setup): ship helexa-ratelimit.conf to the gateway, before the vhost
CI / Classify changes (push) Successful in 15s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Config consistency (push) Successful in 10s
CI / Format (push) Successful in 9s
CI / CUDA type-check (push) Successful in 22m16s
CI / Clippy (push) Successful in 2m38s
CI / Test (push) Successful in 7m34s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
helexa-ratelimit.conf was installed on the web and angels hosts but never
on the gateway, where helexa.internal.conf lives and references the names
it defines — the limit_req_zones and, since the search-logging change,
the `helexa_noquery` log format. The gateway's copy has no such format:
`grep -c helexa_noquery` returns 0 there against 4 uses in the vhost.

The ordering in this block turns that into a live hazard rather than a
caught mistake. The sites-enabled symlink is created BEFORE `nginx -t`
runs, so a config that fails validation is already wired in. The running
nginx survives (it is never reloaded), but the next reload or reboot —
by anything, including cert renewal via step@bench.timer, enabled three
lines later — fails on it.

Found while about to run this script to install the two sudoers grants a
deploy step needs. It would have pushed a vhost the host could not parse.

deploy.yml's deploy-nginx job already ships the pair in this order for
exactly this reason; this brings the hand-run path in line with it.
2026-08-27 23:31:09 +03:00
grenade 9b0b32a43b Merge branch 'fix/restore-max-in-flight-8'
build-prerelease / Resolve version stamps + change detection (push) Successful in 19s
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build cortex binary (push) Successful in 2m16s
build-prerelease / Package cortex RPM (push) Successful in 25s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m52s
build-prerelease / Test (push) Successful in 9m42s
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Successful in 1m28s
build-prerelease / Package helexa-router RPM (push) Successful in 18s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 14s
2026-08-27 23:27:54 +03:00
grenade bc7f4a301d fix(deploy): the nginx preflight must check every grant, not a representative one
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 11s
CI / CUDA type-check (push) Successful in 22m19s
CI / Clippy (push) Successful in 2m42s
CI / Test (push) Successful in 7m59s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
It tested `sudo -n nginx -t` and nothing else. That grant had existed for
some time, so the check passed on a host whose sudoers predated the two
helexa-ratelimit.conf entries added in the very commit that introduced
the check — and the job then failed on an opaque rsync denial, seconds
after reporting the permissions were fine.

That is worse than having no preflight. A check that names a remedy while
probing the wrong permission actively misleads: the operator reads "the
sudoers are OK" and looks elsewhere.

Verified against the live host, where the split is real:

    GRANTED rsync sites-available:  OK      MISSING rsync ratelimit:      FAIL
    GRANTED ln -sfn:                OK      MISSING sha256sum ratelimit:  FAIL
    GRANTED systemctl reload:       OK

Now probes all seven. sha256sum, `nginx -t` and `systemctl` are safe to
invoke directly; rsync and ln have no harmless form that still matches
their sudoers pattern, so those use `sudo -nl`, which answers whether a
command is permitted without running it.

The failure message also now says that having run script/infra-setup.sh
before is not enough — grants accumulate in that file as deploy steps
grow, and a host keeps whatever revision it last saw. That was the actual
situation here.
2026-08-27 23:27:40 +03:00
grenade 318dc5f4e1 Merge branch 'fix/retire-qwen36-27b'
build-prerelease / Lint (fmt + clippy) (push) Blocked by required conditions
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Build cortex binary (push) Blocked by required conditions
build-prerelease / Build helexa-router binary (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 20s
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been cancelled
build-prerelease / Build neuron-ada (push) Has been cancelled
build-prerelease / Package cortex RPM (push) Has been cancelled
build-prerelease / Package helexa-router RPM (push) Has been cancelled
build-prerelease / Package helexa-tools RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-08-27 23:25:55 +03:00
grenade 21eefe7e45 fix(beast): restore max_in_flight = 8 — it never divided the KV budget
CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 16s
CI / Format (push) Has been skipped
CI / Clippy (push) Has been skipped
CI / Test (push) Has been skipped
CI / CUDA type-check (push) Has been skipped
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m47s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
I lowered this to 2 earlier today (#291) on the stated grounds that
"max_in_flight is both the admission bound and the batch engine's slot
count, and the KV budget is divided by it at load". The second half is
false, and has been since eca42510 (2026-08-15) made admission reserve
KV per request:

    admission.enter_with_kv(principal,
        kv_reservation_mb(prompt_len, requested_output, profile, cfg))

`kv_budget` is a semaphore of MiB permits; a request takes what its own
prompt needs and waits for bytes when the pool is short — which is what
`kv_max_wait_secs` is for. Nothing in the tree divides that budget by
the slot count, and the engine's `max_slots` is only the batch width.
The docs said both things twelve lines apart: `kv_max_wait_secs` was
already documented as "how long it waits for enough KV budget".

The measurement in #291 stands as data — seven days of spans, 89.6% of
busy time at one request in flight — but two things undercut the
conclusion. It cannot show that eight slots starve anything, because
they do not. And it described traffic that no longer exists: while it
was taken the router's `helexa/balanced` alias pointed web chat at a
second 27B (3b0b3a22), so per-model concurrency was split across two
models busy evicting each other. Both public tiers now resolve here, so
chat concurrency lands on this model.

Also corrects the operating docs, which carried the same claim, and the
prefix-cache trade table, whose "tokens per slot" column had silently
divided the pool by two. It is a shared pool, so the honest column is
what all concurrent requests draw from: 118,000 tokens at budget_mb =
2048, not 59,000 each.

Refs #291, #257.
2026-08-27 23:24:53 +03:00
grenade 3b0b3a2227 fix(catalogue): retire Qwen3.6-27B — a stale alias was evicting the flagship
CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 15s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m7s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 10m39s
CI / CUDA type-check (push) Successful in 19m28s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
beast served nothing for ~50 minutes on 2026-08-27 with 32 GB held on one
card, the other empty, `models: []`, and `activation.state: "ready"`.

The driver was the router's alias table. The web chat's authenticated
default is `helexa/balanced` (Chat.tsx:35), and on gallumbits that alias
still resolved to Qwen/Qwen3.6-27B while beast's resident model is the
3.8. The two sat at equal `residency_priority` deliberately, so they
displaced each other in both directions — meaning every authenticated
web-chat turn evicted the flagship and cold-loaded the 3.6 (~70 s of TP
layer loading), and the next flagship request swapped it straight back.
At 21:27–21:33 four of those overlapped and left the leader holding
accumulated shards with nothing registered.

Mutual displacement is not the bug; #99 wants it and the equal ranks were
right for the A/B they were set up for (#251). The bug is that the A/B
was over and the loser stayed catalogued, so a stale alias pointed real
user traffic at a model whose only remaining job was to evict the one
people were actually being served.

Retires the 3.6 from models.toml and repoints every operator-facing
mention at the 3.8: the docs under helexa.ai/content/docs/2-operating/,
the example catalogues and router/acp configs, and the helexa-acp README
(which told readers to set a default_model that would now 404).

Deliberately NOT renamed: doc/benchmarks.md and
crates/neuron/tests/fixtures/numerical/ record measurements and a parity
snapshot actually taken against the 3.6 — rewriting those would falsify
the record. Test fixtures elsewhere keep the old string too; it is an
arbitrary model id in those, and the churn would bury the real diff.

The live alias on gallumbits was corrected and the router restarted
ahead of this commit, since it was the active cause; helexa-router.toml
is gitignored, so that file is still hand-synced (#287).
2026-08-27 22:34:40 +03:00
grenade 2d1a4ebb2f Merge branch 'fix/nginx-sync-must-not-block-cortex-deploy'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-08-27 13:24:03 +03:00
grenade d87327c3a2 fix(deploy): the nginx sync must be its own job, not a step that gates the fleet
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Config consistency (push) Successful in 6s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 6m43s
CI / CUDA type-check (push) Successful in 20m55s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Run 1284 proved the first attempt wrong, and worse than predicted. The
sync step hard-fails until an operator runs script/infra-setup.sh to
install its sudoers grant; a failed step fails its job; and
`deploy-neurons` has `needs: [deploy-cortex]`. So a missing nginx
permission skipped the rollout of all three neuron flavours. The fleet
sat two commits behind on eb783c3 while the run reported nothing worse
than "deploy-cortex: failure" — the neuron jobs were *skipped*, which
reads like "nothing to do".

Moving the step later within deploy-cortex, as the previous commit did,
does not fix this. The job still ends red and the dependents still skip.
Order was never the problem.

`deploy-nginx` now stands alone with no `needs`, so its failure is a
statement about the vhost and nothing else. It still fails hard —
drift between the repo copy and the live file is a real defect and
should be red — but it can no longer take the fleet with it.

#287 names this trap: adding a sync step before its sudoers grant exists
breaks the deploy. Having written that down and then done it anyway, the
narrower rule is now recorded beside the job: a step whose prerequisite
is installed by hand does not belong in a job that other jobs depend on.

Refs #287.
2026-08-27 13:02:09 +03:00
grenade a17b3acb70 fix(deploy): the nginx sync must not hold the cortex binary hostage
CI / CUDA type-check (push) Blocked by required conditions
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Config consistency (push) Successful in 16s
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m31s
CI / Test (push) Successful in 6m40s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
The step I added in c6c6fc8a sat before the cortex package update, and
it hard-fails when `gitea_ci` lacks the nginx sudoers grant — which is
installed by hand, by `script/infra-setup.sh`, and has not been run yet.
So until an operator runs it, an unrelated permission blocks the gateway
binary from deploying at all.

That is the exact trap #287 names: adding the sync step before its
sudoers grant exists breaks every deploy. Writing it down and then
walking into it the same day is worth the comment now beside the step.

Moved after `Deploy cortex`, with the reasoning recorded so it does not
drift back. The hard failure stays — a job that could not sync the vhost
should be red, because it did not do everything it promised — but it now
fails after the package work rather than instead of it.
2026-08-27 12:36:47 +03:00
grenade c6c6fc8a50 Merge branch 'feat/nginx-search-alias-and-sync'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m12s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 3m6s
build-prerelease / Build neuron-ada (push) Successful in 3m16s
build-prerelease / Test (push) Successful in 9m39s
build-prerelease / Build neuron-blackwell (push) Successful in 17m3s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m2s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m8s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m6s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 26s
2026-08-27 12:33:49 +03:00
grenade 192030dff5 Merge branch 'fix/wire-surface-in-neuron-spans'
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Build neuron-blackwell (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 14s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m1s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 2m47s
build-prerelease / Build neuron-ada (push) Has been cancelled
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-08-27 12:26:01 +03:00
grenade 24b8de7f82 Merge branch 'fix/295-ignore-tiny-prefill-samples' 2026-08-27 12:26:01 +03:00
grenade 432f8009ce feat(nginx): accept POST on search, and keep queries out of the logs
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 14s
CI / Format (push) Successful in 8s
CI / Clippy (push) Successful in 2m33s
CI / CUDA type-check (push) Successful in 20m21s
CI / Test (push) Successful in 7m4s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
**Queries were being logged.** The `main` log format records
`$request` — method, path *and query string* — so every search a user
typed was written to /var/log/nginx/access.log beside their IP, for
authenticated and anonymous visitors alike:

    10.6.0.193 - - [27/Aug/2026] "GET /tools/web_search?q=bansko+weather&format=json HTTP/2.0" 200

A `helexa_noquery` format logs `$request_method $uri` instead, so the
operational signal survives — who, when, which endpoint, what status,
how big, how long — and the query does not. Rate-limit debugging and
abuse detection need counts, statuses and addresses, never search terms.

Applied to `/tools/fetch` as well: its `url=` parameter is the page a
user is having read to them, which is as revealing as a search query and
was equally in the clear.

**POST is now accepted** on both search locations. It was GET-only
because the SPA only GETs — least privilege by default rather than a
considered restriction. SearXNG accepts both, some clients POST, and a
POST keeps the query out of any intermediary's request line, which is
the same concern the log format addresses. `client_max_body_size 8k`
bounds it: a search query is small, and without a bound this becomes a
way to push arbitrary bytes at the upstream.

The log format lives in helexa-ratelimit.conf because `log_format` is
http-context only, and that file is already the http-context include for
this feature. That creates a dependency the deploy has to respect: the
vhost now references a name defined there, so the sync ships
helexa-ratelimit.conf *first*. Shipping the vhost alone would fail
`nginx -t` on an unknown log format — caught by the gate, but leaving
the two files out of step.

Both files validated together by installing them into the live paths on
hanzalova, running `nginx -t`, and restoring with a verified diff.
2026-08-27 12:10:48 +03:00
grenade 98ba45cbb4 feat(nginx): serve SearXNG's own path, and deploy the vhost from CI
CI / Test (push) Blocked by required conditions
CI / CUDA type-check (push) Blocked by required conditions
CI / Config consistency (push) Successful in 10s
CI / Classify changes (push) Successful in 20s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 11s
CI / Clippy (push) Successful in 2m38s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Three coupled changes; the first is useless without the second.

**`location = /search`.** `/tools/web_search` is our name for the
SearXNG upstream and a generic SearXNG client cannot guess it — they
take a base URL and append `/search`, because that is what every
instance serves. Measured on the live edge:

    /tools/web_search?q=…&format=json  -> application/json, SearXNG output
    /search?q=…&format=json            -> text/html (the SPA catch-all)

So such a client gets HTTP 200 with an HTML body and fails at JSON
parsing — which sends people looking for a bug in their own parser. A
404 would at least be honest. The new block is an alias, not a laxer
door: same upstream, same `helexa_tools` rate-limit zone, same
`limit_except GET`.

**The deploy now syncs the vhost.** `asset/nginx/*.conf` has been
version-controlled from the start but only ever installed by
`script/infra-setup.sh`, a one-time run-by-hand script — so the repo
copy and the live file could diverge with nothing to detect it. That is
the #252 shape: two sources of truth for one thing and nothing forcing
agreement. Digest-compared, `nginx -t` gates the reload, and a config
nginx rejects leaves the previous one serving.

**sites-enabled becomes a symlink.** On this host it was a *regular
file*, not a link to sites-available — two independent copies, of which
nginx reads one (`conf.d/sites-enabled.conf` includes
`sites-enabled/*.conf` and nothing includes sites-available). An
operator editing "the config" would reasonably edit sites-available,
watch `nginx -t` pass, reload, and see nothing change. The sync
(re)points the link as its mechanism rather than as a separate chore, so
the layout cannot drift back.

Validated by installing the candidate into sites-enabled on the real
host, running `nginx -t`, and restoring — `nginx -t` does not reload, so
the running config was never at risk. Worth noting the first attempt
tested sites-available and proved nothing, because nginx does not
include it; that is the same trap the symlink change exists to close.

**Requires one manual step before the next deploy.** The nginx sudoers
entries are added to `asset/sudoers.d/cortex-host.conf`, which is
installed by `script/infra-setup.sh`. Until that is re-run once, the new
deploy step fails — deliberately, with a preflight that names the remedy
rather than surfacing an opaque rsync permission error.
2026-08-27 12:05:35 +03:00
grenade 522682282c fix(observability): record which API surface a request arrived on
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 9m12s
CI / CUDA type-check (push) Successful in 28m29s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
neuron translates `/v1/responses` into a chat request at the door
(`openai_responses::request_to_chat`) and every line after that point is
emitted by the shared chat path. So the journal showed

    tp_chat_stream{req_id=… model=…}: chat_completion (stream): done

for every request, whatever the caller used. That is not a missing
detail; it asserts something false. It cost a real investigation into
why a Responses-configured client "was using chat completions" — it was
not, and nothing in neuron's journal could have said so.

The `tp_chat_stream` span even carried a comment claiming the journal
"can be reconstructed regardless of which API surface the client hit",
while recording nothing that would allow it.

Both spans now carry `wire=openai-chat` or `wire=openai-responses`,
threaded from the four handler entry points alongside the `principal`
that already made the same trip. The strings are deliberately the ones
`cortex-gateway`'s `log_inbound` already emits, so `journalctl | grep
openai-responses` spans both services instead of stopping at the
gateway.

A `WireSurface` enum rather than a `&'static str` parameter: with four
call sites feeding two shared inference paths, a bare string is one
copy-paste away from a span that lies in a new way.

The value is on the span, so every line already being emitted gains it —
finish lines with `reasoning_tokens` / `cached_tokens` / `finish_reason`,
rendered-prompt traces, admission rejections. Those are where the useful
per-turn data lives, and none of it could be attributed to a surface
before.

Note the TP call site is `#[cfg(feature = "cuda")]` and therefore
invisible to a local build: the CPU build was green with that one
argument missing. Caught by grepping the call sites rather than trusting
`cargo build`, which is the second time today that gate has mattered.
2026-08-27 11:35:41 +03:00
grenade 176ce47349 fix(context): ignore prefill samples too small to measure throughput
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 16s
CI / Format (push) Successful in 9s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m22s
CI / Clippy (push) Successful in 2m37s
CI / Test (push) Successful in 6m50s
CI / CUDA type-check (push) Successful in 28m29s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
A restart collapsed the advertised context to a sixth of what the host
can serve, and stayed there until real traffic happened along. beast,
minutes after a deploy:

    prefill_tok_per_sec=145.16  throughput_ceiling=17419  -> context=17408
    limit: {"context": 17408, "input": 9216, ...}

against a `vram_ceiling` of 182,560 and a hard ceiling of 131,072.

The cause is not a missing guard — the guard exists and is displaced.
`derive_limit` falls back to `bootstrap_prefill_tok_per_sec` (800,
≈96k context) only when the EMA has *no* sample, and the deploy's own
smoke probe supplies one: 19 tokens in ~130 ms, which is almost entirely
fixed per-request cost and reads as 145 tok/s on a host that measures
~1,400. One probe replaced a conservative estimate with a wrong one.

`fold_rate` already intended to prevent this — "No-op for degenerate
inputs so a probe request or a clock blip can't poison the average" —
but only guarded zero tokens and zero elapsed, not the small-prompt case
that actually occurs.

Prefill samples below 512 tokens are now ignored. That excludes the
smoke probe and the short auxiliary calls agent harnesses make (observed
at 19, 143 and 469 tokens) while admitting ordinary prompts: at
~1,400 tok/s a 512-token prefill runs ~370 ms, so fixed overhead is
about a tenth of it rather than all of it.

Fixed here rather than by warming the host on deploy, which would have
made one caller's experience better once and left the next restart to
rediscover it.

Prefill only. `record_decode` has the same shape of flaw — a two-token
completion is equally overhead-dominated — but it feeds the #137 metrics
rather than the advertised limit, so it is left for a change that can be
attributed on its own.

Four tests: the 19-token probe leaves the EMA unset so the bootstrap
value stands, the 143/469-token auxiliary calls likewise, a
representative prefill still records and still tracks a shift, and the
threshold itself counts.

Docs updated: the operating page no longer tells operators to warm a
restarted host by hand.
2026-08-27 10:45:21 +03:00
grenade 00acd283b6 docs(operating): document every operator setting a neuron exposes
build-prerelease / Resolve version stamps + change detection (push) Successful in 16s
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
`config.rs` exposed 35 operator-settable keys and 17 appeared nowhere in
the operating docs — including every one this week's defects turned on.
An operator had no way to discover them, and #297 opened because two of
them hid real bugs: `prefix_cache.budget_mb` (the #292 cache plateau)
and `bootstrap_prefill_tok_per_sec` (which exists to prevent #295's
cold-start collapse and is defeated by the first tiny request).

Now 0 of 35 are undocumented.

Written as guidance rather than a key dump. Each lever says what it
controls, what the default is *and why*, and — the part that makes a
reference usable — **what symptom would make you reach for it**.
`budget_mb` is useless as "snapshot byte budget"; it is useful as "if
`cached_tokens` stops growing mid-session, this is why".

New sections in 03-neurons.md:

- **Sampling.** The full precedence chain (request > operator > model >
  built-in), a symptom-to-knob table, and why `presence_penalty` is the
  right lever for a repeating think block while widening `repeat_last_n`
  is the wrong one — the penalty is token-level and source code repeats
  tokens legitimately. Carries the measured finding that 86% of think
  blocks over 60k characters contained verbatim repeated sentences.
- **Reasoning.** `preserve_thinking` framed as the model's own template
  control and an unmeasured experiment rather than a recommendation, and
  `answer_reserve_tokens` with the measurement that motivated it (given
  16,384 tokens the model used 16,384 and returned no answer) plus the
  trade it introduces — a reserve that fires interrupts deliberation and
  the answer tends to state an intention rather than a conclusion.
- **The prefix cache.** Including the counter-intuitive direction: the
  snapshot budget is reserved *before* the KV budget, so raising it too
  far starves KV and shortens the sessions whose snapshots the larger
  cache was meant to hold. Table of the measured 1:1 trade.
- **Image models.** Why the text encoder defaults to CPU, what
  `te_resident` costs on each device, and why `max_dim` rejects before
  admission — a device OOM poisons the worker context, so refusing early
  is the difference between one rejected request and a restart.
- **Where weights come from.** `sources`, `default_source`, `auth_env`
  (an env var name, never a token) and per-source `cache_dir`, with the
  collision that shared caches cause.

05-context-limits.md gains the three settings that shape the derived
limit, and notes that `bootstrap_prefill_tok_per_sec` only applies when
the EMA has no sample at all — so a host that restarted recently should
be warmed before its `/v1/models` is trusted.

Two corrections while in here: the concurrency example advertised
`max_in_flight = 8`, which #291 showed divides the KV budget eight ways
and leaves a single long session unable to use the advertised context;
the example and its guidance now reflect the measured production
setting, with the request-span data behind it.
2026-08-27 10:35:34 +03:00
grenade eb783c345e Merge branch 'fix/300-anthropic-streaming-thinking'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Build helexa-bench binary (push) Successful in 2m8s
build-prerelease / Build neuron-ada (push) Successful in 2m27s
build-prerelease / Build cortex binary (push) Successful in 2m28s
build-prerelease / Build neuron-ampere (push) Successful in 2m33s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m32s
build-prerelease / Package helexa-bench RPM (push) Successful in 24s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m58s
build-prerelease / Test (push) Successful in 7m56s
build-prerelease / Build neuron-blackwell (push) Successful in 21m49s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Successful in 24s
build-prerelease / Package cortex RPM (push) Successful in 28s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m9s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m15s
build-prerelease / Build helexa-router binary (push) Successful in 1m29s
build-prerelease / Package helexa-router RPM (push) Successful in 17s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m13s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 28s
2026-08-27 10:04:24 +03:00
grenade 72d6a37986 fix(anthropic): carry thinking on the streaming path too
CI / Config consistency (push) Successful in 11s
CI / Classify changes (push) Successful in 14s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m37s
CI / Test (push) Successful in 13m46s
CI / CUDA type-check (push) Successful in 19m3s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The #300 fix was incomplete and incomplete in the way that mattered:
it fixed `openai_to_anthropic`, the *non-streaming* translation, while
`AnthropicStreamTranslator` — the path Claude Code actually uses —
still dropped reasoning entirely.

`on_chunk` read `delta.content` and `delta.tool_calls` and nothing
else. `anthropic_sse.rs` had zero occurrences of "thinking". So an
Anthropic client asked for extended thinking via
`thinking.budget_tokens` (which `anthropic_to_openai` deliberately
accepts and normalises onto `reasoning.max_tokens`), was billed for the
reasoning tokens, and received a stream containing none of it — every
turn, silently. Merging the non-streaming fix alone would have left
that untouched and looked like a fix.

`reasoning_content` deltas now open a `thinking` content block and emit
`thinking_delta` events, ahead of the text block and closing before it,
which is Anthropic's ordering and the order the model produces them in.

Emitted without a `signature`, for the same reason as the non-streaming
block: Anthropic signs theirs, we cannot mint a valid one, and
inventing a value would be worse than omitting it. If a client rejects
an unsigned block the fix is a flag, not a return to deleting the
model's work.

Also moves two tests from the previous commit inside the `#[cfg(test)]`
module they were meant to be in — they had landed after its closing
brace, at file top level. They ran and passed there, which is exactly
why it went unnoticed.

Test asserts the full shape: a thinking block opens, `thinking_delta`
carries the text, and a following visible-text delta closes it and
opens a separate text block.
2026-08-27 09:43:36 +03:00
grenade 039a353adf Merge branch 'fix/300-return-the-models-reasoning'
build-prerelease / Resolve version stamps + change detection (push) Successful in 16s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m32s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Successful in 1m54s
build-prerelease / Build helexa-bench binary (push) Successful in 2m7s
build-prerelease / Build neuron-ada (push) Successful in 2m21s
build-prerelease / Build neuron-ampere (push) Successful in 2m24s
build-prerelease / Build cortex binary (push) Successful in 2m22s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m45s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Successful in 20s
build-prerelease / Package helexa-bench RPM (push) Successful in 23s
build-prerelease / Package cortex RPM (push) Successful in 1m14s
build-prerelease / Package helexa-upstream RPM (push) Successful in 1m19s
build-prerelease / Test (push) Successful in 16m23s
build-prerelease / Build neuron-blackwell (push) Successful in 21m42s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m4s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m39s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 3m50s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 31s
2026-08-27 09:29:47 +03:00
grenade 4e4ffd9b60 fix(reasoning): return the model's work to the client on every surface
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 14s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m28s
CI / Test (push) Successful in 10m5s
CI / CUDA type-check (push) Successful in 27m51s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Three of five response paths generated reasoning, metered it in `usage`,
and then discarded it. A client that never receives reasoning cannot
replay it, so on the next turn the model faced its own prior turn as a
bare assertion with the thinking that produced it deleted — and paid
for the deletion.

  /v1/chat/completions  streaming      already returned it
  /v1/chat/completions  non-streaming  DISCARDED -> now returned
  /v1/responses         streaming      already returned it
  /v1/responses         non-streaming  DISCARDED -> now returned
  /v1/messages          (Anthropic)    DISCARDED -> now returned

**chat, non-streaming.** `collect_chat_completion` had
`InferenceEvent::ReasoningDelta(_) => {}` — inherited from the path it
replaced, with a doc comment rationalising it. Now accumulated onto
`message.reasoning_content`, the field the streaming path already emits
as `choice.delta.reasoning_content`. Still kept out of `content`: it is
not the assistant's reply.

**responses, non-streaming.** `build_response` emitted a Message item
and nothing else, while the streaming projector beside it built a full
reasoning-item lifecycle. Observed live: a response reporting
`output_tokens_details.reasoning_tokens: 4000` with an empty `output` —
4,000 tokens generated, billed, invisible. Now emits a `reasoning` item
ahead of the message, same shape and ordering as streaming, via a new
`ResponsesOutputItem::Reasoning`.

**Anthropic.** `anthropic_to_openai` deliberately accepts
`thinking.budget_tokens` — the only quantitative reasoning control any
of our wire formats offers (#223) — and `openai_to_anthropic` returned
a bare text block. Taking a parameter and silently discarding what it
buys is the defect. Now emits a `thinking` block ahead of the text.

The Anthropic block carries no `signature`. Anthropic signs theirs and a
client round-tripping one may expect the field; we cannot mint a valid
signature and inventing one would be worse than omitting it. If a real
client rejects an unsigned block the fix is a flag, not a return to
deleting the model's work — recorded in the code so the next person
does not have to rediscover the trade.

Six tests, one pair per surface: reasoning survives, and absent
reasoning leaves the previous shape untouched (absent, not empty). The
lack of exactly these tests is why three paths drifted apart while a
fourth was correct.

Also carried: two byte-exactness regression tests written while chasing
a suspected content-corruption bug — a file read back through a tool
result, and one through the whole Responses->chat->prompt chain, both
asserting `<`, `&`, backticks, `${}`, quotes and blank lines survive
verbatim. The corruption turned out to be elsewhere; the tests are worth
keeping since that path had none.
2026-08-27 08:46:37 +03:00
grenade 7ad9f48cad Merge branch 'feat/operator-repetition-knobs'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m44s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Successful in 1m41s
build-prerelease / Build helexa-bench binary (push) Successful in 1m56s
build-prerelease / Build cortex binary (push) Successful in 2m13s
build-prerelease / Build neuron-ada (push) Successful in 2m30s
build-prerelease / Build neuron-ampere (push) Successful in 2m30s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m30s
build-prerelease / Test (push) Successful in 8m2s
build-prerelease / Build neuron-blackwell (push) Successful in 21m50s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Successful in 20s
build-prerelease / Package helexa-bench RPM (push) Successful in 21s
build-prerelease / Package helexa-upstream RPM (push) Successful in 22s
build-prerelease / Package cortex RPM (push) Successful in 28s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m2s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m9s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m6s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 25s
2026-08-26 21:09:39 +03:00
grenade 6141befa16 feat(sampling): let an operator pin the seed
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m28s
CI / Test (push) Successful in 8m22s
CI / CUDA type-check (push) Successful in 28m17s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Parameter search needs a control. Comparing two `presence_penalty`
values across unseeded runs cannot separate the parameter's effect from
sampling variance, and the client that would otherwise pin the seed
cannot: pi's Responses request builder sets `model`, `effort`, `tools`,
`signal` and `timeout` — no `seed`. A caller-side seed is unavailable on
the path that matters, so the operator side is the only place it can go.

Verified reproducible on `Qwen/Qwen3.8-27B` before adding this, four
runs of one prompt at `seed: 424242`, byte-identical every time:

  - B=1, cold prefill (cached_tokens=0)
  - B=1, prefix-cache hit (cached_tokens=13)
  - B=2, decoding alongside a concurrent 835-token stream

Two of those were worth checking rather than assuming: a restored
prefix-cache snapshot yields the same token choices as a fresh prefill,
and batch composition does not perturb the result even though `(B,1)`
matmuls could plausibly change reduction order. Untested at length —
29 tokens is a short window and divergence compounds, so a 28k think
block is a much harder claim.

Precedence unchanged: request > operator > per-request random. An unset
override still gets a fresh seed per request, pinned by a test, so this
cannot accidentally freeze the fleet's RNG.

Worth an operator's deliberate choice rather than a default: with this
set, every identical prompt yields an identical answer for every caller.
Defensible — predictable, cacheable — but it removes the variation a
user may expect from retrying, which is a product decision and not just
a testing convenience.
2026-08-26 20:39:57 +03:00
grenade e55998d224 feat(sampling): presence and frequency penalties, OpenAI semantics
CI / CUDA type-check (push) Blocked by required conditions
CI / Classify changes (push) Successful in 14s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Config consistency (push) Successful in 22s
CI / Format (push) Successful in 8s
CI / Clippy (push) Successful in 2m56s
CI / Test (push) Successful in 7m8s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Qwen's own remedy for the endless repetition their models fall into
during long reasoning is `presence_penalty` (0..2, 1.5 when severe) —
and we did not implement it. Zero occurrences in the tree before this.

The distinction that matters: `repetition_penalty` is scored over a
trailing window (`repeat_last_n`, default 64), while presence and
frequency are scored over the **whole** generated sequence. Inside a
28,672-token think block a passage restated 5,000 tokens later is
invisible to a 64-token window, which is how the same derivation
appears five times in one block. A wider window is not the fix: the
penalty is token-level and source code legitimately repeats tokens
constantly, so widening it trades a visible pathology for an invisible
one. Sequence-wide penalties are the instrument the model's authors
name.

    logit -= frequency * count + presence * (count > 0)

Additive on logits, not multiplicative like `repeat_penalty`, because
that is what the OpenAI parameters mean — a client sending
`presence_penalty: 1.5` and expecting OpenAI behaviour gets it.

Also closes a wire gap: both are OpenAI *core* parameters. Clients could
send them and we discarded them silently, which is the #293 class of
defect and what #294's schema conformance would catch.

Threaded through the whole chain — `/v1/chat/completions`,
`/v1/responses`, `SamplingOverride` (so an operator can set them per
model in `models.toml` and `asset/neuron/<host>.toml`, gated by
`check-config-consistency.py`), and the resolution order request >
operator > model > built-in.

Defaults are `0.0`, which is exactly a no-op — the tensor is returned
untouched rather than round-tripped through the CPU — so a deployment
that sets nothing behaves as before. Pinned by a test.

Three tests: the OpenAI semantics (presence once regardless of count,
frequency scaling with it, composing additively), a token 5,000
positions outside the window still being penalised, and zero being a
true no-op.

No default change. What value helps without degrading code is a
measurement (#299), not a guess.
2026-08-26 20:24:25 +03:00
grenade c7df005d37 feat(sampling): let an operator reach the repetition knobs
CI / Test (push) Blocked by required conditions
CI / CUDA type-check (push) Blocked by required conditions
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 15s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m29s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
`repeat_last_n` defaults to 64 — candle's chat-example value — and a
reasoning model's think block runs to tens of thousands of tokens.
Anything the model restates more than 64 tokens later is invisible to
the penalty, which is how a 28,672-token think block ends up containing
the same derivation five times over. Observed live.

The constant's own doc comment already said this ("poorly suited to a
24k-token reasoning block ... which is why it is now a per-request knob
rather than a constant"). But per-request was not reachable in practice:
`SamplingOverride` (#283) carried only temperature/top_p/top_k, and
OpenAI-shaped clients do not send repetition parameters at all. The 64
was effectively hardcoded in production, by a route nobody had cause to
look at.

So `repeat_penalty` and `repeat_last_n` join the same precedence chain
as the rest — request > operator > model > built-in — settable in
`models.toml` and `asset/neuron/<host>.toml`, and gated by
`check-config-consistency.py` like every other sampling field. Defaults
are untouched: an operator who sets nothing gets exactly the previous
behaviour, pinned by a test.

Deliberately no default change. A wide window is not a free win —
the penalty is token-level, and source code legitimately repeats tokens
constantly (`const`, `function`, brace runs), so a wide window at a
meaningful penalty distorts the output this fleet exists to produce.
That trade almost certainly explains the original 64. Sequence-level
repetition wants a sequence-level instrument (frequency/presence
penalties, or a DRY-style sampler); this makes the token-level trade
measurable per model instead of guessed once, and tells us whether the
sequence-level work is warranted.
2026-08-26 20:14:01 +03:00
grenade 9c1dfc1f59 fix(beast): 2 GiB of snapshots, not 3 — the budget is taken from KV
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
The snapshot budget is reserved *before* the KV budget, not from the
remainder. Measured on this host after deploying 3072:

    kv_budget_mb: 4713 -> 2665      (exactly 5737 - budget_mb)

Per slot at max_in_flight=2 that is 42,600 tokens, and the session this
change was made for reached 52,869. So a 3 GiB cache made the model
unable to reach the session size whose snapshot the cache had been
enlarged to hold — self-defeating, and it would have surfaced mid-run as
KV exhaustion rather than as a clean limit.

My sizing was wrong in its premise, not its arithmetic: I treated the
KV budget as fixed and the cache as fitting in the leftover headroom.
The causality is the other way round.

2 GiB satisfies both floors — kv_budget 3,689 MB (~59,000 tokens/slot,
comfortably past the deepest observed session) and enough for a
52,869-token snapshot at ~1,652 MB of K/V plus GatedDeltaNet state.

The trade table is now in the config next to the value, since the
1:1 relationship is the thing that makes this number non-obvious.
2026-08-26 19:26:12 +03:00
grenade 35e87b88ff Merge branch 'fix/292-prefix-cache-budget'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m6s
build-prerelease / Test (push) Successful in 11m1s
build-prerelease / Build neuron-blackwell (push) Successful in 14m35s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Successful in 1m36s
build-prerelease / Build helexa-router binary (push) Successful in 1m41s
build-prerelease / Build neuron-ada (push) Successful in 2m8s
build-prerelease / Build cortex binary (push) Successful in 2m22s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m39s
build-prerelease / Build neuron-ampere (push) Successful in 3m4s
build-prerelease / Package helexa-router RPM (push) Successful in 20s
build-prerelease / Package helexa-bench RPM (push) Successful in 23s
build-prerelease / Package cortex RPM (push) Successful in 23s
build-prerelease / Package helexa-upstream RPM (push) Successful in 21s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m4s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m8s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m8s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 26s
2026-08-26 18:52:49 +03:00
grenade 3edd9c8010 Merge branch 'feat/stamp-preserve-thinking-state' 2026-08-26 18:52:49 +03:00
grenade 4102cbe202 fix(beast): 3 GiB of prefix-cache snapshots, not 1
CI / Classify changes (push) Successful in 14s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Config consistency (push) Successful in 8s
CI / CUDA type-check (push) Has been skipped
CI / Format (push) Has been skipped
CI / Clippy (push) Has been skipped
CI / Test (push) Has been skipped
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
`cached_tokens` was pinned at 2,016 for ten consecutive turns of a live
agentic session while the prompt grew to 52,869 — every turn
re-prefilling ~50k tokens at ~37 s, against ~20 s of actual decode. The
cache was not missing the prefix; the snapshot could not be stored.

At the measured `kv_bytes_per_token_per_card = 32768`, the 1 GiB default
cannot hold more than ~32k tokens of K/V, and less once `snapshot.rs`
deep-copies the GatedDeltaNet conv/recurrent state on top — 48 of 64
layers, in f32 per `mamba_ssm_dtype`. Sessions on this host run well
past that, so the deeper the session the more of every turn was spent
re-doing work.

Sized from this deployment rather than rounded: free at load 9,285 MB,
less the 4,713 MB KV budget, leaves 4,572 MB that snapshots share. A
52,869-token snapshot is ~1,652 MB of K/V, so 3 GiB holds a full deep
session with slack. 4 GiB would take 90% of the headroom, which is not
a trade worth making on a host that also cold-swaps image models.

One variable: `max_entries` stays at 8. The byte budget binds first
here, so the count cap is not what forces snapshots to be small, and
changing both would make the result unattributable.

Config-only, so this deploys without a binary rotation (#291) — the
classifier treats `asset/neuron/*.toml` as inert.

Expected: `cached_tokens` tracks the prompt instead of freezing, and
per-turn prefill falls from ~37 s toward the low seconds. If it stays
frozen at 2,016, the budget is not the cap and #292 moves to the
retention policy.
2026-08-26 18:51:08 +03:00
grenade 236f08e730 feat(reasoning): advertise and log the preserve_thinking a run ran under
CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 8s
CI / Clippy (push) Successful in 2m56s
CI / Test (push) Successful in 8m15s
CI / CUDA type-check (push) Successful in 20m44s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
`preserve_thinking` is an A/B knob, and a comparison whose arms cannot
be told apart afterwards is not a comparison. Recovering the value from
deploy history is guesswork — a config sync can land mid-session, and
the two tower-defence arms will be reviewed hours later from a session
file and a journal.

Two places, because the two ways a run gets reviewed are different:

- `ModelInfo.preserve_thinking` on neuron's `/models` (and through to
  cortex's `/v1/models`), so a harness can stamp the value into its own
  record at the start of a run, and so it can be checked at any moment
  without grepping logs.
- An `INFO` line at each load site naming the effective value, so a
  journal that already carries the run's turns also carries the arm
  they belong to. `None` is logged explicitly rather than omitted —
  "unset, template default" is a distinct arm from `true`, and a
  missing line is indistinguishable from an older build.

Absent from the wire when unset (`skip_serializing_if`), so a client
that does not read it is unaffected, and absent-vs-false stays the
load-bearing distinction it is everywhere else in this struct.

No behaviour change: this only reports what is already in effect.
2026-08-26 16:41:37 +03:00
grenade f1012e8308 Merge branch 'fix/296-one-step-one-assistant-turn'
build-prerelease / Resolve version stamps + change detection (push) Successful in 15s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m34s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Successful in 1m44s
build-prerelease / Build helexa-router binary (push) Successful in 1m53s
build-prerelease / Build cortex binary (push) Successful in 2m25s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m43s
build-prerelease / Build neuron-ada (push) Successful in 2m56s
build-prerelease / Build neuron-ampere (push) Successful in 3m4s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Test (push) Successful in 9m33s
build-prerelease / Build neuron-blackwell (push) Successful in 16m55s
build-prerelease / Package helexa-router RPM (push) Successful in 20s
build-prerelease / Package helexa-upstream RPM (push) Successful in 22s
build-prerelease / Package helexa-bench RPM (push) Successful in 29s
build-prerelease / Package cortex RPM (push) Successful in 29s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m5s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m9s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m15s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 28s
2026-08-26 16:35:39 +03:00
grenade 82a7b8420c fix(responses): one agentic step is one assistant turn
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 17s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 11s
CI / Clippy (push) Successful in 2m30s
CI / Test (push) Successful in 10m9s
CI / CUDA type-check (push) Successful in 28m25s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The model was copying a transcript we built for it.

Responses splits one agentic step across several input items —
`reasoning`, then `message`, then `function_call`. `request_to_chat`
emitted a chat message per item, so a step became two *consecutive*
assistant turns. Rendered through Qwen3.8-27B's template that is:

    <|im_start|>assistant
    <think> …28,672 tokens… </think><|im_end|>      ← deliberates, then stops
    <|im_start|>assistant
    <think>
    </think>
    <tool_call>…                                    ← acts with no thought

Neither shape occurs in training, and consecutive assistant turns are
not a valid transcript at all.

Captured from a real session's rendered prompt (#296): 9 assistant
turns for 4 agentic steps, 4 empty `<think></think>` blocks, and **4
assistant turns ending with no tool call**. On the fifth step the model
produced a think block and the text "Now let me verify the JavaScript
parses cleanly — I want to catch any typos before calling it done:",
then emitted EOS without the call. It had been shown that exact pattern
four times in its own context.

`function_call` items now fold into the assistant turn they belong to,
which is the shape chat has always wanted: one message carrying
`content` and `tool_calls` together. Consecutive calls accumulate on
the same turn — that is what parallel tool calling looks like on the
wire.

The fold is deliberately narrow. Only a bare tool-call carrier merges:
empty content, no reasoning of its own, and no extension key other than
`tool_calls`. Anything else keeps its own turn, so a genuine two-turn
sequence is never collapsed and no field is silently dropped — trading
one invisible loss for another is how this defect survived.

Reasoning attachment (#277) already rode the next assistant message and
is unchanged; the call was the orphan.

Two tests pin it: one step yields `[user, assistant, tool, user]` with
reasoning, text and the call all on the one turn, and parallel calls
accumulate rather than splitting.

Not a fix for the model's judgement — a fix for what we showed it.
2026-08-26 15:51:54 +03:00
grenade c37d9de527 feat(reasoning): gate reasoning replay on the model's own template switch
CI / Config consistency (push) Successful in 12s
CI / Classify changes (push) Successful in 15s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m32s
CI / CUDA type-check (push) Successful in 20m18s
CI / Test (push) Successful in 9m8s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Makes H7's premise testable without trusting it.

H7 (`fix/replay-only-the-latest-reasoning`, unmerged) would replay only
the newest turn's reasoning. The premise is plausible — full replay is
what makes prompt growth track total completion rather than visible
output, pushing against the prefix-cache budget (#292) and the
throughput-derived context ceiling — but it was never measured, and
"withhold information from the model" is not a change to make on a
hunch.

It also turns out not to need new mechanism. `Qwen/Qwen3.8-27B`'s
template already has the switch:

    {%- if preserve_thinking is undefined or preserve_thinking is true
           or loop.index0 > ns.last_query_index %}
        ... '<think>\n' + reasoning_content + '\n</think>\n\n' + content
    {%- else %}
        ... content

`preserve_thinking = false` keeps reasoning only for turns after the
last user query — the turn in progress — which is exactly what H7 does,
expressed by the people who trained the model. And the default is
`true`: full replay is the authors' choice, not our accident.

So this exposes `preserve_thinking` as operator config rather than
changing any default. Precedence is request > operator > template
default; unset changes nothing. Flipping it is a config-only deploy
(~6 min, no binary rotation) now that the CI classifier treats
`asset/neuron/*.toml` as inert, which makes an A/B cheap enough to run
properly instead of once.

Carried on `ModelSpec` and `ModelProfile` both, so a cold-loaded model
behaves like a resident one — two config files describing one model is
how #252 happened. `check-config-consistency.py` now gates on it: a
model that replays when resident and not when cold-loaded would make
the measurement unreadable.

`main` still ships Qwen's default. Nothing changes until an operator
sets it, which is the point.
2026-08-26 14:19:31 +03:00
grenade d3341cb45c Merge branch 'fix/client-capability-misattributions'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Successful in 1m49s
build-prerelease / Package helexa-bench RPM (push) Successful in 23s
build-prerelease / Build cortex binary (push) Successful in 2m33s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m48s
build-prerelease / Build neuron-ampere (push) Successful in 3m0s
build-prerelease / Build neuron-ada (push) Successful in 3m0s
build-prerelease / Build helexa-upstream binary (push) Successful in 3m2s
build-prerelease / Test (push) Successful in 7m56s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Successful in 1m20s
build-prerelease / Build neuron-blackwell (push) Successful in 15m46s
build-prerelease / Package helexa-router RPM (push) Successful in 20s
build-prerelease / Package cortex RPM (push) Successful in 26s
build-prerelease / Package helexa-upstream RPM (push) Successful in 25s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m5s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m8s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 3m56s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 32s
2026-08-26 14:10:56 +03:00
grenadeandClaude Opus 5 8dd0bb5332 fix(responses): replay only the newest turn's reasoning
CI / Classify changes (push) Successful in 18s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 12s
CI / Clippy (push) Successful in 2m29s
CI / Test (push) Successful in 9m5s
CI / CUDA type-check (push) Successful in 28m8s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
#277 restored the reasoning round-trip this morning and over-corrected.
An agentic harness replays every turn verbatim, so the model's entire
thinking history was being rendered back into the prompt — Qwen3.8's
template defaults `preserve_thinking` to true, and we now supply the
content it preserves.

Measured on a live session, ten assistant turns in:

    thinking:  95,311 chars   (62% of replayed assistant content)
    text:         467 chars   (total, across all ten turns)
    single blocks: 38,871 … 34,593 … 12,572 chars

The model's per-turn reasoning had collapsed to 34–661 tokens and it was
making increasingly shallow passes at a task two other models finished
from one prompt. Reasoning models are trained on histories carrying
*final content*, not the deliberation behind it; Qwen's own multi-turn
convention strips prior think blocks. We were feeding it a context that
was mostly its own self-quotation.

The newest block is different: when a turn is cut off mid-thought and
the caller says "continue", that thinking is the work being resumed,
which is the case #277 exists to serve. So the newest assistant turn
keeps its reasoning and everything older loses it — the turns
themselves, and their content, are untouched.

Worth recording why the template's own switch cannot express this:
`preserve_thinking: false` keeps thinking on assistant turns *after* the
last user message (the agentic loop — where it is noise) and drops it on
turns *before* (the continuation — where it is the point). Exactly
backwards for both, so the policy has to live here.

Two tests: an agentic history keeps only the latest thinking while every
turn survives, and a truncated-then-continued turn keeps its own.

This is a correlation with a mechanism, not a proven cause — the shallow
passes could be the model. It is testable now: the next session runs
against a prompt that is no longer 62% prior thinking.

Refs #277, #223.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-25 08:04:55 +03:00
grenade b8098c4096 test(upstream): unique fixtures so web_pg can run twice
CI / Classify changes (push) Successful in 16s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m55s
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 5m4s
CI / Test (push) Successful in 8m17s
CI / CUDA type-check (push) Successful in 29m13s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Two tests inserted hard-coded secrets under unique indexes:

    verify_endpoint_consumes_token_once      "verify-raw-token-xyz"
      → email_tokens.token_hash (primary key)
    fingerprint_abuse_silently_deactivates…  "sk-helexa-deactivated-probe"
      → api_keys.key_hash (unique on active keys)

They pass against a virgin database and fail with `23505 duplicate key`
on every run afterwards. That looked like flakiness or parallel
contention — it is neither. CI always provisions a fresh database, so
the defect was invisible there and only appears when a developer runs
the suite twice against the same one, which is exactly when a fast local
loop matters most.

`unique_secret()` mirrors the existing `unique_email()`. Verified by
running the whole workspace three times against one database: all pass,
where runs 2 and 3 previously failed.

Neither test was removed. Single-use email-verification tokens and
silent deactivation on multi-account abuse are both real properties
worth holding; the fixtures were the problem, not the coverage.
2026-08-17 15:52:05 +03:00
grenadeandClaude Opus 5 7a75703d23 fix(web): /docs returned 403 — a prerendered directory shadows its route
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m0s
CI / CUDA type-check (push) Has been skipped
CI / Format (push) Has been skipped
CI / Clippy (push) Has been skipped
CI / Test (push) Has been skipped
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Clicking the documentation link answered 403. Prerendering writes real
directories under the webroot, and a real directory shadows the
client-side route of the same name: nginx's `try_files $uri $uri/
/index.html` matches `$uri/`, looks for an index inside it, finds none,
and refuses. It never reaches the SPA shell, so the route that used to
work stopped working the moment its pages became files.

Only the leaves had indexes, so /docs/using/api served fine while /docs,
/docs/using and /docs/operating all 403'd. That is the worst arrangement
to notice: every link in the sidebar works, and only the addresses a
person reaches by trimming a URL — or by clicking the nav link — fail.

Every directory the prerenderer creates now gets an index, derived from
the page slugs rather than from the section list so a page nested deeper
than one level covers its intermediate directories too. They are worth
having in their own right: a crawler or a reader without JavaScript gets
a table of contents instead of a redirect.

A build-time check now fails the build if any directory under docs/ lacks
an index. Introducing this class of 403 costs a deploy and a bug report
to notice otherwise, and the condition is trivially checkable at the
point the files are written. Verified by adding a nested page and
confirming the check fires before it was generalised, then that the
generalised version covers it.

The SPA also treated a section path as a missing page, because no page
has that slug. It now redirects to the section's first page — those URLs
are reachable, both from the prerendered index and from someone trimming
a path back to its parent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0165r11RzqkMqWWXfJE8tAVU
2026-08-15 09:45:02 +03:00
grenadeandClaude Fable 5 0705768f24 feat(angels): angels.internal for mesh clients
CI / Format (push) Successful in 12s
CI / Clippy (push) Successful in 2m24s
CI / Test (push) Successful in 10m17s
CI / CUDA type-check (push) Successful in 19m3s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
An internal-only vhost for the portal, for networks where the public name
does not work correctly from inside — the usual case without NAT
reflection or split-horizon DNS. Optional: skip it if your public name
resolves and connects from everywhere.

Served by the edge nearest helexa-angels, with an internal-CA cert
renewed by step@angels.timer, and the same confidentiality headers and
per-IP limits as the public vhost. Needs a matching record on the local
resolver.

Verified: correct internal certificate, /health 200, / 303, invite link
redirects to sign-in. Public access unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0165r11RzqkMqWWXfJE8tAVU
2026-08-04 08:51:05 +03:00
128 changed files with 6674 additions and 408 deletions
+12
View File
@@ -177,6 +177,12 @@ jobs:
steps:
- uses: actions/checkout@v4
- run: python3 script/check-config-consistency.py
# Two files describe one routing decision and nothing forced them
# to agree. On 2026-08-27 the router still aliased
# `helexa/balanced` to a model retired from the catalogue, so
# every web-chat turn evicted beast's resident flagship to
# cold-load it. Same shape as the check above (#252/#287).
- run: python3 script/check-alias-targets.py
web:
name: Web (lint + typecheck + i18n + build)
@@ -199,6 +205,12 @@ jobs:
# reached main exactly that way.
- run: npm run lint
- run: npm run typecheck
# Behaviour, not just types. Gating from the first commit was the
# deliberate call (#304): with no existing suite there is nothing to
# stabilise, so a warn-first period would only create a window in
# which a vacuous test lands and gets normalised — and a suite
# nobody trusts is worse than none, because it gets cited.
- run: npm test
# 42 locales with strict key parity is a promise that breaks
# quietly. Adding an English string without translating it must fail
# here, not in review, and certainly not in front of somebody
+261
View File
@@ -59,6 +59,12 @@ jobs:
steps:
- uses: actions/checkout@v4
- run: python3 script/check-config-consistency.py
# Two files describe one routing decision and nothing forced them
# to agree. On 2026-08-27 the router still aliased
# `helexa/balanced` to a model retired from the catalogue, so
# every web-chat turn evicted beast's resident flagship to
# cold-load it. Same shape as the check above (#252/#287).
- run: python3 script/check-alias-targets.py
deploy-cortex:
needs: [config-check]
@@ -526,6 +532,161 @@ jobs:
# dependency on cortex or the neurons (the sweep loop is version-aware
# and picks up whatever each neuron reports whenever). So it runs
# alongside the cortex→neurons chain rather than after it.
# The vhost sync gets its own job, with NO `needs`, because it is not
# part of deploying the gateway binary and must not be able to stop
# one.
#
# It was briefly a step inside deploy-cortex, and that broke the fleet
# on its first run (2026-08-27, run 1284): the step hard-fails until an
# operator has run script/infra-setup.sh to install its sudoers grant,
# a failed step fails the job, and `deploy-neurons` has
# `needs: [deploy-cortex]` — so a missing nginx permission silently
# skipped the rollout of all three neuron flavours. The fleet stayed
# two commits behind while the run reported only "deploy-cortex:
# failure".
#
# #287 names this ordering trap. The lesson is narrower than "order the
# steps": a step whose prerequisite is installed by hand does not
# belong in a job that something else depends on, because its failure
# is not a statement about that job. Moving it later within
# deploy-cortex would not have helped — the job still ends red and the
# dependents still skip.
#
# Failing hard here is still right: drift between the repo and the live
# file is a real defect, and this job going red says that and nothing
# else.
deploy-nginx:
runs-on: fedora-43
if: >-
${{
github.event_name == 'workflow_dispatch'
|| github.event.workflow_run.conclusion == 'success'
}}
steps:
- uses: actions/checkout@v4
- name: SSH init
run: |
mkdir -p ~/.ssh
echo "${DEPLOY_KEY}" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new \
gitea_ci@hanzalova.internal 'hostname -f'
# nginx vhost config for the internal site. Version-controlled in
# `asset/nginx/` since the beginning, but until now only ever
# installed by `script/infra-setup.sh` — a one-time, run-by-hand
# script. So the repo copy and the live file could diverge with
# nothing to detect it, which is the #252 shape: two sources of
# truth for one thing and nothing forcing agreement.
#
# Writes to sites-available and (re)points sites-enabled at it as
# a symlink. On this host sites-enabled was a *regular file*, not a
# symlink — two independent copies, of which nginx reads one. An
# operator editing "the config" would reasonably edit
# sites-available, watch `nginx -t` pass, reload, and see nothing
# change. Making the link canonical is part of the sync rather
# than a separate chore, so the layout cannot drift back.
#
# `nginx -t` gates the reload: a config that fails validation is
# left un-reloaded with the previous one still serving.
- name: Sync internal nginx vhost
run: |
src="asset/nginx/helexa.internal.conf"
# The sudoers drop-in that permits this is installed by
# script/infra-setup.sh, which is run by hand, so the live copy
# can lag the repo by however long it has been since anyone ran
# it. Check EVERY grant the step needs, not a representative
# one: the first version of this preflight tested `nginx -t`
# alone, which had existed for ages, so it passed and the job
# then died on the two helexa-ratelimit.conf grants added in the
# same commit as the check. A preflight that names a remedy
# while probing the wrong permission is worse than none — it
# tells you the sudoers are fine seconds before they are not.
#
# Each probe runs the command in a harmless form that still
# matches its sudoers pattern (sha256sum reads; nginx -t and
# `systemctl show` do not mutate). rsync and ln have no such
# form, so they are checked with `sudo -nl`, which reports
# whether a command is permitted without running it.
missing=""
probe() { # probe <description> <command...>
local what="$1"; shift
ssh gitea_ci@hanzalova.internal "$@" >/dev/null 2>&1 \
|| missing="${missing}
- ${what}"
}
probe "nginx -t" \
'sudo -n /usr/sbin/nginx -t'
probe "sha256sum /etc/nginx/sites-available/helexa.internal.conf" \
'sudo -n /usr/bin/sha256sum /etc/nginx/sites-available/helexa.internal.conf'
probe "sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf" \
'sudo -n /usr/bin/sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf'
probe "rsync -> /etc/nginx/sites-available/helexa.internal.conf" \
'sudo -nl /usr/bin/rsync -a x /etc/nginx/sites-available/helexa.internal.conf'
probe "rsync -> /etc/nginx/conf.d/helexa-ratelimit.conf" \
'sudo -nl /usr/bin/rsync -a x /etc/nginx/conf.d/helexa-ratelimit.conf'
probe "ln -sfn sites-available -> sites-enabled" \
'sudo -nl /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf /etc/nginx/sites-enabled/helexa.internal.conf'
probe "systemctl reload nginx" \
'sudo -nl /usr/bin/systemctl reload nginx'
if [ -n "${missing}" ]; then
echo "FAIL: gitea_ci is missing nginx sudoers grants on hanzalova:${missing}" >&2
echo "" >&2
echo " Re-run 'script/infra-setup.sh' once from an operator shell to" >&2
echo " install the current asset/sudoers.d/cortex-host.conf. Having run" >&2
echo " it before is not enough — grants are added to that file as deploy" >&2
echo " steps grow, and the host keeps whatever revision it last saw." >&2
exit 1
fi
rl="asset/nginx/helexa-ratelimit.conf"
local_sum=$(sha256sum "${src}" | cut -d' ' -f1)
local_rl=$(sha256sum "${rl}" | cut -d' ' -f1)
remote_sum=$(ssh gitea_ci@hanzalova.internal \
'sudo sha256sum /etc/nginx/sites-available/helexa.internal.conf 2>/dev/null | cut -d" " -f1' || true)
remote_rl=$(ssh gitea_ci@hanzalova.internal \
'sudo sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf 2>/dev/null | cut -d" " -f1' || true)
link_ok=$(ssh gitea_ci@hanzalova.internal \
'test -L /etc/nginx/sites-enabled/helexa.internal.conf && echo yes || echo no')
if [ "${local_sum}" = "${remote_sum}" ] \
&& [ "${local_rl}" = "${remote_rl}" ] \
&& [ "${link_ok}" = "yes" ]; then
echo "nginx config unchanged, sites-enabled is a symlink — nothing to do"
exit 0
fi
# Order matters: the vhost references names defined at http
# context by helexa-ratelimit.conf (`limit_req_zone`s and the
# `helexa_noquery` log format). Ship that first, or a vhost
# referencing a not-yet-installed name fails `nginx -t` and the
# deploy stops with the two files out of step.
echo "syncing nginx rate-limit/log-format: ${remote_rl:-<absent>} -> ${local_rl}"
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
"${rl}" gitea_ci@hanzalova.internal:/etc/nginx/conf.d/helexa-ratelimit.conf
echo "syncing nginx vhost: ${remote_sum:-<absent>} -> ${local_sum} (symlink=${link_ok})"
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
"${src}" gitea_ci@hanzalova.internal:/etc/nginx/sites-available/helexa.internal.conf
ssh gitea_ci@hanzalova.internal 'bash -s' <<'NGINX'
set -eu
sudo /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf \
/etc/nginx/sites-enabled/helexa.internal.conf
if ! sudo /usr/sbin/nginx -t; then
echo "FAIL: nginx -t rejected the synced config; not reloading" >&2
exit 1
fi
sudo /usr/bin/systemctl reload nginx
echo "nginx reloaded"
NGINX
# Roll helexa-router + helexa-upstream onto the federation service host
# (gallumbits). Same manifest-gated dnf pattern as deploy-cortex; the
# two services are independent daemons on one host, so one job handles
@@ -539,6 +700,10 @@ jobs:
|| github.event.workflow_run.conclusion == 'success'
}}
steps:
# Needed since "Sync router config" reads helexa-router.toml out of
# the tree. This job managed without a checkout while it only ran
# package upgrades over ssh.
- uses: actions/checkout@v4
- name: SSH init
run: |
mkdir -p ~/.ssh
@@ -606,6 +771,96 @@ jobs:
done
DEPLOY
# Router config (#287). Tracked because it holds no secrets, and
# synced here because being untracked is what let its
# `helexa/balanced` alias keep pointing at a retired model until
# beast fell over on 2026-08-27.
#
# After the package step, deliberately: this needs a sudoers grant
# installed by the hand-run script/infra-setup.sh, so on a host that
# has not seen the current grants it fails — and a step that fails
# before the package work would hold the service binaries hostage to
# an unrelated permission. deploy-gallumbits has no dependent jobs,
# so the blast radius stops here either way.
#
# Restarts only on a digest change: the router re-reads its aliases
# at startup, and a needless restart drops in-flight federation
# requests.
- name: Sync router config
run: |
missing=""
probe() {
local what="$1"; shift
ssh gitea_ci@gallumbits.kosherinata.internal "$@" >/dev/null 2>&1 \
|| missing="${missing}
- ${what}"
}
probe "sha256sum /etc/helexa-router/helexa-router.toml" \
'sudo -n /usr/bin/sha256sum /etc/helexa-router/helexa-router.toml'
probe "rsync -> /etc/helexa-router/helexa-router.toml" \
'sudo -nl /usr/bin/rsync -a x /etc/helexa-router/helexa-router.toml'
probe "systemctl restart helexa-router.service" \
'sudo -nl /usr/bin/systemctl restart helexa-router.service'
if [ -n "${missing}" ]; then
echo "FAIL: gitea_ci is missing sudoers grants on gallumbits:${missing}" >&2
echo "" >&2
echo " Re-run 'script/infra-setup.sh' once from an operator shell to" >&2
echo " install the current asset/sudoers.d/gallumbits-host.conf." >&2
exit 1
fi
local_sum=$(sha256sum helexa-router.toml | cut -d' ' -f1)
remote_sum=$(ssh gitea_ci@gallumbits.kosherinata.internal \
'sudo -n /usr/bin/sha256sum /etc/helexa-router/helexa-router.toml 2>/dev/null | cut -d" " -f1' || true)
if [ "${local_sum}" = "${remote_sum}" ]; then
echo "router config unchanged (${local_sum})"
exit 0
fi
# Two questions, not one. The bytes decide whether to SYNC; the
# parsed settings decide whether to RESTART. The first run of
# this step proved why they differ: the only change was a
# comment header, and it bounced the router anyway, dropping
# whatever federation requests were in flight. The router reads
# its aliases at startup, so a comment is not a reason to
# interrupt anyone.
#
# Reading the remote copy into CI is safe here precisely because
# this file holds no secrets — the same property that lets it be
# tracked at all.
#
# The `cat` grant is deliberately NOT in the preflight above: if
# it is missing the read yields nothing, the comparison reports
# "changed", and the step restarts on any byte difference. That
# is the old behaviour, not a broken one, so a host mid-way
# through a sudoers rollout degrades instead of failing.
remote_copy="$(mktemp)"
trap 'rm -f "${remote_copy}"' EXIT
ssh gitea_ci@gallumbits.kosherinata.internal \
'sudo -n /usr/bin/cat /etc/helexa-router/helexa-router.toml' \
> "${remote_copy}" 2>/dev/null || true
settings_changed=yes
if python3 script/toml-settings-equal.py "${remote_copy}" helexa-router.toml; then
settings_changed=no
fi
echo "syncing helexa-router.toml: ${remote_sum:-<absent>} -> ${local_sum}"
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
helexa-router.toml \
gitea_ci@gallumbits.kosherinata.internal:/etc/helexa-router/helexa-router.toml
if [ "${settings_changed}" = yes ]; then
ssh gitea_ci@gallumbits.kosherinata.internal \
'sudo -n /usr/bin/systemctl restart helexa-router.service'
echo "helexa-router restarted: settings changed"
else
echo "helexa-router left running: only comments or formatting changed"
fi
- name: Ensure firewalld allows helexa-router + helexa-upstream + helexa-tools + helexa-angels
run: |
ssh gitea_ci@gallumbits.kosherinata.internal '
@@ -870,6 +1125,12 @@ jobs:
cd helexa.ai
npm ci
npm run i18n:check
# Also run here, not just in ci.yml, for the same reason
# i18n:check is: pushes to main skip ci.yml, so this is the only
# gate a merge itself passes through. Cheap, and it is the exact
# case the web CI comment describes — a defect reaching main and
# surfacing as a broken deploy afterwards.
npm test
npm run build
- name: SSH init
+14 -1
View File
@@ -8,10 +8,23 @@ helexa.ai/.env.local
*.swo
.idea/
.vscode/
# Secret-bearing service configs. These carry API keys, a JWT secret, DB
# passwords and SMTP credentials, so they are hand-synced by
# script/infra-setup.sh rather than shipped by CI (#287 Gap 1).
#
# helexa-router.toml is deliberately NOT here: it holds no secrets, only
# a listen address, the product-tier aliases and a mesh endpoint. It was
# ignored by association, and that is precisely how its `helexa/balanced`
# alias came to point at a retired model for long enough to take beast
# down (2026-08-27). A file nothing can diff is a file nothing can catch.
# Operator-owned credential layer, merged after the CI-deployed config
# by cortex / helexa-upstream / helexa-angels. Must never be committed.
secrets.toml
**/secrets.toml
cortex.toml
doc/plan/*
/target-cuda/
.claude/
helexa-router.toml
helexa-upstream.toml
helexa-angels.toml
+72 -25
View File
@@ -7,10 +7,10 @@
# ─────────────────────────────────────────────────────────────────────
# neuron.toml for beast.hanzalova.internal
#
# 2x RTX 5090 (32 GB each) — TP-2 capable. Pre-warms Qwen3.6-27B with
# q5k ISQ across both GPUs at activation, matching the validate-neuron
# 2x RTX 5090 (32 GB each) — TP-2 capable. Pre-warms Qwen3.8-27B with
# q6k ISQ across both GPUs at activation, matching the validate-neuron
# invocation: `validate-neuron.sh beast.hanzalova.internal
# Qwen/Qwen3.6-27B q5k 2`.
# Qwen/Qwen3.8-27B q6k 2`.
#
# Synced to /etc/neuron/neuron.toml by script/infra-setup.sh. Edits
# take effect after the next deploy workflow run restarts the service
@@ -29,31 +29,78 @@ name = "candle"
# count. NEURON_BATCHING=0 (systemd drop-in) is the kill switch;
# removing this section reverts to batch-1 serialization.
[harness.candle.admission]
# 2, not 8 (#291). max_in_flight is both the admission bound and the
# batch engine's slot count, and the KV budget is divided by it at load.
# At 8 that gave each slot 18,852 tokens out of 150,816 — against a
# 131,072-token advertised context, which needs 4 GB of KV per card on
# its own and 32 GB across eight slots, on a 31 GB card already holding
# ~21 GB of weights. The advertised context and the slot count were
# incompatible by roughly 7x, and an agentic session ran in a ~19k
# window while seven idle slots held reservations.
# 8. max_in_flight bounds how many requests run concurrently. It does
# NOT partition the KV budget: since #257 (`eca42510`, 2026-08-15) KV is
# a shared pool and each request reserves what its own prompt needs via
# `enter_with_kv(principal, kv_reservation_mb(prompt_len, ...))`. A
# request too big for what is free waits for bytes to come back rather
# than being handed a fixed 1/N slice.
#
# Seven days of paired request spans on this host: 89.6% of busy time
# was ONE request in flight, 93.6% was two or fewer, and utilisation was
# 4.9%. Most of what little concurrency exists is helexa-bench's own
# synthetic bursts. Two slots cover 93.6% of observed load and give each
# ~75k tokens; the batch engine is still worth 1.79x there (5.24x at
# eight is throughput nobody is asking this host for). Anything above
# two queues on admission (#53) with Retry-After rather than being
# refused.
max_in_flight = 2
# This value was briefly set to 2 (#291) on the stated grounds that "the
# KV budget is divided by it at load, so 8 gives each slot 18,852
# tokens". That was true before #257 and false after it, and #291 was
# filed and actioned after that landed. Nothing in the tree divides the
# budget by the slot count — `kv_budget` is a semaphore of MiB permits
# and the engine's `max_slots` is only the batch width.
#
# The concurrency measurement in #291 stands as data (89.6% of busy time
# at one request in flight, 4.9% utilisation over seven days) but the
# conclusion drawn from it does not, and the traffic it described has
# changed: while it was taken, the router's `helexa/balanced` alias
# pointed web chat at a second 27B, so per-model concurrency was split
# across two models that were evicting each other. With both tiers now
# resolving to this model, chat concurrency lands here.
#
# What actually bounds a busy moment is the pool: 3,689 MiB at
# 32 KiB/token/card is ~118k tokens of KV to share. Eight ordinary chat
# turns fit easily; a few very long agentic sessions will queue on bytes,
# which is the correct behaviour and is what #257 built.
max_in_flight = 8
# Snapshot budget for the cross-request prefix cache (#292).
#
# 3 GiB, not the 1 GiB default. At the measured 32 KiB/token/card, a
# 1 GiB budget cannot hold a snapshot beyond ~32k tokens of K/V alone —
# less once the GatedDeltaNet conv/recurrent state is deep-copied on top,
# which on this arch is 48 of 64 layers in f32. Agentic sessions here run
# well past that: `cached_tokens` was observed pinned at 2,016 for ten
# consecutive turns while the prompt grew to 52,869, so every turn
# re-prefilled ~50k tokens — 37 s each, against ~20 s of actual decode.
# The cache was not missing; the snapshot would not fit.
#
# The snapshot budget is reserved BEFORE the KV budget, not from what is
# left over: measured on this host, kv_budget_mb = 5737 - budget_mb,
# exactly 1:1. So this number is a direct trade against how long a
# single session can get, and both sides have a floor:
#
# budget_mb kv_budget tokens/slot (2 slots) holds a 52.9k snapshot
# 1024 4713 75,400 no ← the #292 symptom
# 3072 2665 42,600 yes, but sessions
# cannot reach 52.9k
# 2048 3689 59,000 yes
#
# 2 GiB is the value that satisfies both: a 52,869-token snapshot is
# ~1,652 MB of K/V plus the GatedDeltaNet state (~1.73 GB total), and
# per-slot KV still covers sessions comfortably past the deepest
# observed. 3 GiB was tried first and starved KV — a session could no
# longer reach the size whose snapshot the cache was being enlarged to
# hold, which is self-defeating.
#
# max_entries stays at the default 8: the byte budget binds first here,
# so the count cap is not what is forcing snapshots to be small.
[harness.candle.prefix_cache]
budget_mb = 2048
[[default_models]]
# Qwen3.8-27B is the resident flagship (#251). Architecturally identical
# to the 3.6-27B it replaced — same model_type, layer counts and vision
# tower — so the load spec is unchanged. The 3.6-27B stays catalogued at
# the same residency class, so it can still be cold-loaded on demand and
# the two displace each other in either direction.
# Qwen3.8-27B is the resident flagship (#251). It replaced the
# 3.6-27B — architecturally identical, same model_type, layer counts and
# vision tower — so the load spec is unchanged.
#
# The 3.6-27B is no longer catalogued (retired 2026-08-27). While both
# were listed at the same residency class they could displace each other
# in either direction, which is what let a stale `helexa/balanced` alias
# evict this model on every authenticated web-chat turn. See models.toml
# for the incident that ended the arrangement.
model_id = "Qwen/Qwen3.8-27B"
harness = "candle"
quant = "q6k"
+58
View File
@@ -0,0 +1,58 @@
# Internal-only vhost for the investor portal (helexa-angels).
#
# Useful where the public name does not resolve or connect correctly from
# inside the network that hosts it — common without NAT reflection or
# split-horizon DNS. Optional: skip this file if your public name works
# from everywhere.
#
# Cert: internal CA, renewed by step@angels.timer.
#
# Enable with a relative symlink into sites-enabled:
# ln -sf ../sites-available/angels.internal.conf /etc/nginx/sites-enabled/
upstream angels_up {
server gallumbits.kosherinata.internal:8092;
keepalive 16;
}
server {
listen 127.0.0.1:14443 ssl proxy_protocol;
http2 on;
server_name angels.internal;
ssl_certificate /etc/nginx/tls/cert/angels.internal.pem;
ssl_certificate_key /etc/nginx/tls/key/angels.internal.pem;
ssl_protocols TLSv1.3;
ssl_trusted_certificate /etc/pki/ca-trust/source/anchors/root-internal.pem;
# The material is confidential whichever name reaches it, so the same
# headers as the public vhost. `always` so they survive error responses.
add_header X-Robots-Tag "noindex, nofollow, noarchive, nosnippet" always;
add_header Referrer-Policy "no-referrer" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "DENY" always;
add_header Content-Security-Policy
"default-src 'none'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; form-action 'self'; base-uri 'none'; frame-ancestors 'none'" always;
# Same per-IP limits as the public path: an invite code is equally
# guessable from here, and argon2 costs the same CPU whoever spends it.
location /i/ {
limit_req zone=angels_invite burst=5 nodelay;
proxy_pass http://angels_up;
include /etc/nginx/conf.d/angels-proxy-params.conf;
}
location /signin {
limit_req zone=angels_auth burst=5 nodelay;
proxy_pass http://angels_up;
include /etc/nginx/conf.d/angels-proxy-params.conf;
}
location /register {
limit_req zone=angels_auth burst=5 nodelay;
proxy_pass http://angels_up;
include /etc/nginx/conf.d/angels-proxy-params.conf;
}
location / {
proxy_pass http://angels_up;
include /etc/nginx/conf.d/angels-proxy-params.conf;
}
}
+21
View File
@@ -24,3 +24,24 @@ limit_req_zone $binary_remote_addr zone=helexa_tools:10m rate=30r/m;
limit_req_zone $binary_remote_addr zone=angels_invite:1m rate=20r/m;
limit_req_zone $binary_remote_addr zone=angels_auth:1m rate=12r/m;
limit_req_status 429;
# Access-log format for the search locations, which must not record what
# people searched for.
#
# The default `main` format logs `$request` — method, full URI *and
# query string*. For a search proxy that means every query a user types
# is written to /var/log/nginx/access.log next to their IP, for
# authenticated and anonymous visitors alike. SearXNG's own web UI
# defaults to POST partly to avoid exactly this.
#
# `$uri` is the path with the query stripped, so the operational signal
# survives — who called, when, which endpoint, what status, how big, how
# long — and the query itself does not. `$request_method` is kept
# explicit since `$uri` drops it.
#
# Rate-limit debugging and abuse detection both work from this: they
# need counts, statuses and client addresses, never the search terms.
log_format helexa_noquery '$remote_addr - $remote_user [$time_local] '
'"$request_method $uri $server_protocol" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $request_time';
+37 -1
View File
@@ -73,6 +73,9 @@ server {
location = /tools/fetch {
limit_req zone=helexa_tools burst=10 nodelay;
limit_except GET { deny all; }
# The `url=` parameter is as revealing as a search query: it is
# the page a user is having read to them. Same treatment.
access_log /var/log/nginx/access.log helexa_noquery;
proxy_pass http://gallumbits.kosherinata.internal:8889/fetch;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
@@ -83,7 +86,40 @@ server {
# federation service host. GET only; the SPA requests format=json.
location = /tools/web_search {
limit_req zone=helexa_tools burst=10 nodelay;
limit_except GET { deny all; }
# GET and POST: SearXNG accepts both, and a client that POSTs
# keeps its query out of any intermediary's request line. Other
# methods have no meaning here.
limit_except GET POST { deny all; }
# A search query is small. Bounding the body stops this becoming
# a way to push arbitrary bytes at the upstream.
client_max_body_size 8k;
# Never log the query — see helexa_noquery in
# conf.d/helexa-ratelimit.conf.
access_log /var/log/nginx/access.log helexa_noquery;
proxy_pass http://gallumbits.kosherinata.internal:8888/search;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_read_timeout 30s;
}
# The same upstream under SearXNG's own path, for clients that speak
# the SearXNG API rather than our app's.
#
# `/tools/web_search` is our name for it, and a generic SearXNG
# client cannot guess it: they take a base URL and append `/search`,
# because that is what every SearXNG instance serves. Without this
# block such a request falls through to `location /` and is answered
# by the SPA — HTTP 200, `text/html`, and a JSON parse error at the
# client. A 404 would at least be honest; 200-with-HTML sends people
# looking for a bug in their own parser.
#
# Same upstream, same rate-limit zone, same GET-only restriction:
# this is an alias, not a second, laxer door.
location = /search {
limit_req zone=helexa_tools burst=10 nodelay;
limit_except GET POST { deny all; }
client_max_body_size 8k;
access_log /var/log/nginx/access.log helexa_noquery;
proxy_pass http://gallumbits.kosherinata.internal:8888/search;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+12
View File
@@ -13,6 +13,18 @@ gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/cortex/models.toml
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /var/www/bench.helexa.ai/
# deploy-website rsyncs the built helexa.ai SPA into the nginx webroot.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /var/www/helexa.ai/
# deploy.yml's "Sync internal nginx vhost" step: write the vhost, make
# sites-enabled a symlink to it, validate, reload. Scoped to the one
# file and the one service — `nginx -t` is included because a reload
# must never happen on a config nginx has rejected.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/nginx/sites-available/helexa.internal.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/nginx/conf.d/helexa-ratelimit.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/nginx/sites-available/helexa.internal.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf /etc/nginx/sites-enabled/helexa.internal.conf
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/nginx -t
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl reload nginx
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start cortex.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop cortex.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now cortex.service
+25 -3
View File
@@ -4,9 +4,17 @@
# rolls out helexa-router (data plane), helexa-upstream (account/budget
# authority), helexa-tools (grounding fetcher) and helexa-angels (the
# confidential investor portal) package upgrades.
# Configs are NOT synced by CI — they carry secrets (DB password, JWT
# secret, operator tokens) and ride in script/infra-setup.sh under the
# operator's own sudo instead.
# The SECRET-BEARING configs are not synced by CI — helexa-upstream.toml
# and helexa-angels.toml carry a DB password, a JWT secret and SMTP
# credentials, and ride in script/infra-setup.sh under the operator's own
# sudo instead (#287 Gap 1, still open).
#
# helexa-router.toml is different and IS synced here: it holds no secrets,
# only a listen address, the product-tier aliases and a mesh endpoint. It
# was excluded by association with its neighbours, which is how its
# `helexa/balanced` alias came to point at a model retired from the
# catalogue and took beast down on 2026-08-27. A config CI cannot write is
# a config no check can defend.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start helexa-router.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop helexa-router.service
@@ -21,6 +29,20 @@ gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start helexa-angels.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop helexa-angels.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now helexa-angels.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload
# deploy-gallumbits' "Sync router config" step. Scoped to the one file:
# the neighbouring service configs stay operator-only until #287 Gap 1
# gives them a render-from-secrets path.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/helexa-router/helexa-router.toml
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/helexa-router/helexa-router.toml
# Read-back, so the sync can tell a settings change from a comment
# change and restart only for the former. Safe to grant and safe to read
# into CI because this file holds no secrets — the same property that
# lets it be tracked. If this grant is absent the step degrades to
# restarting on any byte change, which is why it is not in the step's
# fail-hard preflight.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/cat /etc/helexa-router/helexa-router.toml
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl restart helexa-router.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/dnf install --refresh --allowerasing -y helexa-router
gitea_ci ALL=(root) NOPASSWD: /usr/bin/dnf upgrade --refresh --allowerasing -y helexa-router
gitea_ci ALL=(root) NOPASSWD: /usr/bin/dnf install --refresh --allowerasing -y helexa-upstream
+8
View File
@@ -67,6 +67,13 @@ pub struct ModelProfile {
/// model, with nothing forcing them to agree.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub sampling: Option<SamplingOverride>,
/// The model's own `preserve_thinking` template control, forwarded
/// on cold load so a model this gateway loads behaves the same as
/// one the host was configured to hold resident. Two config files
/// describing one model is how #252 happened; this rides along for
/// the same reason `sampling` does.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub preserve_thinking: Option<bool>,
// ── Enrichment (issue #62) ────────────────────────────────
/// Per-model token budget. When present, advertised in `/v1/models`
@@ -251,6 +258,7 @@ mod tests {
residency_priority: None,
source: None,
sampling: None,
preserve_thinking: None,
limit: None,
cost: None,
capabilities: vec![],
+141
View File
@@ -143,15 +143,48 @@ pub struct NeuronEndpoint {
impl GatewayConfig {
/// Load configuration from a TOML file, with environment variable overrides.
/// Env vars are prefixed with `CORTEX_` and use `__` as a separator.
///
/// A sibling `secrets.toml`, if present, is merged **after** the main
/// file and therefore wins. It exists so the two have different
/// owners: CI writes the main config, and only the operator writes
/// the secrets — see [`secrets_path`].
pub fn load(path: impl AsRef<Path>) -> Result<Self, Box<figment::Error>> {
let path = path.as_ref();
Figment::new()
.merge(Toml::file(path))
.merge(Toml::file(secrets_path(path)))
.merge(Env::prefixed("CORTEX_").split("__"))
.extract()
.map_err(Box::new)
}
}
/// `secrets.toml` beside the main config.
///
/// The split exists to make a class of outage impossible rather than
/// merely unlikely. Config that CI cannot write is config no check can
/// defend: `helexa-router.toml` was excluded from git because it sat
/// beside secret-bearing files, and its `helexa/balanced` alias then
/// pointed at a retired model for long enough to take a node down
/// (2026-08-27). The fix is not to hand CI the credentials — it is to
/// stop mixing the two in one file.
///
/// So `cortex.toml` holds structure and is deployed by CI, while
/// `secrets.toml` holds API keys and the upstream bearer, is written
/// only by the operator, and is never read, written or diffed by the
/// pipeline. Because it merges last it also wins, so a value present in
/// both is the operator's.
///
/// A missing file is not an error — figment treats an absent
/// `Toml::file` as an empty source — so a host that keeps everything in
/// one file, and every existing deployment, keeps working unchanged.
fn secrets_path(config: &Path) -> std::path::PathBuf {
config
.parent()
.unwrap_or_else(|| Path::new("."))
.join("secrets.toml")
}
impl Default for GatewayConfig {
fn default() -> Self {
Self {
@@ -170,3 +203,111 @@ impl Default for GatewayConfig {
}
}
}
#[cfg(test)]
mod secrets_layer_tests {
use super::*;
use std::io::Write;
/// Write `main` (and optionally `secrets.toml`) into a fresh directory
/// under the target dir, and load through the real code path.
fn load_with(dir: &str, main: &str, secrets: Option<&str>) -> GatewayConfig {
// `CARGO_TARGET_TMPDIR` is only set for integration tests, not for
// unit tests inside src/, so derive a unique directory instead.
let base = std::env::temp_dir().join(format!("helexa-cortex-config-{dir}"));
let _ = std::fs::remove_dir_all(&base);
std::fs::create_dir_all(&base).expect("create test dir");
let cfg = base.join("cortex.toml");
write!(
std::fs::File::create(&cfg).expect("create config"),
"{main}"
)
.expect("write");
if let Some(s) = secrets {
let p = base.join("secrets.toml");
write!(std::fs::File::create(&p).expect("create secrets"), "{s}").expect("write");
}
GatewayConfig::load(&cfg).expect("config loads")
}
const MAIN: &str = r#"
[gateway]
listen = "0.0.0.0:31313"
metrics_listen = "0.0.0.0:31314"
[eviction]
strategy = "lru"
defrag_after_cycles = 50
[entitlements]
require_auth = true
[[neurons]]
name = "beast"
endpoint = "http://beast.invalid:13131"
"#;
/// The whole point: a host with no secrets.toml behaves exactly as
/// before. Every current deployment is in this state, so if this
/// regressed the change would take the fleet down on rollout rather
/// than on adoption.
#[test]
fn an_absent_secrets_file_is_not_an_error() {
let cfg = load_with("secrets_absent", MAIN, None);
assert!(cfg.entitlements.require_auth);
assert!(
cfg.entitlements.keys.is_empty(),
"no keys configured anywhere means no keys"
);
}
/// The array-of-tables case that ruled out env-var overrides:
/// `[[entitlements.keys]]` cannot be expressed as `CORTEX_*` env vars,
/// so the second TOML layer is what carries it.
#[test]
fn secrets_file_supplies_the_entitlement_keys() {
let cfg = load_with(
"secrets_keys",
MAIN,
Some(
r#"
[[entitlements.keys]]
key = "sk-test-aaa"
account_id = "acct-a"
key_id = "a"
[[entitlements.keys]]
key = "sk-test-bbb"
account_id = "acct-b"
key_id = "b"
"#,
),
);
assert_eq!(cfg.entitlements.keys.len(), 2);
assert_eq!(cfg.entitlements.keys[0].key, "sk-test-aaa");
assert_eq!(cfg.entitlements.keys[1].account_id, "acct-b");
assert!(
cfg.entitlements.require_auth,
"structure from the CI-owned file must survive the merge"
);
}
/// Ordering is the safety property. If CI ever ships a placeholder in
/// the main file, the operator's real value must still win — otherwise
/// a deploy silently swaps a live credential for a dummy.
#[test]
fn the_operators_value_wins_over_the_deployed_one() {
let cfg = load_with(
"secrets_precedence",
&format!(
"{MAIN}\n[upstream]\nenabled = true\nurl = \"https://example.invalid\"\nbearer = \"PLACEHOLDER\"\n"
),
Some("[upstream]\nbearer = \"real-token\"\n"),
);
assert_eq!(
cfg.upstream.bearer, "real-token",
"secrets.toml merges last, so it wins"
);
assert_eq!(
cfg.upstream.url, "https://example.invalid",
"fields the operator does not override keep the deployed value"
);
}
}
+112 -1
View File
@@ -55,13 +55,78 @@ pub struct SamplingOverride {
pub top_p: Option<f64>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub top_k: Option<usize>,
/// Multiplier applied to recently-generated tokens before sampling.
/// `1.0` disables it; above that, recently-emitted tokens get less
/// likely.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub repeat_penalty: Option<f32>,
/// How many recently-generated tokens the penalty considers.
///
/// The reason this is operator-settable at all: the built-in default
/// is 64, inherited from candle's chat example, and a reasoning
/// model's think block runs to tens of thousands of tokens. Anything
/// restated more than 64 tokens later is invisible to the penalty,
/// which is how a 28,672-token think block can contain the same
/// derivation five times over.
///
/// Raising it is a real trade, not a free win: the penalty is
/// token-level, and source code legitimately repeats tokens
/// constantly (`const`, `function`, brace runs). A wide window at a
/// meaningful penalty distorts exactly the output this fleet is for.
/// Sequence-level repetition wants a sequence-level instrument
/// (frequency/presence penalties, or a DRY-style sampler); this knob
/// exists so the trade can be *measured* per model rather than
/// guessed once and frozen into a constant.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub repeat_last_n: Option<usize>,
/// Sequence-wide one-time penalty for already-seen tokens.
///
/// This, not `repeat_last_n`, is the remedy Qwen prescribe for the
/// endless repetition their models fall into during long reasoning
/// (0..2, 1.5 when severe). It is scored over everything generated
/// so far, so a passage restated 5,000 tokens later is still
/// penalised — the case a trailing window cannot see.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub presence_penalty: Option<f32>,
/// Sequence-wide penalty proportional to a token's existing count.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub frequency_penalty: Option<f32>,
/// Pin the sampler's RNG for every request to this model.
///
/// Exists because parameter search needs a control: comparing two
/// `presence_penalty` values across unseeded runs cannot separate
/// the parameter's effect from sampling variance, and the clients
/// that would otherwise pin it cannot — pi's request builder sends
/// no `seed` at all, so a caller-side seed is not available on the
/// path that matters.
///
/// Note what this does and does not give. It removes sampling noise
/// as a confounder; it does not make two runs comparable
/// token-by-token, because changing a penalty changes the logits
/// and the sequences diverge at the first step where a token choice
/// flips.
///
/// Leaving this set is a product decision, not just a testing one:
/// every identical prompt then yields an identical answer for every
/// caller. That is defensible — predictable and cacheable — but it
/// removes the variation a user may expect from retrying. A request
/// that names its own seed still wins.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub seed: Option<u64>,
}
impl SamplingOverride {
/// True when the operator set nothing — an empty `[sampling]` table
/// must behave exactly as no table at all.
pub fn is_empty(&self) -> bool {
self.temperature.is_none() && self.top_p.is_none() && self.top_k.is_none()
self.temperature.is_none()
&& self.top_p.is_none()
&& self.top_k.is_none()
&& self.repeat_penalty.is_none()
&& self.repeat_last_n.is_none()
&& self.presence_penalty.is_none()
&& self.frequency_penalty.is_none()
&& self.seed.is_none()
}
}
@@ -86,6 +151,27 @@ pub struct ModelSpec {
/// still round-trips the spec.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub sampling: Option<SamplingOverride>,
/// Whether prior assistant turns keep their reasoning when the chat
/// template re-renders the conversation.
///
/// This is the model's own `preserve_thinking` template control, not
/// an invention of ours. `Qwen/Qwen3.8-27B` defaults it to *true* —
/// the whole transcript's think blocks are replayed — and `false`
/// keeps reasoning only for turns after the last user query, i.e.
/// the turn in progress.
///
/// `None` leaves the kwarg unset so the template's own default
/// applies, which is the only safe default: the model's authors
/// chose it, and overriding that fleet-wide on a hunch is how you
/// ship a regression you cannot see.
///
/// Exposed as operator config so the premise can be A/B'd on a real
/// workload — full replay is the larger prompt and pushes against
/// the prefix-cache budget and the throughput-derived context
/// ceiling, but whether it *helps* the model is a measurement, not
/// a belief. A request's own `chat_template_kwargs` still wins.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub preserve_thinking: Option<bool>,
}
/// Per-model token budget advertised by the catalogue or neuron.
@@ -173,6 +259,21 @@ pub struct ReasoningBudgetRung {
/// Qwen3.8 that silently means `xhigh`.
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub default: bool,
/// Why this rung is not being offered right now, or `None` when it
/// is. Absent from the wire in the normal case.
///
/// Availability is a *capacity* statement, not a permission: the
/// request path does not reject an unavailable rung. A caller that
/// insists still gets it, which keeps this from becoming a new
/// refusal path for every existing API client. It exists so a UI can
/// avoid offering an option that will serve the user badly, and so
/// an integrator reading `/v1/models` can see the same thing.
///
/// Today the only reason is concurrency: on a node running many
/// slots, the longest rung is reported unavailable. See
/// `reasoning_budget::availability` for the argument and the knob.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub unavailable_reason: Option<String>,
}
/// Operator-set pricing, **USD per 1,000,000 tokens, as JSON numbers**
@@ -247,6 +348,16 @@ pub struct ModelInfo {
/// marker tokens (`<think>` / `<\/think>` or similar).
#[serde(default)]
pub reasoning: bool,
/// The operator's `preserve_thinking` for this loaded model, or
/// `None` when unset and the template's own default applies.
///
/// Advertised so a run can be *stamped* with the value it ran
/// under. This is an A/B knob whose whole purpose is comparison
/// between runs, and a comparison whose arms cannot be told apart
/// afterwards is not a comparison. Recovering it from deploy
/// history is guesswork — a config sync can land mid-session.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub preserve_thinking: Option<bool>,
/// Whether this node can actually serve a request for this model
/// right now (#245).
+12
View File
@@ -67,6 +67,18 @@ pub struct ModelEntry {
/// older persisted/serialised entries deserialisable.
#[serde(default)]
pub capabilities: Vec<String>,
/// The reasoning ladder this *model* offers, with any rung the node
/// is currently withholding marked unavailable.
///
/// Per model, not per node. It was held on `NodeState` on the
/// reasoning that "every model on a neuron reports the same rungs",
/// which was true while a rung was only a name from the model's
/// template. It stopped being true once availability began depending
/// on the model's own `max_in_flight`: two models on one host can
/// have different admission settings, and the node-level copy would
/// attribute one model's withheld rung to the other.
#[serde(default)]
pub reasoning_budget: Vec<crate::harness::ReasoningBudgetRung>,
/// Runtime-detected capability flags from the neuron's `/models`
/// response (`ModelInfo`). `false` when the neuron predates these
/// fields or hasn't reported them yet.
+14
View File
@@ -43,6 +43,20 @@ pub struct ChatCompletionRequest {
/// why it is a knob rather than a constant (#272).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub repeat_last_n: Option<usize>,
/// OpenAI core, `-2.0..=2.0`. Subtracted once from any token that
/// has already appeared, regardless of how often.
///
/// Unlike `repetition_penalty` this is scored over the **whole**
/// generated sequence, not a trailing window — which is what a
/// reasoning model needs, and what Qwen recommend (0..2, 1.5 for
/// severe cases) for the endless-repetition failure their own
/// models exhibit during long thinking.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub presence_penalty: Option<f32>,
/// OpenAI core, `-2.0..=2.0`. Subtracted in proportion to how many
/// times a token has already appeared.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub frequency_penalty: Option<f32>,
/// The deprecated spelling of the output cap. Still what most
/// clients send; see [`ChatCompletionRequest::effective_max_tokens`].
#[serde(skip_serializing_if = "Option::is_none")]
+32
View File
@@ -70,6 +70,11 @@ pub struct ResponsesRequest {
/// Window `repetition_penalty` considers (#272).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub repeat_last_n: Option<usize>,
/// OpenAI core; forwarded to the chat path unchanged.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub presence_penalty: Option<f32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub frequency_penalty: Option<f32>,
/// Chained-conversation identifier. We don't store responses
/// server-side yet; if this is `Some`, the handler returns 400.
#[serde(default, skip_serializing_if = "Option::is_none")]
@@ -271,6 +276,15 @@ pub enum ResponsesOutputItem {
#[serde(default = "default_item_status")]
status: String,
},
/// A think block, as its own output item ahead of the message —
/// the same shape the streaming projector emits, so a client sees
/// one contract whether or not it streams (#300).
Reasoning {
id: String,
summary: Vec<ResponsesSummaryPart>,
#[serde(default = "default_item_status")]
status: String,
},
/// Reserved for the day tool-call extraction lands. The wire
/// shape mirrors `ResponsesInputItem::FunctionCall`.
FunctionCall {
@@ -283,6 +297,24 @@ pub enum ResponsesOutputItem {
},
}
/// One `summary_text` part of a reasoning output item.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ResponsesSummaryPart {
/// Always `"summary_text"`.
#[serde(rename = "type")]
pub part_type: String,
pub text: String,
}
impl ResponsesSummaryPart {
pub fn summary_text(text: impl Into<String>) -> Self {
Self {
part_type: "summary_text".into(),
text: text.into(),
}
}
}
fn default_item_status() -> String {
"completed".into()
}
+204 -7
View File
@@ -130,6 +130,8 @@ pub fn anthropic_to_openai(req: MessagesRequest) -> ChatCompletionRequest {
seed: None,
repetition_penalty: None,
repeat_last_n: None,
presence_penalty: None,
frequency_penalty: None,
max_tokens: Some(req.max_tokens),
// Anthropic has one spelling; it maps onto the legacy field.
max_completion_tokens: None,
@@ -413,17 +415,26 @@ fn anthropic_content_to_text(content: AnthropicContent) -> String {
pub fn openai_to_anthropic(resp: ChatCompletionResponse) -> MessagesResponse {
let choice = resp.choices.into_iter().next();
let (content_text, stop_reason) = match choice {
let (content_text, reasoning_text, stop_reason) = match choice {
Some(c) => {
let text = match c.message.content {
MessageContent::Text(t) => t,
MessageContent::Parts(parts) => serde_json::to_string(&parts).unwrap_or_default(),
MessageContent::Null => String::new(),
};
// The model's think block, which the harness returns on
// `message.reasoning_content` (#300).
let reasoning = c
.message
.extra
.get("reasoning_content")
.and_then(|v| v.as_str())
.map(str::to_string)
.filter(|t| !t.is_empty());
let stop = c.finish_reason.map(|r| map_stop_reason(&r));
(text, stop)
(text, reasoning, stop)
}
None => (String::new(), None),
None => (String::new(), None, None),
};
let usage = resp.usage.unwrap_or(Usage {
@@ -439,10 +450,32 @@ pub fn openai_to_anthropic(resp: ChatCompletionResponse) -> MessagesResponse {
id: resp.id,
response_type: "message".into(),
role: "assistant".into(),
content: vec![ContentBlock {
block_type: "text".into(),
data: json!({ "text": content_text }),
}],
// A `thinking` block ahead of the text, which is Anthropic's
// own ordering. This path accepted `thinking.budget_tokens` on
// the way in (see `anthropic_to_openai`) and returned nothing on
// the way out: the caller spent the tokens, could not see them,
// and could not replay the model's reasoning next turn (#300).
//
// Emitted without a `signature`. Anthropic signs its thinking
// blocks and a client that round-trips one may expect the field;
// we cannot mint a valid signature and inventing one would be
// worse than omitting it. If a real client rejects an unsigned
// block, the fix is to gate this behind a flag — not to go back
// to silently deleting the model's work.
content: {
let mut blocks = Vec::with_capacity(2);
if let Some(thinking) = reasoning_text {
blocks.push(ContentBlock {
block_type: "thinking".into(),
data: json!({ "thinking": thinking }),
});
}
blocks.push(ContentBlock {
block_type: "text".into(),
data: json!({ "text": content_text }),
});
blocks
},
model: resp.model,
stop_reason,
usage: AnthropicUsage {
@@ -499,6 +532,7 @@ pub struct AnthropicStreamTranslator {
enum OpenBlock {
Text,
ToolUse,
Thinking,
}
impl AnthropicStreamTranslator {
@@ -538,6 +572,32 @@ impl AnthropicStreamTranslator {
}
for choice in &chunk.choices {
// The model's think block. Dropped entirely until #300: this
// translator read `content` and `tool_calls` and nothing
// else, so an Anthropic client asked for extended thinking
// (`thinking.budget_tokens`, which `anthropic_to_openai`
// accepts), was billed for the tokens, and received a stream
// with no thinking in it — every turn, silently. Fixing only
// the non-streaming translation would have missed this,
// because the clients that ask for thinking stream.
if let Some(text) = choice
.delta
.get("reasoning_content")
.and_then(Value::as_str)
&& !text.is_empty()
{
self.ensure_thinking_block(&mut out);
let index = self.open_block.as_ref().map(|(i, _)| *i).unwrap_or(0);
out.push((
"content_block_delta".to_string(),
json!({
"type": "content_block_delta",
"index": index,
"delta": { "type": "thinking_delta", "thinking": text }
}),
));
}
if let Some(text) = choice.delta.get("content").and_then(Value::as_str)
&& !text.is_empty()
{
@@ -651,6 +711,32 @@ impl AnthropicStreamTranslator {
out
}
/// Open a `thinking` block, closing whatever preceded it.
///
/// Emitted without a `signature`. Anthropic signs its thinking
/// blocks; we cannot mint a valid one, and inventing a value would
/// be worse than omitting it. If a client rejects an unsigned block
/// the fix is a flag — not a return to deleting the model's work.
fn ensure_thinking_block(&mut self, out: &mut Vec<(String, Value)>) {
match &self.open_block {
Some((_, OpenBlock::Thinking)) => {}
_ => {
self.close_open_block(out);
let index = self.next_index;
self.next_index += 1;
self.open_block = Some((index, OpenBlock::Thinking));
out.push((
"content_block_start".to_string(),
json!({
"type": "content_block_start",
"index": index,
"content_block": { "type": "thinking", "thinking": "" }
}),
));
}
}
}
fn ensure_text_block(&mut self, out: &mut Vec<(String, Value)>) {
match &self.open_block {
Some((_, OpenBlock::Text)) => {}
@@ -1060,4 +1146,115 @@ mod request_tests {
let tools = openai.extra.get("tools").and_then(Value::as_array).unwrap();
assert_eq!(tools[0]["function"]["name"], "x");
}
/// The Anthropic path must return the model's thinking (#300).
///
/// It accepted `thinking.budget_tokens` on the way in and returned
/// a bare text block on the way out: the caller spent the tokens,
/// never saw them, and could not replay the model's reasoning on
/// the next turn. Taking a parameter and silently discarding what
/// it buys is the defect, not the missing field.
#[test]
fn anthropic_response_carries_a_thinking_block() {
let mut extra = serde_json::Map::new();
extra.insert("reasoning_content".into(), json!("weighing the options"));
let resp = ChatCompletionResponse {
id: "id".into(),
object: "chat.completion".into(),
created: 0,
model: "m".into(),
choices: vec![crate::openai::ChatCompletionChoice {
index: 0,
message: ChatMessage {
role: "assistant".into(),
content: MessageContent::Text("the answer".into()),
extra: Value::Object(extra),
},
finish_reason: Some("stop".into()),
extra: Value::Null,
}],
usage: None,
extra: Value::Null,
};
let a = openai_to_anthropic(resp);
assert_eq!(a.content.len(), 2, "expected thinking + text");
assert_eq!(a.content[0].block_type, "thinking", "thinking comes first");
assert_eq!(a.content[0].data["thinking"], "weighing the options");
assert_eq!(a.content[1].block_type, "text");
assert_eq!(a.content[1].data["text"], "the answer");
}
/// No thinking means the unchanged single text block.
#[test]
fn anthropic_response_without_thinking_is_unchanged() {
let resp = ChatCompletionResponse {
id: "id".into(),
object: "chat.completion".into(),
created: 0,
model: "m".into(),
choices: vec![crate::openai::ChatCompletionChoice {
index: 0,
message: ChatMessage {
role: "assistant".into(),
content: MessageContent::Text("plain".into()),
extra: Value::Object(Default::default()),
},
finish_reason: Some("stop".into()),
extra: Value::Null,
}],
usage: None,
extra: Value::Null,
};
let a = openai_to_anthropic(resp);
assert_eq!(a.content.len(), 1);
assert_eq!(a.content[0].block_type, "text");
}
/// Streaming Anthropic must carry thinking too (#300).
///
/// Fixing only the non-streaming translation would have missed this
/// entirely — and missed it for the clients that matter, since the
/// ones asking for extended thinking stream.
#[test]
fn anthropic_stream_emits_thinking_deltas() {
use crate::openai::{ChatCompletionChunk, ChunkChoice};
let mut t = AnthropicStreamTranslator::new();
let chunk = |delta: Value| ChatCompletionChunk {
id: "c".into(),
object: "chat.completion.chunk".into(),
created: 0,
model: "m".into(),
choices: vec![ChunkChoice {
index: 0,
delta,
finish_reason: None,
extra: Value::Null,
}],
usage: None,
extra: Value::Null,
};
let ev = t.on_chunk(&chunk(json!({ "reasoning_content": "weighing it" })));
let start = ev
.iter()
.find(|(n, _)| n == "content_block_start")
.expect("a block must open");
assert_eq!(start.1["content_block"]["type"], "thinking");
let delta = ev
.iter()
.find(|(n, _)| n == "content_block_delta")
.expect("a delta must follow");
assert_eq!(delta.1["delta"]["type"], "thinking_delta");
assert_eq!(delta.1["delta"]["thinking"], "weighing it");
// Visible text opens a separate block, after thinking closes.
let ev = t.on_chunk(&chunk(json!({ "content": "the answer" })));
let names: Vec<&str> = ev.iter().map(|(n, _)| n.as_str()).collect();
assert!(
names.contains(&"content_block_stop"),
"the thinking block must close first: {names:?}"
);
let start = ev.iter().find(|(n, _)| n == "content_block_start").unwrap();
assert_eq!(start.1["content_block"]["type"], "text");
}
}
+14 -2
View File
@@ -847,8 +847,16 @@ async fn list_models(State(fleet): State<Arc<CortexState>>) -> Json<Value> {
// First non-empty wins: a catalogue-seeded entry
// starts with none, and neurons serving the same
// model are configured alike.
// The model's own ladder wins; the node-level copy is
// only a fallback for a neuron too old to report one
// per model. Preferring the node copy would discard
// the per-model availability computed at the source.
if e.reasoning_budget.is_empty() {
e.reasoning_budget = node.reasoning_budget.clone();
e.reasoning_budget = if entry.reasoning_budget.is_empty() {
node.reasoning_budget.clone()
} else {
entry.reasoning_budget.clone()
};
}
})
.or_insert_with(|| CortexModelEntry {
@@ -872,7 +880,11 @@ async fn list_models(State(fleet): State<Arc<CortexState>>) -> Json<Value> {
context_window: None,
context_length: None,
// What this neuron says each effort level buys (#223).
reasoning_budget: node.reasoning_budget.clone(),
reasoning_budget: if entry.reasoning_budget.is_empty() {
node.reasoning_budget.clone()
} else {
entry.reasoning_budget.clone()
},
});
}
}
+9 -6
View File
@@ -130,12 +130,13 @@ async fn poll_neuron(fleet: &CortexState, name: &str, endpoint: &str) {
Ok(resp) if resp.status().is_success() => {
match resp.json::<Vec<ModelInfo>>().await {
Ok(models) => {
// The reasoning-effort ladder is host configuration:
// every model on a neuron reports the same rungs, so
// it is held once per node rather than copied onto
// each model entry where the two could drift (#223).
// Taken from the first model that offers one — a
// non-reasoning model advertises none.
// Node-level ladder, kept only as a fallback for a
// model whose own entry carries none. It is NOT the
// source of truth any more: availability depends on
// the model's `max_in_flight`, so two models on one
// host can legitimately offer different rungs and a
// single node-level copy would attribute one's
// withheld rung to the other. Per-model is set below.
node.reasoning_budget = models
.iter()
.map(|m| m.reasoning_budget.clone())
@@ -158,6 +159,7 @@ async fn poll_neuron(fleet: &CortexState, name: &str, endpoint: &str) {
// authoritative source the gateway advertises.
e.limit = upstream.limit.clone();
e.servable = upstream.servable.clone();
e.reasoning_budget = upstream.reasoning_budget.clone();
})
.or_insert_with(|| ModelEntry {
id: upstream.id.clone(),
@@ -169,6 +171,7 @@ async fn poll_neuron(fleet: &CortexState, name: &str, endpoint: &str) {
reasoning: upstream.reasoning,
limit: upstream.limit.clone(),
servable: upstream.servable.clone(),
reasoning_budget: upstream.reasoning_budget.clone(),
});
}
+7
View File
@@ -482,6 +482,11 @@ async fn cold_load(
// would make the cold-load path immediately
// un-route what it had just placed.
servable: None,
// Same reasoning: the ladder and its availability are
// the neuron's to report, and the next poll (~10s)
// fills it. Guessing here risks advertising a rung
// this host is withholding.
reasoning_budget: Vec::new(),
},
);
}
@@ -539,6 +544,7 @@ async fn profile_to_spec(
// (#283), so a cold-loaded model samples the same way as one
// this host was configured to hold resident.
sampling: profile.sampling.clone(),
preserve_thinking: profile.preserve_thinking,
}
}
@@ -647,6 +653,7 @@ mod tests {
residency_priority: None,
source: source.map(String::from),
sampling: None,
preserve_thinking: None,
limit: None,
cost: None,
capabilities: vec![],
+39 -7
View File
@@ -13,27 +13,56 @@ use cortex_gateway::state::CortexState;
use serde_json::json;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicU32, Ordering};
use tokio::net::TcpListener;
/// Write a `models.toml` with one alias to a unique temp path. Returns
/// the path; the file persists for the test process and gets reaped by
/// the OS at exit. Using $XDG_RUNTIME_DIR fallback for the temp dir
/// keeps the file off shared /tmp on CI without pulling in tempfile.
/// Write a `models.toml` with one alias into a directory of its own
/// under the system temp dir, and prove the bytes landed before
/// returning. The file is reaped by the OS at exit.
///
/// Each call gets its own directory rather than a distinct filename in
/// a shared one. Two things make that worth the extra syscall:
///
/// - The tests in this file run concurrently in one binary and write
/// *different* alias tables. A shared name is one collision away from
/// a test loading another's catalogue.
/// - The failure is silent. `ModelCatalogue::load` maps a missing,
/// unreadable, unparseable **or empty** file onto an empty catalogue,
/// and `resolve_alias` then returns the alias unchanged — so the
/// mismatch surfaces at whatever assertion happens to touch it, tens
/// of lines from the cause. Observed on CI 2026-08-30 as
/// `left: "helexa/small", right: "test-model"` while the same commit
/// passed on its branch and 60/60 locally.
///
/// The read-back is for the same reason: an empty catalogue is
/// indistinguishable from a catalogue that never loaded, so a test may
/// not simply assume its fixture is on disk.
fn write_models_toml(alias: &str, target: &str) -> PathBuf {
static SEQ: AtomicU32 = AtomicU32::new(0);
let contents = format!(
r#"
[aliases]
"{alias}" = "{target}"
"#
);
let mut path = std::env::temp_dir();
let mut dir = std::env::temp_dir();
let pid = std::process::id();
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos();
path.push(format!("cortex-test-models-{pid}-{now}.toml"));
std::fs::write(&path, contents).expect("write temp models.toml");
let seq = SEQ.fetch_add(1, Ordering::Relaxed);
dir.push(format!("cortex-test-models-{pid}-{now}-{seq}"));
std::fs::create_dir_all(&dir).expect("create temp catalogue dir");
let path = dir.join("models.toml");
std::fs::write(&path, &contents).expect("write temp models.toml");
let seen = std::fs::read_to_string(&path).expect("read back temp models.toml");
assert_eq!(
seen,
contents,
"temp catalogue did not round-trip at {}",
path.display()
);
path
}
@@ -81,6 +110,7 @@ async fn test_alias_resolves_in_chat_completions() {
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
}
@@ -168,6 +198,7 @@ async fn test_aliases_surface_in_v1_models() {
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
}
@@ -256,6 +287,7 @@ async fn test_alias_falls_through_for_unmapped_model() {
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
}
+1
View File
@@ -125,6 +125,7 @@ async fn spawn_gateway(neuron_url: &str, entitlements: EntitlementsConfig) -> St
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
}
@@ -100,6 +100,7 @@ async fn spawn_gateway(neuron_url: &str, key: ApiKeyConfig) -> (Arc<CortexState>
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
}
@@ -519,6 +519,7 @@ pub async fn spawn_gateway_with_state(mock_url: &str) -> (Arc<CortexState>, Stri
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
}
+4
View File
@@ -97,6 +97,7 @@ fn loaded(id: &str, age_secs: i64) -> ModelEntry {
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
}
}
@@ -168,6 +169,7 @@ async fn test_evict_lru_model() {
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
node.models.insert(
@@ -182,6 +184,7 @@ async fn test_evict_lru_model() {
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
}
@@ -250,6 +253,7 @@ async fn test_eviction_increments_lifecycle_cycles() {
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
}
+1
View File
@@ -183,6 +183,7 @@ async fn spawn_keyed_gateway(neuron_url: &str, hard_cap: u64) -> String {
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
}
@@ -38,6 +38,7 @@ async fn seed_loaded(fleet: &CortexState, node: &str, in_flight: usize, queue_de
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
n.model_load.insert(
+2
View File
@@ -73,6 +73,7 @@ async fn spawn_metered_gateway(neuron_url: &str) -> (Arc<CortexState>, String) {
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
}
@@ -179,6 +180,7 @@ async fn anonymous_request_records_no_spend() {
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
}
@@ -82,6 +82,7 @@ capabilities = ["text"]
// No cap: effort is expressed to the model through its
// template, not enforced by truncating its thinking.
tokens: None,
unavailable_reason: None,
})
.collect();
node.models.insert(
@@ -105,6 +106,7 @@ capabilities = ["text"]
output_ceiling: 32768,
}),
servable: None,
reasoning_budget: Vec::new(),
},
);
// A model with no derivable limit: the flat #78 fields must be
@@ -121,6 +123,7 @@ capabilities = ["text"]
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
}
+2
View File
@@ -330,6 +330,7 @@ async fn test_poller_removes_stale_models() {
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
node.models.insert(
@@ -344,6 +345,7 @@ async fn test_poller_removes_stale_models() {
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
}
@@ -99,6 +99,7 @@ async fn spawn_gateway(neuron: &str, context: usize) -> String {
output_ceiling: 0,
}),
servable: None,
reasoning_budget: Vec::new(),
},
);
}
@@ -198,6 +198,7 @@ async fn test_recovering_model_returns_503_and_stays_listed() {
reasoning: false,
limit: None,
servable: None,
reasoning_budget: Vec::new(),
},
);
}
@@ -28,6 +28,7 @@ fn entry(id: &str, servable: Option<ModelServability>) -> ModelEntry {
reasoning: false,
limit: None,
servable,
reasoning_budget: Vec::new(),
}
}
+5 -5
View File
@@ -80,7 +80,7 @@ The fastest path: env-var single-endpoint config.
```sh
export HELEXA_ACP_BASE_URL=http://hanzalova.internal:31313/v1
export HELEXA_ACP_MODEL=Qwen/Qwen3.6-27B
export HELEXA_ACP_MODEL=Qwen/Qwen3.8-27B
helexa-acp # speaks ACP over stdin/stdout; not interactive
```
@@ -117,7 +117,7 @@ default_endpoint = "helexa"
name = "helexa"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.6-27B"
default_model = "Qwen/Qwen3.8-27B"
max_tokens = 8192
context_window = 32768
@@ -138,7 +138,7 @@ default_model = "claude-opus-4"
Restart Zed. The model dropdown lists every model from every
configured endpoint with the `endpoint:model` selector
(`helexa:Qwen/Qwen3.6-27B`, `openrouter:anthropic/claude-opus-4`,
(`helexa:Qwen/Qwen3.8-27B`, `openrouter:anthropic/claude-opus-4`,
…). Switch mid-session; the next prompt routes to the new endpoint.
When only one endpoint is configured the prefix is dropped (model
@@ -166,7 +166,7 @@ Copy-pasteable blocks. Mix and match.
name = "helexa"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.6-27B"
default_model = "Qwen/Qwen3.8-27B"
max_tokens = 8192
context_window = 32768
```
@@ -241,7 +241,7 @@ vLLM.
name = "lan"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.6-27B"
default_model = "Qwen/Qwen3.8-27B"
[[endpoints]]
name = "cloud"
+29 -1
View File
@@ -214,11 +214,39 @@ fn default_notify_to() -> String {
}
impl AngelsConfig {
/// A sibling `secrets.toml`, if present, merges after the main file
/// and wins — see [`secrets_path`].
pub fn load(path: impl AsRef<Path>) -> anyhow::Result<Self> {
let path = path.as_ref();
let cfg: Self = Figment::new()
.merge(Toml::file(path.as_ref()))
.merge(Toml::file(path))
.merge(Toml::file(secrets_path(path)))
.merge(Env::prefixed("ANGELS_").split("__"))
.extract()?;
Ok(cfg)
}
}
/// `secrets.toml` beside the main config.
///
/// The main file holds structure and is deployed by CI; this one holds
/// credentials, is written only by the operator, and is never read,
/// written or diffed by the pipeline. Merging it last means the
/// operator's value wins, so a deploy cannot swap a live credential for
/// a placeholder.
///
/// The split exists because config CI cannot write is config no check
/// can defend — `helexa-router.toml` was kept out of git for sitting
/// beside secret-bearing files, and its stale alias then took a node
/// down (2026-08-27). The answer is to separate secrets from structure,
/// not to hand the pipeline the credentials.
///
/// A missing file is not an error: figment treats an absent
/// `Toml::file` as an empty source, so existing single-file deployments
/// keep working untouched.
fn secrets_path(config: &Path) -> std::path::PathBuf {
config
.parent()
.unwrap_or_else(|| Path::new("."))
.join("secrets.toml")
}
+2
View File
@@ -217,6 +217,7 @@ impl TargetClient {
// entry supplies the override; it would matter if this were
// ever used to load a model the host has no config for.
sampling: None,
preserve_thinking: None,
})
}
@@ -281,6 +282,7 @@ impl TargetClient {
cost: None,
tool_call: false,
reasoning: false,
preserve_thinking: None,
// A generic OpenAI endpoint says nothing about
// serviceability, and absent means "no opinion".
servable: None,
+112 -1
View File
@@ -90,7 +90,6 @@ fn router_entry(cortex: &str, e: &CortexModelEntry) -> CortexModelEntry {
max_output_tokens: None,
context_window: None,
context_length: None,
reasoning_budget: Vec::new(),
..e.clone()
}
}
@@ -108,6 +107,7 @@ fn merge_into(acc: &mut CortexModelEntry, cortex: &str, e: &CortexModelEntry) {
}
acc.tool_call |= e.tool_call;
acc.reasoning |= e.reasoning;
acc.reasoning_budget = merge_reasoning_budget(&acc.reasoning_budget, &e.reasoning_budget);
acc.limit = tightest_limit(acc.limit.take(), e.limit.clone());
acc.cost = cheapest_cost(acc.cost.take(), e.cost.clone());
}
@@ -272,3 +272,114 @@ mod tests {
assert_eq!(out[0].max_output_tokens, Some(4096));
}
}
/// Fold two operators' reasoning ladders into the federated view.
///
/// A rung is offered if **any** operator can currently serve it, because
/// the router can route there. Marking it unavailable because one busy
/// operator is withholding it would hide capacity the federation has.
///
/// Conversely a rung only carries an unavailable reason when every
/// operator offering it is withholding it — and the first such reason is
/// kept, since they are diagnostics rather than a set to be merged.
///
/// Order comes from whichever ladder is longer, so the rungs still read
/// in the order a client would climb them.
fn merge_reasoning_budget(
acc: &[cortex_core::harness::ReasoningBudgetRung],
other: &[cortex_core::harness::ReasoningBudgetRung],
) -> Vec<cortex_core::harness::ReasoningBudgetRung> {
if acc.is_empty() {
return other.to_vec();
}
if other.is_empty() {
return acc.to_vec();
}
let (long, short) = if acc.len() >= other.len() {
(acc, other)
} else {
(other, acc)
};
long.iter()
.map(|rung| {
let counterpart = short.iter().find(|r| r.effort == rung.effort);
let Some(counterpart) = counterpart else {
// Only one operator offers this rung at all; take it as-is.
return rung.clone();
};
let available_somewhere =
rung.unavailable_reason.is_none() || counterpart.unavailable_reason.is_none();
cortex_core::harness::ReasoningBudgetRung {
effort: rung.effort.clone(),
tokens: rung.tokens.or(counterpart.tokens),
default: rung.default || counterpart.default,
unavailable_reason: if available_somewhere {
None
} else {
rung.unavailable_reason
.clone()
.or_else(|| counterpart.unavailable_reason.clone())
},
}
})
.collect()
}
#[cfg(test)]
mod reasoning_budget_merge_tests {
use super::*;
use cortex_core::harness::ReasoningBudgetRung;
fn rung(effort: &str, unavailable: Option<&str>) -> ReasoningBudgetRung {
ReasoningBudgetRung {
effort: effort.into(),
tokens: None,
default: false,
unavailable_reason: unavailable.map(str::to_string),
}
}
#[test]
fn a_rung_one_operator_can_serve_is_offered() {
// The federation's whole point: a busy operator withholding xhigh
// must not hide an idle operator that can serve it.
let busy = vec![rung("low", None), rung("xhigh", Some("max_in_flight=8"))];
let idle = vec![rung("low", None), rung("xhigh", None)];
let merged = merge_reasoning_budget(&busy, &idle);
let x = merged.iter().find(|r| r.effort == "xhigh").expect("xhigh");
assert_eq!(x.unavailable_reason, None);
}
#[test]
fn a_rung_every_operator_withholds_stays_unavailable() {
let a = vec![rung("xhigh", Some("max_in_flight=8"))];
let b = vec![rung("xhigh", Some("max_in_flight=4"))];
let merged = merge_reasoning_budget(&a, &b);
assert!(merged[0].unavailable_reason.is_some());
}
#[test]
fn an_empty_ladder_does_not_erase_a_populated_one() {
// A non-reasoning model, or a cortex too old to report rungs,
// must not blank out an operator that does.
let populated = vec![rung("low", None), rung("medium", None)];
assert_eq!(merge_reasoning_budget(&populated, &[]).len(), 2);
assert_eq!(merge_reasoning_budget(&[], &populated).len(), 2);
}
#[test]
fn a_rung_only_one_operator_offers_survives_the_merge() {
let few = vec![rung("low", None)];
let many = vec![rung("low", None), rung("xhigh", None)];
let merged = merge_reasoning_budget(&few, &many);
assert_eq!(merged.len(), 2, "the longer ladder sets the shape");
}
#[test]
fn the_default_flag_survives_from_either_side() {
let mut d = rung("xhigh", None);
d.default = true;
let merged = merge_reasoning_budget(&[rung("xhigh", None)], &[d]);
assert!(merged[0].default);
}
}
+29
View File
@@ -287,15 +287,44 @@ fn default_from_addr() -> String {
impl UpstreamConfig {
/// Load from a TOML file with `UPSTREAM_`-prefixed env overrides
/// (`__` nesting separator).
///
/// A sibling `secrets.toml`, if present, merges after the main file
/// and wins — see [`secrets_path`].
pub fn load(path: impl AsRef<Path>) -> Result<Self, Box<figment::Error>> {
let path = path.as_ref();
Figment::new()
.merge(Toml::file(path))
.merge(Toml::file(secrets_path(path)))
.merge(Env::prefixed("UPSTREAM_").split("__"))
.extract()
.map_err(Box::new)
}
}
/// `secrets.toml` beside the main config.
///
/// The main file holds structure and is deployed by CI; this one holds
/// credentials, is written only by the operator, and is never read,
/// written or diffed by the pipeline. Merging it last means the
/// operator's value wins, so a deploy cannot swap a live credential for
/// a placeholder.
///
/// The split exists because config CI cannot write is config no check
/// can defend — `helexa-router.toml` was kept out of git for sitting
/// beside secret-bearing files, and its stale alias then took a node
/// down (2026-08-27). The answer is to separate secrets from structure,
/// not to hand the pipeline the credentials.
///
/// A missing file is not an error: figment treats an absent
/// `Toml::file` as an empty source, so existing single-file deployments
/// keep working untouched.
fn secrets_path(config: &Path) -> std::path::PathBuf {
config
.parent()
.unwrap_or_else(|| Path::new("."))
.join("secrets.toml")
}
#[cfg(test)]
mod tests {
use super::*;
+16 -4
View File
@@ -53,6 +53,18 @@ fn unique_email() -> String {
format!("u-{}@test.local", uuid::Uuid::new_v4())
}
/// A fixture value that is unique per run.
///
/// Rows keyed on a hash of these (`email_tokens.token_hash`,
/// `api_keys.key_hash`) are uniquely indexed, so a hard-coded literal
/// passes against a virgin database and then fails with `23505` on every
/// run afterwards — the tests only looked deterministic because CI always
/// gets a fresh one. Anything inserted under a unique constraint gets a
/// fresh value here.
fn unique_secret(prefix: &str) -> String {
format!("{prefix}-{}", uuid::Uuid::new_v4())
}
async fn post(url: String, body: Value, bearer: Option<&str>) -> reqwest::Response {
let c = reqwest::Client::new();
let mut req = c.post(url).json(&body);
@@ -85,13 +97,13 @@ async fn verify_endpoint_consumes_token_once() {
.await
.unwrap()
.get("id");
let raw = "verify-raw-token-xyz";
let raw = unique_secret("verify-raw-token");
pool.execute(
sqlx::query(
"INSERT INTO email_tokens (token_hash, user_id, kind, expires_at) \
VALUES ($1, $2, 'verify', now() + interval '1 hour')",
)
.bind(sha256(raw))
.bind(sha256(&raw))
.bind(user_id),
)
.await
@@ -273,13 +285,13 @@ async fn fingerprint_abuse_silently_deactivates_all_no_clue() {
.await
.unwrap()
.get("id");
let raw = "sk-helexa-deactivated-probe";
let raw = unique_secret("sk-helexa-deactivated-probe");
pool.execute(
sqlx::query(
"INSERT INTO api_keys (account_id, key_hash, key_prefix) VALUES ($1, $2, 'sk-helexa-')",
)
.bind(acct)
.bind(sha256(raw)),
.bind(sha256(&raw)),
)
.await
.unwrap();
+32 -9
View File
@@ -2,6 +2,7 @@
use crate::activation::ActivationTracker;
use crate::harness::HarnessRegistry;
use crate::harness::candle::WireSurface;
use crate::harness::candle::{CandleHarness, InferenceError};
use crate::harness::preflight::PreflightError;
use crate::health::HealthCache;
@@ -362,7 +363,7 @@ async fn chat_completions(
if req.stream.unwrap_or(false) {
match candle
.chat_completion_stream_with(req, chat_config, principal)
.chat_completion_stream_with(req, chat_config, principal, WireSurface::Chat)
.await
{
Ok(rx) => {
@@ -381,7 +382,10 @@ async fn chat_completions(
Err(e) => inference_error_response(e),
}
} else {
match candle.chat_completion(req, principal).await {
match candle
.chat_completion(req, principal, WireSurface::Chat)
.await
{
Ok(resp) => Json(resp).into_response(),
Err(e) => inference_error_response(e),
}
@@ -434,7 +438,13 @@ async fn responses(
if stream_requested {
match candle
.responses_stream(chat_req, response_id, message_item_id, principal)
.responses_stream(
chat_req,
response_id,
message_item_id,
principal,
WireSurface::Responses,
)
.await
{
Ok(rx) => {
@@ -458,7 +468,10 @@ async fn responses(
// and translate the result. We don't currently re-tokenise
// to compute usage; the harness returns it via the chat
// response and we pass it through.
match candle.chat_completion(chat_req, principal).await {
match candle
.chat_completion(chat_req, principal, WireSurface::Responses)
.await
{
Ok(chat_resp) => {
// Extract the assistant text (chat completions
// always emits one choice on the candle path).
@@ -486,9 +499,9 @@ async fn responses(
input_tokens: u.prompt_tokens,
output_tokens: u.completion_tokens,
total_tokens: u.prompt_tokens + u.completion_tokens,
// Carry the reasoning sub-count through from the chat
// usage — the non-streaming path now splits off the
// `<think>` span and counts it (see `split_off_reasoning`).
// Carry the reasoning sub-count through from the
// chat usage: the collector counts the `<think>`
// span and returns its text separately.
output_tokens_details: u.completion_tokens_details.as_ref().map(|d| {
OutputTokensDetails {
reasoning_tokens: d.reasoning_tokens,
@@ -502,8 +515,18 @@ async fn responses(
model_id,
message_item_id: mint_message_item_id(),
};
let _ = chat_resp; // make the borrow-checker happy if `text` consumed it
let resp = openai_responses::build_response(&meta, text, finish, usage);
// The model's think block, which the chat collector now
// returns on `message.reasoning_content` (#300). Without
// this the caller is billed for `reasoning_tokens` it
// never sees, and cannot replay the model's own
// reasoning on the next turn.
let reasoning = chat_resp
.choices
.first()
.and_then(|c| c.message.extra.get("reasoning_content"))
.and_then(|v| v.as_str())
.map(str::to_string);
let resp = openai_responses::build_response(&meta, text, finish, usage, reasoning);
Json(resp).into_response()
}
Err(e) => inference_error_response(e),
+1 -1
View File
@@ -94,7 +94,7 @@ pub struct CandleHarnessConfig {
/// configurable because the right values depend on the model and the
/// hardware.
#[serde(default)]
pub reasoning_budget: crate::harness::reasoning_budget::ReasoningBudgetLadder,
pub reasoning_budget: crate::harness::reasoning_budget::ReasoningBudgetConfig,
/// Admission control (#53): bounds the per-model wait queue so a busy
/// model returns a fast, retryable `429`/`503` instead of stalling new
@@ -1,4 +1,4 @@
//! Norm primitives for Qwen3-Next.
//! Norm primitives for Qwen3-Next, shared with `qwen4_exp`.
//!
//! Two reasons we can't reuse `candle_nn::RmsNorm` directly:
//!
@@ -11,13 +11,25 @@
//! introduced the `(1 + w)` form to recover from the zero-init.
//!
//! 2. **Gated variant.** The linear-attention layer post-normalises
//! its output by an RMSNorm *gated* with a per-element SiLU on
//! a sibling `z` projection — fused for numerical reasons (the
//! norm's float32 promotion has to happen before the SiLU
//! its output by an RMSNorm *gated* with a per-element nonlinearity
//! on a sibling `z` projection — fused for numerical reasons (the
//! norm's float32 promotion has to happen before the gate
//! multiply). Not a single existing candle op.
//!
//! Both ops accept inputs in any compute dtype; promotion to f32 for
//! the variance calculation matches the Python reference.
//!
//! `qwen4_exp` (`doc/qwen4_exp-port-spec.md`) reuses both with two
//! additions, because the conventions are otherwise identical:
//!
//! - **`group_size`** on the plain norm, for the 10240-wide norms whose
//! four hidden-size slices normalise independently.
//! - **`OutputGate`** on the gated norm, because that arch sets
//! `output_gate_type = "sigmoid"` where Qwen3.6 falls back to SiLU.
//!
//! Both default to the Qwen3.6 behaviour, so existing callers — the
//! decoder, the attention q/k norms, and the TP analogues in
//! `harness/tp/tp_qwen3_5.rs` — are unaffected.
use anyhow::{Context, Result};
use candle_core::{D, Module, Tensor};
@@ -39,16 +51,52 @@ pub fn l2norm(x: &Tensor, eps: f32) -> candle_core::Result<Tensor> {
/// Qwen3-Next's RMSNorm. Stores the raw weight tensor; forward applies
/// `(1.0 + weight) * x_normed`.
///
/// With `group_size` set, the last axis is split into
/// `size / group_size` groups that are normalised **independently**
/// before the scale is applied across the whole axis — upstream's
/// `Qwen4ExpTextRMSNorm(dim, group_size=...)`. `qwen4_exp` uses this
/// for every 10240-wide norm (`hc_norm`, PLE's three norms,
/// `mtp.pre_fc_norm_hidden`), all with `group_size = hidden_size`, i.e.
/// four independent normalisations rather than one over 10240. A single
/// norm over the full axis compiles, runs, and is quietly wrong — see
/// `doc/qwen4_exp-port-spec.md` §8.
pub struct Qwen3_5RmsNorm {
weight: Tensor,
eps: f32,
size: usize,
/// `None` = one normalisation over the whole last axis (Qwen3.6).
group_size: Option<usize>,
}
impl Qwen3_5RmsNorm {
/// Load `weight` from the ShardedVarBuilder. `vb` should already be
/// `.pp(...)`-ed to the norm's tensor prefix.
pub fn load(vb: &ShardedVarBuilder, size: usize, eps: f64) -> Result<Self> {
Self::load_inner(vb, size, eps, None)
}
/// Grouped variant: normalise each `group_size`-wide slice of the
/// last axis independently.
pub fn load_grouped(
vb: &ShardedVarBuilder,
size: usize,
group_size: usize,
eps: f64,
) -> Result<Self> {
anyhow::ensure!(
group_size > 0 && size.is_multiple_of(group_size),
"grouped RMSNorm needs size ({size}) divisible by group_size ({group_size})"
);
Self::load_inner(vb, size, eps, Some(group_size))
}
fn load_inner(
vb: &ShardedVarBuilder,
size: usize,
eps: f64,
group_size: Option<usize>,
) -> Result<Self> {
let weight = vb
.get(size, "weight")
.with_context(|| format!("load '{}/weight'", vb.prefix()))?;
@@ -56,9 +104,23 @@ impl Qwen3_5RmsNorm {
weight,
eps: eps as f32,
size,
group_size,
})
}
/// Direct constructor — used by unit tests that build a norm
/// without going through a VarBuilder.
#[cfg(test)]
pub(crate) fn from_weight(weight: Tensor, eps: f64, group_size: Option<usize>) -> Self {
let size = weight.dims()[0];
Self {
weight,
eps: eps as f32,
size,
group_size,
}
}
pub fn size(&self) -> usize {
self.size
}
@@ -68,8 +130,28 @@ impl Module for Qwen3_5RmsNorm {
fn forward(&self, x: &Tensor) -> candle_core::Result<Tensor> {
let dtype = x.dtype();
let x_f32 = x.to_dtype(candle_core::DType::F32)?;
let var = x_f32.sqr()?.mean_keepdim(D::Minus1)?;
let normed = x_f32.broadcast_mul(&(var + self.eps as f64)?.sqrt()?.recip()?)?;
// Grouped: reshape the last axis into (groups, group_size),
// normalise over group_size, then flatten back. The scale below
// still applies across the full axis.
let normed = match self.group_size {
Some(gs) => {
let rank = x_f32.rank();
let mut shape = x_f32.dims().to_vec();
let last = shape.pop().expect("tensor has at least one axis");
shape.push(last / gs);
shape.push(gs);
let grouped = x_f32.reshape(shape)?;
let var = grouped.sqr()?.mean_keepdim(D::Minus1)?;
let normed = grouped.broadcast_mul(&(var + self.eps as f64)?.sqrt()?.recip()?)?;
// rank-1 is the groups axis; flattening from there
// restores the original rank.
normed.flatten_from(rank - 1)?
}
None => {
let var = x_f32.sqr()?.mean_keepdim(D::Minus1)?;
x_f32.broadcast_mul(&(var + self.eps as f64)?.sqrt()?.recip()?)?
}
};
// Promote weight to f32 and shift by 1.0 *before* multiplying.
// Doing the (1 + w) operation in fp16 lands at -inf for the
// bottom-of-range weights at load time.
@@ -90,10 +172,49 @@ pub struct Qwen3_5RmsNormGated {
weight: Tensor,
eps: f32,
size: usize,
activation: OutputGate,
}
/// Which nonlinearity the output gate applies. Qwen3.6 leaves
/// `output_gate_type` unset and falls back to `hidden_act` (SiLU);
/// `qwen4_exp` sets it to `sigmoid` explicitly. Inheriting SiLU there
/// would be wrong on all 36 linear-attention layers, and wrong in a way
/// that still produces fluent output.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum OutputGate {
Silu,
Sigmoid,
}
impl OutputGate {
/// Map a config's `output_gate_type` (or `hidden_act` fallback).
pub fn from_name(name: &str) -> Result<Self> {
match name {
"silu" => Ok(Self::Silu),
"sigmoid" => Ok(Self::Sigmoid),
other => anyhow::bail!("unsupported output_gate_type '{other}'"),
}
}
fn apply(self, gate: &Tensor) -> candle_core::Result<Tensor> {
match self {
Self::Silu => candle_nn::ops::silu(gate),
Self::Sigmoid => candle_nn::ops::sigmoid(gate),
}
}
}
impl Qwen3_5RmsNormGated {
pub fn load(vb: &ShardedVarBuilder, size: usize, eps: f64) -> Result<Self> {
Self::load_with_gate(vb, size, eps, OutputGate::Silu)
}
pub fn load_with_gate(
vb: &ShardedVarBuilder,
size: usize,
eps: f64,
activation: OutputGate,
) -> Result<Self> {
let weight = vb
.get(size, "weight")
.with_context(|| format!("load '{}/weight'", vb.prefix()))?;
@@ -101,6 +222,7 @@ impl Qwen3_5RmsNormGated {
weight,
eps: eps as f32,
size,
activation,
})
}
@@ -113,6 +235,7 @@ impl Qwen3_5RmsNormGated {
weight,
eps: eps as f32,
size,
activation: OutputGate::Silu,
}
}
@@ -128,11 +251,11 @@ impl Qwen3_5RmsNormGated {
let normed = x_f32.broadcast_mul(&(var + self.eps as f64)?.sqrt()?.recip()?)?;
let w = self.weight.to_dtype(candle_core::DType::F32)?;
let out = normed.broadcast_mul(&w)?;
// SiLU on the float32 gate, multiply back into the normed
// Activate the float32 gate, multiply back into the normed
// tensor, then cast to the model dtype.
let g = gate.to_dtype(candle_core::DType::F32)?;
let silu_gate = candle_nn::ops::silu(&g)?;
(out * silu_gate)?.to_dtype(dtype)
let activated = self.activation.apply(&g)?;
(out * activated)?.to_dtype(dtype)
}
}
@@ -158,4 +281,98 @@ mod tests {
let v: Vec<f32> = out.to_vec1().unwrap();
assert!(v.iter().all(|x| x.is_finite()));
}
/// Grouped norm over `[1, 1, 4]` with `group_size = 2`: each pair is
/// normalised on its own RMS, so the second pair — an order of
/// magnitude larger — comes back at the same scale as the first.
/// Ungrouped, the shared RMS would leave the pairs far apart, which
/// is exactly the silent failure this guards.
#[test]
fn grouped_rmsnorm_normalises_each_group_independently() {
let dev = Device::Cpu;
let x = Tensor::new(&[[[3.0_f32, 4.0, 30.0, 40.0]]], &dev).unwrap();
let w = Tensor::zeros(4, candle_core::DType::F32, &dev).unwrap();
let grouped = Qwen3_5RmsNorm::from_weight(w.clone(), 1e-6, Some(2));
let got: Vec<f32> = grouped
.forward(&x)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
// RMS([3,4]) = sqrt(12.5); RMS([30,40]) = sqrt(1250).
let a = (12.5_f32).sqrt();
let b = (1250.0_f32).sqrt();
let want = [3.0 / a, 4.0 / a, 30.0 / b, 40.0 / b];
for (g, w) in got.iter().zip(want.iter()) {
assert!((g - w).abs() < 1e-4, "grouped: got {got:?} want {want:?}");
}
// Both groups land at the same magnitude — the whole point.
assert!((got[0] - got[2]).abs() < 1e-4);
}
#[test]
fn ungrouped_rmsnorm_is_unchanged_by_the_group_option() {
let dev = Device::Cpu;
let x = Tensor::new(&[[[3.0_f32, 4.0, 30.0, 40.0]]], &dev).unwrap();
let w = Tensor::zeros(4, candle_core::DType::F32, &dev).unwrap();
let plain = Qwen3_5RmsNorm::from_weight(w, 1e-6, None);
let got: Vec<f32> = plain
.forward(&x)
.unwrap()
.flatten_all()
.unwrap()
.to_vec1()
.unwrap();
// One RMS over all four: sqrt((9+16+900+1600)/4).
let rms = ((9.0_f32 + 16.0 + 900.0 + 1600.0) / 4.0).sqrt();
let want = [3.0 / rms, 4.0 / rms, 30.0 / rms, 40.0 / rms];
for (g, w) in got.iter().zip(want.iter()) {
assert!((g - w).abs() < 1e-4, "plain: got {got:?} want {want:?}");
}
}
/// The grouped reshape must not disturb the leading axes.
#[test]
fn grouped_rmsnorm_preserves_shape() {
let dev = Device::Cpu;
let x = Tensor::rand(0f32, 1f32, (2, 3, 8), &dev).unwrap();
let w = Tensor::zeros(8, candle_core::DType::F32, &dev).unwrap();
let n = Qwen3_5RmsNorm::from_weight(w, 1e-6, Some(4));
assert_eq!(n.forward(&x).unwrap().dims(), &[2, 3, 8]);
}
#[test]
fn output_gate_sigmoid_differs_from_silu() {
let dev = Device::Cpu;
let g = Tensor::new(&[1.0_f32, -2.0, 0.5], &dev).unwrap();
let sig: Vec<f32> = OutputGate::Sigmoid.apply(&g).unwrap().to_vec1().unwrap();
let silu: Vec<f32> = OutputGate::Silu.apply(&g).unwrap().to_vec1().unwrap();
// sigmoid(1) = 0.7311; silu(1) = 1 * 0.7311 = 0.7311 — equal at
// x = 1, so check a point where they must diverge.
assert!(
(sig[1] - 0.11920292).abs() < 1e-5,
"sigmoid(-2) = {}",
sig[1]
);
assert!(
(silu[1] + 0.23840584).abs() < 1e-5,
"silu(-2) = {}",
silu[1]
);
assert!((sig[1] - silu[1]).abs() > 0.3);
}
#[test]
fn output_gate_rejects_unknown_names() {
assert!(OutputGate::from_name("silu").is_ok());
assert!(OutputGate::from_name("sigmoid").is_ok());
assert!(OutputGate::from_name("gelu").is_err());
}
}
+497 -76
View File
@@ -104,7 +104,7 @@ pub struct CandleHarness {
context_limit_cfg: crate::config::ContextLimitConfig,
/// Effort-level → reasoning-token rungs (#223), resolved per request
/// so a caller can bound how long the model thinks.
reasoning_budget_cfg: super::reasoning_budget::ReasoningBudgetLadder,
reasoning_budget_cfg: super::reasoning_budget::ReasoningBudgetConfig,
/// Admission-control settings (#53), used to build each loaded model's
/// [`super::admission::AdmissionController`] at load time.
admission_cfg: crate::config::AdmissionConfig,
@@ -429,6 +429,17 @@ impl LoadedHandle {
}
}
/// The operator's `preserve_thinking` for this model, advertised so
/// an A/B run can be stamped with the value it ran under.
pub fn preserve_thinking(&self) -> Option<bool> {
match self {
LoadedHandle::Single(m) => m.preserve_thinking,
#[cfg(feature = "cuda")]
LoadedHandle::Tp(m) => m.preserve_thinking,
LoadedHandle::Image(_) => None,
}
}
/// Effort rungs the model's template accepts (#290).
pub fn reasoning_efforts(&self) -> super::reasoning_effort::SupportedEfforts {
match self {
@@ -502,11 +513,21 @@ impl LoadedHandle {
LoadedHandle::Image(_) => return None,
};
let rate = rate.unwrap_or(cfg.bootstrap_prefill_tok_per_sec);
// The KV budget admission will actually grant. Advertising past
// it hands a client a context it can only discover is a lie by
// hitting `413 prompt_too_long_for_vram` mid-session.
let kv_budget_mb = match self {
LoadedHandle::Single(m) => m.admission.kv_budget_mb(),
#[cfg(feature = "cuda")]
LoadedHandle::Tp(m) => m.admission.kv_budget_mb(),
LoadedHandle::Image(_) => unreachable!("derived_limit returned above"),
};
let limit = super::context_limit::derive_limit(
&profile,
free_mb,
rate,
max_prompt_tokens_clamp(),
kv_budget_mb,
cfg,
);
// Refresh the request-path enforcement cap (#67 phase 5b): the
@@ -593,12 +614,10 @@ impl LoadedImageModel {
/// the four request-handling paths.
/// The reasoning-token budget this request asked for, if any (#223).
///
/// Three inlets converge on `extra.reasoning`: Anthropic's
/// `thinking.budget_tokens` (normalised in translation), the explicit
/// `reasoning.max_tokens` extension, and OpenAI's `reasoning.effort`
/// ladder resolved against the operator's configured rungs. An explicit
/// token count wins over a ladder rung — it is the more specific thing
/// the caller reached for.
/// Two inlets converge on `extra.reasoning`: Anthropic's
/// `thinking.budget_tokens` (normalised in translation) and the explicit
/// `reasoning.max_tokens` extension. `reasoning.effort` is deliberately
/// **not** one of them — see below.
///
/// An explicit `reasoning.max_tokens` wins. Failing that, the budget is
/// the caller's own output budget less `answer_reserve_tokens`, so an
@@ -606,7 +625,9 @@ impl LoadedImageModel {
/// gets an answer rather than a think block that eats the lot.
///
/// Effort levels used to be mapped onto a token ladder here and enforced
/// by force-injecting `</think>` at the limit. That made effort mean two
/// by force-injecting `</think>` at the limit. That ladder was removed
/// outright on 2026-08-30 — it had been unreachable since #290 and read
/// as though effort still set a budget. That made effort mean two
/// incompatible things at once: a prompt instruction to the model, and a
/// guillotine applied to it. On `Qwen/Qwen3.8-27B` the model was told to
/// reason at `xhigh` (the template default, because the caller's effort
@@ -626,7 +647,7 @@ impl LoadedImageModel {
/// the output cap bite than to emit a stub).
fn requested_reasoning_budget(
request: &ChatCompletionRequest,
ladder: &super::reasoning_budget::ReasoningBudgetLadder,
cfg: &super::reasoning_budget::ReasoningBudgetConfig,
max_new: usize,
) -> Option<usize> {
if let Some(explicit) = request
@@ -638,7 +659,7 @@ fn requested_reasoning_budget(
{
return Some(explicit as usize);
}
let reserve = ladder.answer_reserve_tokens;
let reserve = cfg.answer_reserve_tokens;
if reserve == 0 || reserve >= max_new {
return None;
}
@@ -653,6 +674,36 @@ fn requested_sampling(request: &ChatCompletionRequest) -> super::sampling::Reque
seed: request.seed,
repeat_penalty: request.repetition_penalty,
repeat_last_n: request.repeat_last_n,
presence_penalty: request.presence_penalty,
frequency_penalty: request.frequency_penalty,
}
}
/// Which API surface a request arrived on.
///
/// Carried into the tracing span because neuron translates
/// `/v1/responses` into a chat request at the door
/// (`openai_responses::request_to_chat`) and every line after that
/// point — spans, finish lines, rendered prompts — is emitted by the
/// shared chat path. Without this the journal reads as though every
/// client used `/v1/chat/completions`, which is not a missing detail
/// but an actively wrong one: it says something false about the caller.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum WireSurface {
/// `POST /v1/chat/completions`
Chat,
/// `POST /v1/responses`
Responses,
}
impl WireSurface {
/// The value recorded as `wire=` on every span, matching the names
/// cortex already logs via `log_inbound` so the two journals join.
pub fn as_str(self) -> &'static str {
match self {
Self::Chat => "openai-chat",
Self::Responses => "openai-responses",
}
}
}
@@ -726,6 +777,9 @@ pub struct LoadedModel {
/// no effort control. Drives both what `/models` advertises and
/// which value we forward on a request.
pub reasoning_efforts: super::reasoning_effort::SupportedEfforts,
/// Operator's `preserve_thinking` for this model, or `None` to let
/// the template decide (#H7 A/B). See `ModelSpec::preserve_thinking`.
pub preserve_thinking: Option<bool>,
/// Open/close token IDs for the model's tool-call marker
/// pair (`<tool_call>` / `</tool_call>` on Qwen3-Coder / Hermes
/// / DeepSeek / gpt-oss). `None` for models that don't emit
@@ -884,6 +938,9 @@ pub struct TpLoadedModel {
/// no effort control. Drives both what `/models` advertises and
/// which value we forward on a request.
pub reasoning_efforts: super::reasoning_effort::SupportedEfforts,
/// Operator's `preserve_thinking` for this model, or `None` to let
/// the template decide (#H7 A/B). See `ModelSpec::preserve_thinking`.
pub preserve_thinking: Option<bool>,
/// Same shape as [`LoadedModel::tool_call_tokens`].
pub tool_call_tokens: Option<ToolCallTokenPair>,
/// Same shape as [`LoadedModel::chat_template`].
@@ -2012,6 +2069,45 @@ async fn acquire_pool_lock<'a>(
}
}
/// Subtract OpenAI-semantics presence/frequency penalties from `logits`.
///
/// `logit -= frequency * count + presence * (count > 0)`, scored over
/// the whole generated sequence. Both default to `0.0`, which is
/// exactly a no-op — an operator or caller who sets neither sees the
/// previous behaviour, and the tensor is returned untouched rather than
/// round-tripped through the CPU.
///
/// Deliberately additive on logits, not multiplicative like
/// `repeat_penalty`: that is what the OpenAI parameters mean, and a
/// client that sends `presence_penalty: 1.5` expecting OpenAI behaviour
/// must get it.
fn apply_sequence_penalties(
logits: &Tensor,
history: &[u32],
presence: f32,
frequency: f32,
) -> Result<Tensor> {
if (presence == 0.0 && frequency == 0.0) || history.is_empty() {
return Ok(logits.clone());
}
let device = logits.device().clone();
let mut values = logits.to_dtype(candle_core::DType::F32)?.to_vec1::<f32>()?;
let mut counts: std::collections::HashMap<u32, f32> = std::collections::HashMap::new();
for t in history {
*counts.entry(*t).or_insert(0.0) += 1.0;
}
for (token, count) in counts {
let Some(slot) = values.get_mut(token as usize) else {
// A history token outside the vocabulary would be a bug
// elsewhere; skip it rather than panicking mid-generation.
continue;
};
*slot -= frequency * count + presence;
}
let len = values.len();
Ok(Tensor::from_vec(values, len, &device)?)
}
/// Apply the repetition penalty (if any) to the prediction logits and
/// then sample. Centralises the prefill / generation-loop call sites
/// so they share identical sampling behaviour.
@@ -2040,6 +2136,19 @@ pub(crate) fn sample_with_penalty(
&history[start..],
)?
};
// Sequence-wide penalties, over everything generated so far rather
// than a trailing window. This is what a reasoning model needs: at
// `repeat_last_n = 64` inside a 28,672-token think block, a passage
// restated 5,000 tokens later is invisible to the windowed penalty,
// and Qwen's own guidance for the endless-repetition failure their
// models show during long thinking is presence_penalty (0..2, 1.5
// when severe) — not a wider window.
let penalised = apply_sequence_penalties(
&penalised,
history,
sampling.presence_penalty,
sampling.frequency_penalty,
)?;
let sampled = logits_processor.sample(&penalised)?;
// The reasoning budget governs which token is *emitted*, not which
// is sampled: over budget inside a think block, the model gets its
@@ -2808,8 +2917,9 @@ impl CandleHarness {
&self,
request: ChatCompletionRequest,
principal: Option<String>,
wire: WireSurface,
) -> Result<ChatCompletionResponse, InferenceError> {
let stream = self.inference_stream(request, principal).await?;
let stream = self.inference_stream(request, principal, wire).await?;
wire_chat::collect_chat_completion(
stream.events,
stream.id,
@@ -2836,11 +2946,13 @@ impl CandleHarness {
&self,
request: ChatCompletionRequest,
principal: Option<String>,
wire: WireSurface,
) -> Result<mpsc::Receiver<ChatCompletionChunk>, InferenceError> {
self.chat_completion_stream_with(
request,
wire_chat::ChatProjectionConfig::default(),
principal,
wire,
)
.await
}
@@ -2854,8 +2966,9 @@ impl CandleHarness {
request: ChatCompletionRequest,
mut config: wire_chat::ChatProjectionConfig,
principal: Option<String>,
wire: WireSurface,
) -> Result<mpsc::Receiver<ChatCompletionChunk>, InferenceError> {
let stream = self.inference_stream(request, principal).await?;
let stream = self.inference_stream(request, principal, wire).await?;
// Fill in the model's reasoning markers if the caller
// didn't pre-populate them — they're a property of the
// loaded model (which the HTTP handler doesn't reach into
@@ -2883,9 +2996,10 @@ impl CandleHarness {
response_id: String,
message_item_id: String,
principal: Option<String>,
wire: WireSurface,
) -> Result<mpsc::Receiver<crate::wire::openai_responses::ResponseStreamFrame>, InferenceError>
{
let stream = self.inference_stream(request, principal).await?;
let stream = self.inference_stream(request, principal, wire).await?;
let meta = crate::wire::openai_responses::ResponseMeta {
response_id,
created_at: stream.created,
@@ -2907,6 +3021,7 @@ impl CandleHarness {
&self,
request: ChatCompletionRequest,
principal: Option<String>,
wire: WireSurface,
) -> Result<InferenceStream, InferenceError> {
let handle = {
let models = self.models.read().await;
@@ -2931,7 +3046,7 @@ impl CandleHarness {
LoadedHandle::Single(m) => m,
#[cfg(feature = "cuda")]
LoadedHandle::Tp(m) => {
return self.inference_tp_stream(m, request, principal).await;
return self.inference_tp_stream(m, request, principal, wire).await;
}
LoadedHandle::Image(_) => {
return Err(InferenceError::WrongModality {
@@ -2944,6 +3059,7 @@ impl CandleHarness {
loaded.chat_template.as_deref(),
&request,
&loaded.reasoning_efforts,
loaded.preserve_thinking,
)?;
let encoding = loaded
.tokenizer
@@ -3068,7 +3184,12 @@ impl CandleHarness {
// executor so we capture the span explicitly and re-enter it
// inside the closure to keep the req_id on every emitted line.
let req_id = new_req_id();
let span = tracing::info_span!("chat_stream", req_id = %req_id, model = %model_id);
let span = tracing::info_span!(
"chat_stream",
req_id = %req_id,
model = %model_id,
wire = wire.as_str()
);
let prompt_len = prompt_tokens.len();
let req_start = std::time::Instant::now();
// Cloned `Arc<LoadedModel>` so the spawned task can mark the
@@ -3517,6 +3638,7 @@ impl Harness for CandleHarness {
cost: None,
tool_call: h.has_tool_call(),
reasoning: h.has_reasoning(),
preserve_thinking: h.preserve_thinking(),
servable: h.servability(&self.context_limit_cfg),
// The rungs this model's own template accepts (#290),
// discovered at load rather than invented. Advertising a
@@ -3525,20 +3647,33 @@ impl Harness for CandleHarness {
// strongest setting. (Whether a given client reads this
// is another matter — pi-ai does not; see
// `reasoning_effort`'s module doc.)
reasoning_budget: h
.reasoning_efforts()
.levels
.iter()
.map(|effort| cortex_core::harness::ReasoningBudgetRung {
default: h.reasoning_efforts().default.as_deref() == Some(effort.as_str()),
effort: effort.clone(),
// No cap: effort reaches the model through its
// template, and how many tokens it then spends
// is the model's business. Enforcing effort by
// truncation is the defect #290 fixed.
tokens: None,
})
.collect(),
reasoning_budget: {
let efforts = h.reasoning_efforts();
let slots = h.capacity().0;
// Withdraw the deepest rung on a busy node, but never
// a model's only rung — the rule steers a choice, and
// with one option there is no choice to steer.
let withhold = (efforts.levels.len() > 1
&& !super::reasoning_budget::long_rung_available(slots))
.then(|| super::reasoning_budget::longest_rung(&efforts.levels))
.flatten()
.cloned();
efforts
.levels
.iter()
.map(|effort| cortex_core::harness::ReasoningBudgetRung {
default: efforts.default.as_deref() == Some(effort.as_str()),
effort: effort.clone(),
// No cap: effort reaches the model through its
// template, and how many tokens it then spends
// is the model's business. Enforcing effort by
// truncation is the defect #290 fixed.
tokens: None,
unavailable_reason: (withhold.as_deref() == Some(effort.as_str()))
.then(|| format!("max_in_flight={slots}")),
})
.collect()
},
});
}
// Models mid-recovery whose registry slot is absent (the
@@ -3556,6 +3691,7 @@ impl Harness for CandleHarness {
capabilities: snap.capabilities.clone(),
limit: None,
cost: None,
preserve_thinking: None,
tool_call: false, // snapshot doesn't carry these; safe default
reasoning: false,
// Mid-recovery: the handle is gone, so there is
@@ -3846,10 +3982,19 @@ impl Harness for CandleHarness {
.transpose()
.unwrap_or_default()
.unwrap_or_default();
// Stamp the A/B arm in the journal at load, so a session
// reviewed later can be attributed to the value it ran under
// without reconstructing deploy history.
tracing::info!(
model = %spec.model_id,
preserve_thinking = ?spec.preserve_thinking,
"reasoning replay: preserve_thinking for this load (None = the template's own default)"
);
let loaded = Arc::new(LoadedModel {
model_id: spec.model_id.clone(),
generation_defaults,
reasoning_efforts,
preserve_thinking: spec.preserve_thinking,
arch: arch_local,
tokenizer,
device,
@@ -4499,10 +4644,19 @@ impl CandleHarness {
.transpose()
.unwrap_or_default()
.unwrap_or_default();
// Stamp the A/B arm in the journal at load, so a session
// reviewed later can be attributed to the value it ran under
// without reconstructing deploy history.
tracing::info!(
model = %spec.model_id,
preserve_thinking = ?spec.preserve_thinking,
"reasoning replay: preserve_thinking for this load (None = the template's own default)"
);
let tp_loaded = StdArc::new(TpLoadedModel {
model_id: spec.model_id.clone(),
generation_defaults,
reasoning_efforts,
preserve_thinking: spec.preserve_thinking,
tokenizer,
devices: devices.clone(),
pool: StdArc::new(TMutex::new(pool)),
@@ -4622,6 +4776,7 @@ impl CandleHarness {
tp: Arc<TpLoadedModel>,
request: ChatCompletionRequest,
principal: Option<String>,
wire: WireSurface,
) -> Result<InferenceStream, InferenceError> {
if tp.poisoned.load(Ordering::Acquire) {
return Err(self.trigger_recovery(&request.model).await);
@@ -4639,8 +4794,12 @@ impl CandleHarness {
});
}
let prompt =
build_prompt_for_request(tp.chat_template.as_deref(), &request, &tp.reasoning_efforts)?;
let prompt = build_prompt_for_request(
tp.chat_template.as_deref(),
&request,
&tp.reasoning_efforts,
tp.preserve_thinking,
)?;
let encoding = tp
.tokenizer
.encode(prompt.as_str(), true)
@@ -4769,14 +4928,19 @@ impl CandleHarness {
// of this inference; concurrent requests against the same TP
// model serialise behind it.
//
// Tagged with the same req_id span as the non-streaming path
// so the journal can be reconstructed regardless of which API
// surface the client hit.
// Tagged with the same req_id span as the non-streaming path,
// and with the surface the request actually arrived on. Both
// matter: neuron translates `/v1/responses` into a chat request
// at the door, so without `wire` every line here reads as
// though the caller used `/v1/chat/completions`. The value
// matches what cortex logs via `log_inbound`, so the two
// journals join on it.
let req_id = new_req_id();
let span = tracing::info_span!(
"tp_chat_stream",
req_id = %req_id,
model = %model_id
model = %model_id,
wire = wire.as_str()
);
let req_start = std::time::Instant::now();
let (vram_free_mb, vram_total_mb) = tp.query_vram().await;
@@ -5612,8 +5776,33 @@ fn parse_qwen_xml_tool_call(
schemas: &ToolSchemas,
) -> Option<(String, String, String)> {
// `<function=NAME>` — name runs to the closing `>` or end of line.
let after_fn = body.split("<function=").nth(1)?;
let name = after_fn.split(['>', '\n']).next()?.trim().to_string();
//
// Falls back to a recovery for a malformation seen in production
// (2026-08-29): the model tagged the function itself as a parameter,
// emitting `<tool_call><parameter=web_search><parameter=query>…`
// with no `<function=` at all. Strict parsing returned None, so the
// whole block fell through into `content` and the user's answer was
// a screenful of raw markup.
//
// The recovery is deliberately narrow: a leading `<parameter=X>` is
// only promoted when X is a tool this request actually offered. That
// is what keeps it from inventing function names out of ordinary
// parameters — an unknown name still returns None, and the caller
// still owes the user something better than markup.
let (name, after_fn, skip_param) = match body.split_once("<function=") {
Some((_, after)) => {
let n = after.split(['>', '\n']).next()?.trim().to_string();
(n, after, None)
}
None => {
let after = body.split_once("<parameter=").map(|(_, a)| a)?;
let n = after.split(['>', '\n']).next()?.trim().to_string();
if !schemas.contains_key(&n) {
return None;
}
(n.clone(), after, Some(n))
}
};
if name.is_empty() {
return None;
}
@@ -5636,7 +5825,8 @@ fn parse_qwen_xml_tool_call(
.to_string();
let after_gt = seg.split_once('>').map(|(_, after)| after).unwrap_or("");
let value_raw = after_gt.split("</parameter>").next().unwrap_or("").trim();
if !key.is_empty() {
// Skip the pseudo-parameter that was actually the function name.
if !key.is_empty() && Some(&key) != skip_param.as_ref() {
let declared = param_types.and_then(|m| m.get(&key)).map(String::as_str);
args.insert(key, coerce_param_value(value_raw, declared));
}
@@ -5904,10 +6094,44 @@ fn apply_effort_kwarg(
serde_json::Value::Object(obj)
}
/// Apply the operator's `preserve_thinking`, if they set one.
///
/// Precedence is request > operator > the model's own template default.
/// A caller that sends `chat_template_kwargs.preserve_thinking` has
/// said something specific and keeps it; an operator who sets nothing
/// changes nothing, so the template's `|default` still governs.
///
/// Deliberately *not* defaulted by us. On `Qwen/Qwen3.8-27B` the
/// template preserves every turn's reasoning unless told otherwise —
/// its authors' choice — and `false` narrows replay to the turn in
/// progress. Both are defensible and the difference is measurable, so
/// this exists to run that measurement rather than to encode a guess.
fn apply_preserve_thinking_kwarg(
kwargs: serde_json::Value,
configured: Option<bool>,
) -> serde_json::Value {
let Some(configured) = configured else {
return kwargs;
};
if kwargs.get("preserve_thinking").is_some() {
return kwargs;
}
let mut obj = match kwargs {
serde_json::Value::Object(o) => o,
_ => serde_json::Map::new(),
};
obj.insert(
"preserve_thinking".into(),
serde_json::Value::Bool(configured),
);
serde_json::Value::Object(obj)
}
fn build_prompt_for_request(
chat_template: Option<&str>,
request: &ChatCompletionRequest,
efforts: &super::reasoning_effort::SupportedEfforts,
preserve_thinking: Option<bool>,
) -> Result<String, InferenceError> {
if !super::chat_template::chat_templates_enabled() {
return Ok(format_qwen3_prompt(&request.messages));
@@ -5932,6 +6156,9 @@ fn build_prompt_for_request(
// request was instructed to think maximally and then truncated at
// whatever budget the caller had actually asked for.
let kwargs = apply_effort_kwarg(kwargs, request, efforts);
// Whether prior turns keep their think blocks. Unset unless the
// operator chose, so the template's own default stands.
let kwargs = apply_preserve_thinking_kwarg(kwargs, preserve_thinking);
let tools = request
.extra
.get("tools")
@@ -7321,6 +7548,8 @@ mod tests {
seed: None,
repetition_penalty: None,
repeat_last_n: None,
presence_penalty: None,
frequency_penalty: None,
max_tokens: None,
max_completion_tokens: None,
stream: None,
@@ -7384,6 +7613,49 @@ mod tests {
assert_eq!(effort_in(&out), None);
}
/// Unset by default: the model's authors chose the template's
/// default and we do not override it fleet-wide on a hunch.
#[test]
fn no_operator_setting_leaves_preserve_thinking_alone() {
let out = apply_preserve_thinking_kwarg(serde_json::Value::Null, None);
assert!(out.get("preserve_thinking").is_none());
assert!(out.is_null(), "an untouched Null must stay Null");
}
/// The A/B knob: an operator can narrow replay to the turn in
/// progress, or pin full replay, without a rebuild.
#[test]
fn an_operator_setting_reaches_the_template() {
for configured in [true, false] {
let out = apply_preserve_thinking_kwarg(serde_json::Value::Null, Some(configured));
assert_eq!(out["preserve_thinking"], serde_json::json!(configured));
}
}
/// A caller that named it has said something specific; the
/// operator's default must not overwrite it.
#[test]
fn a_requested_preserve_thinking_outranks_the_operator() {
let out = apply_preserve_thinking_kwarg(
serde_json::json!({ "preserve_thinking": true }),
Some(false),
);
assert_eq!(out["preserve_thinking"], serde_json::json!(true));
}
/// It must not clobber the effort kwarg sharing the same object —
/// both are applied to one `chat_template_kwargs`.
#[test]
fn it_preserves_kwargs_already_present() {
let out = apply_preserve_thinking_kwarg(
serde_json::json!({ "reasoning_effort": "medium", "custom": 1 }),
Some(false),
);
assert_eq!(out["reasoning_effort"], "medium");
assert_eq!(out["custom"], 1);
assert_eq!(out["preserve_thinking"], serde_json::json!(false));
}
/// A model whose template has no effort control must be left alone.
#[test]
fn a_model_without_rungs_gets_no_effort_kwarg() {
@@ -7395,35 +7667,62 @@ mod tests {
assert_eq!(effort_in(&out), None);
}
/// The wire surface must serialise as the string cortex logs, so a
/// journal search joins the two.
///
/// neuron translates `/v1/responses` into a chat request at the
/// door, and every line after that is emitted by the shared chat
/// path. Before `wire` was recorded the journal showed
/// `chat_completion (stream): done` for every request whatever the
/// surface — worse than omitting the detail, because it asserts
/// something false about the caller. It cost a real investigation
/// into why a Responses-configured client "was using chat
/// completions". It was not.
#[test]
fn the_wire_surface_matches_what_cortex_logs() {
// These strings are load-bearing: `cortex-gateway`'s
// `log_inbound` emits the same two, so `journalctl | grep
// openai-responses` spans both services.
assert_eq!(WireSurface::Chat.as_str(), "openai-chat");
assert_eq!(WireSurface::Responses.as_str(), "openai-responses");
}
/// Effort must not select a *rung's* budget — that was the
/// guillotine (#290): the model was told to reason at `xhigh` and
/// cut off at `medium`'s 12,288. The reserve below is not that; it
/// does not vary with effort.
#[test]
fn effort_no_longer_selects_a_rungs_token_budget() {
let ladder = super::super::reasoning_budget::ReasoningBudgetLadder::default();
let medium = requested_reasoning_budget(&req_with_effort(Some("medium")), &ladder, 32_768);
let xhigh = requested_reasoning_budget(&req_with_effort(Some("xhigh")), &ladder, 32_768);
let cfg = super::super::reasoning_budget::ReasoningBudgetConfig::default();
let medium = requested_reasoning_budget(&req_with_effort(Some("medium")), &cfg, 32_768);
let xhigh = requested_reasoning_budget(&req_with_effort(Some("xhigh")), &cfg, 32_768);
assert_eq!(medium, xhigh, "the bound must not depend on effort");
// 12,288 was `medium`'s rung in the cfg this replaced. The
// literal is the point: if a future change reintroduces a
// per-effort budget, this is the number it would land on.
assert_ne!(
medium,
Some(ladder.medium),
"medium must not become a {}-token guillotine",
ladder.medium
Some(12_288),
"medium must not become a 12288-token guillotine"
);
assert_eq!(
medium,
Some(32_768 - cfg.answer_reserve_tokens),
"the bound is the caller's budget less the answer reserve"
);
}
/// The regression this reserve exists to prevent (#223, reopened by
/// #290): an OpenAI-shaped client cannot send `reasoning.max_tokens`,
/// so removing the effort→ladder mapping left it with no bound at
/// so removing the effort→cfg mapping left it with no bound at
/// all. Qwen3.8-27B then spent the entire 32,768-token budget
/// thinking and returned no answer — twice, on real sessions.
#[test]
fn a_caller_that_names_no_budget_still_keeps_room_to_answer() {
let ladder = super::super::reasoning_budget::ReasoningBudgetLadder::default();
let budget = requested_reasoning_budget(&req_with_effort(Some("xhigh")), &ladder, 32_768)
let cfg = super::super::reasoning_budget::ReasoningBudgetConfig::default();
let budget = requested_reasoning_budget(&req_with_effort(Some("xhigh")), &cfg, 32_768)
.expect("an unbounded think block is what broke");
assert_eq!(budget, 32_768 - ladder.answer_reserve_tokens);
assert_eq!(budget, 32_768 - cfg.answer_reserve_tokens);
assert!(
budget < 32_768,
"some of the caller's own budget must survive for the answer"
@@ -7434,12 +7733,12 @@ mod tests {
/// for, so it wins over the reserve in both directions.
#[test]
fn an_explicit_max_tokens_still_wins() {
let ladder = super::super::reasoning_budget::ReasoningBudgetLadder::default();
let cfg = super::super::reasoning_budget::ReasoningBudgetConfig::default();
let mut req = req_with_effort(None);
if let serde_json::Value::Object(o) = &mut req.extra {
o.insert("reasoning".into(), serde_json::json!({ "max_tokens": 500 }));
}
assert_eq!(requested_reasoning_budget(&req, &ladder, 32_768), Some(500));
assert_eq!(requested_reasoning_budget(&req, &cfg, 32_768), Some(500));
// Above the reserve-derived value, too — not clamped to it.
if let serde_json::Value::Object(o) = &mut req.extra {
@@ -7448,30 +7747,23 @@ mod tests {
serde_json::json!({ "max_tokens": 31_000 }),
);
}
assert_eq!(
requested_reasoning_budget(&req, &ladder, 32_768),
Some(31_000)
);
assert_eq!(requested_reasoning_budget(&req, &cfg, 32_768), Some(31_000));
}
/// Reserving from a budget too small to think in would emit a stub.
/// Better to think freely and let the output cap bite.
#[test]
fn a_budget_smaller_than_the_reserve_stays_unbounded() {
let ladder = super::super::reasoning_budget::ReasoningBudgetLadder::default();
for max_new in [0, 1, ladder.answer_reserve_tokens] {
let cfg = super::super::reasoning_budget::ReasoningBudgetConfig::default();
for max_new in [0, 1, cfg.answer_reserve_tokens] {
assert_eq!(
requested_reasoning_budget(&req_with_effort(None), &ladder, max_new),
requested_reasoning_budget(&req_with_effort(None), &cfg, max_new),
None,
"max_new={max_new} must not produce a bound"
);
}
assert_eq!(
requested_reasoning_budget(
&req_with_effort(None),
&ladder,
ladder.answer_reserve_tokens + 1
),
requested_reasoning_budget(&req_with_effort(None), &cfg, cfg.answer_reserve_tokens + 1),
Some(1),
"one token above the reserve is the first bounded case"
);
@@ -7481,12 +7773,11 @@ mod tests {
/// thing creating a bound.
#[test]
fn a_zero_reserve_restores_unbounded_reasoning() {
let ladder = super::super::reasoning_budget::ReasoningBudgetLadder {
let cfg = super::super::reasoning_budget::ReasoningBudgetConfig {
answer_reserve_tokens: 0,
..Default::default()
};
assert_eq!(
requested_reasoning_budget(&req_with_effort(Some("xhigh")), &ladder, 32_768),
requested_reasoning_budget(&req_with_effort(Some("xhigh")), &cfg, 32_768),
None
);
}
@@ -7507,9 +7798,84 @@ mod tests {
seed: 0,
repeat_penalty,
repeat_last_n,
presence_penalty: 0.0,
frequency_penalty: 0.0,
}
}
/// Presence is a *one-time* subtraction for any token already seen,
/// however often. Frequency scales with the count. These are the
/// OpenAI semantics, and a client sending `presence_penalty: 1.5`
/// expecting them must get them.
#[test]
fn sequence_penalties_follow_openai_semantics() {
use candle_core::{Device, Tensor};
let logits = Tensor::from_vec(vec![0.0f32; 6], 6, &Device::Cpu).unwrap();
// token 1 appears three times, token 2 once.
let history = [1u32, 1, 1, 2];
let p = apply_sequence_penalties(&logits, &history, 1.0, 0.0)
.unwrap()
.to_vec1::<f32>()
.unwrap();
assert_eq!(p[1], -1.0, "presence subtracts once regardless of count");
assert_eq!(p[2], -1.0);
assert_eq!(p[0], 0.0, "unseen tokens untouched");
let f = apply_sequence_penalties(&logits, &history, 0.0, 1.0)
.unwrap()
.to_vec1::<f32>()
.unwrap();
assert_eq!(f[1], -3.0, "frequency scales with the count");
assert_eq!(f[2], -1.0);
assert_eq!(f[0], 0.0);
// Both together compose additively.
let b = apply_sequence_penalties(&logits, &history, 0.5, 1.0)
.unwrap()
.to_vec1::<f32>()
.unwrap();
assert_eq!(b[1], -3.5);
}
/// The whole point: a passage restated far outside `repeat_last_n`
/// is invisible to the windowed penalty but not to this one. At the
/// built-in 64-token window inside a 28,672-token think block, that
/// is every repetition worth catching (#299).
#[test]
fn a_repeat_far_outside_the_window_is_still_penalised() {
use candle_core::{Device, Tensor};
let logits = Tensor::from_vec(vec![0.0f32; 4], 4, &Device::Cpu).unwrap();
// token 3 was emitted 5,000 tokens ago and never since.
let mut history = vec![3u32];
history.extend(std::iter::repeat_n(0u32, 5_000));
let windowed = &history[history.len().saturating_sub(64)..];
assert!(
!windowed.contains(&3),
"fixture: the token must be outside the 64-token window"
);
let out = apply_sequence_penalties(&logits, &history, 1.5, 0.0)
.unwrap()
.to_vec1::<f32>()
.unwrap();
assert_eq!(out[3], -1.5, "sequence-wide penalty still sees it");
}
/// Zero on both is exactly a no-op — an operator who sets neither
/// must get byte-identical behaviour to before this existed.
#[test]
fn zero_sequence_penalties_change_nothing() {
use candle_core::{Device, Tensor};
let logits = Tensor::from_vec(vec![1.5f32, -2.0, 0.25], 3, &Device::Cpu).unwrap();
let out = apply_sequence_penalties(&logits, &[0u32, 1, 1], 0.0, 0.0)
.unwrap()
.to_vec1::<f32>()
.unwrap();
assert_eq!(out, vec![1.5, -2.0, 0.25]);
}
/// Two inlets converge on one field: an explicit token count, or
/// Anthropic's normalised `budget_tokens`, which translation folds
/// into the same shape.
@@ -7523,7 +7889,7 @@ mod tests {
#[test]
fn the_reasoning_budget_resolves_from_every_inlet() {
use cortex_core::openai::{ChatCompletionRequest, MessageContent};
let ladder = crate::harness::reasoning_budget::ReasoningBudgetLadder::default();
let cfg = crate::harness::reasoning_budget::ReasoningBudgetConfig::default();
let req = |extra: serde_json::Value| ChatCompletionRequest {
model: "m".into(),
messages: vec![],
@@ -7533,6 +7899,8 @@ mod tests {
seed: None,
repetition_penalty: None,
repeat_last_n: None,
presence_penalty: None,
frequency_penalty: None,
max_tokens: None,
max_completion_tokens: None,
stream: None,
@@ -7543,14 +7911,14 @@ mod tests {
// Every case below shares one output budget, so the only thing
// that varies is the caller's reasoning control.
const OUT: usize = 32_768;
let reserved = OUT - ladder.answer_reserve_tokens;
let reserved = OUT - cfg.answer_reserve_tokens;
// Explicit count — also the shape Anthropic's `budget_tokens`
// is normalised into during translation.
assert_eq!(
requested_reasoning_budget(
&req(serde_json::json!({"reasoning": {"max_tokens": 5000}})),
&ladder,
&cfg,
OUT
),
Some(5000)
@@ -7563,7 +7931,7 @@ mod tests {
assert_eq!(
requested_reasoning_budget(
&req(serde_json::json!({"reasoning": {"effort": "medium"}})),
&ladder,
&cfg,
OUT
),
Some(reserved)
@@ -7572,7 +7940,7 @@ mod tests {
assert_eq!(
requested_reasoning_budget(
&req(serde_json::json!({"reasoning": {"effort": "low", "max_tokens": 999}})),
&ladder,
&cfg,
OUT
),
Some(999)
@@ -7581,7 +7949,7 @@ mod tests {
// is the case pi sends, and the one that returned no answer at
// all while it was unbounded.
assert_eq!(
requested_reasoning_budget(&req(serde_json::json!({})), &ladder, OUT),
requested_reasoning_budget(&req(serde_json::json!({})), &cfg, OUT),
Some(reserved)
);
// `none` disables thinking via the template, so the governor
@@ -7589,7 +7957,7 @@ mod tests {
assert_eq!(
requested_reasoning_budget(
&req(serde_json::json!({"reasoning": {"effort": "none"}})),
&ladder,
&cfg,
OUT
),
Some(reserved)
@@ -8382,7 +8750,8 @@ mod tests {
let cfg = crate::config::ContextLimitConfig::default();
let profile = backstop_profile();
let limit = super::super::context_limit::derive_limit(&profile, 40_000, 800.0, None, &cfg);
let limit =
super::super::context_limit::derive_limit(&profile, 40_000, 800.0, None, 0, &cfg);
let cap = limit.input.expect("input budget derived");
assert_eq!(cap, 87_040);
@@ -8482,6 +8851,58 @@ mod tests {
assert_eq!(v["s"], "hello"); // un-JSON-parseable → string
}
/// The shape Qwen3.8-27B emitted on helexa.ai on 2026-08-29: it
/// tagged the function itself as a parameter, so there is no
/// `<function=` anywhere. Strict parsing returned None and the whole
/// block was re-emitted as text — the user's answer to "what's the
/// best hardware to run X on?" was a screenful of raw markup.
#[test]
fn recovers_a_tool_call_that_tagged_the_function_as_a_parameter() {
let body = concat!(
"<parameter=web_search>",
" <parameter=query> \"Qwen3.8 Flash Next\" hardware GPU </parameter>",
" </function>"
);
let mut schemas = ToolSchemas::new();
let mut params = std::collections::HashMap::new();
params.insert("query".to_string(), "string".to_string());
schemas.insert("web_search".to_string(), params);
let (_, name, args) = parse_tool_call_body(body, 0, &schemas).expect("recovered");
assert_eq!(name, "web_search");
let v: serde_json::Value = serde_json::from_str(&args).unwrap();
assert_eq!(v["query"], "\"Qwen3.8 Flash Next\" hardware GPU");
// The promoted name must not also appear as an argument.
assert!(
v.get("web_search").is_none(),
"the pseudo-parameter carrying the function name is not a parameter"
);
}
/// The recovery only promotes a name the request actually offered.
/// Without that guard any `<parameter=x>` would mint a function call,
/// and a model rambling in angle brackets would start invoking tools.
#[test]
fn does_not_invent_a_function_from_an_unknown_parameter() {
let body = "<parameter=not_a_tool> <parameter=query> hi </parameter>";
let mut schemas = ToolSchemas::new();
schemas.insert("web_search".to_string(), std::collections::HashMap::new());
assert!(parse_tool_call_body(body, 0, &schemas).is_none());
}
/// A well-formed call must not be affected by the recovery path.
#[test]
fn well_formed_call_still_takes_the_function_branch() {
let body = "<function=web_search>\n<parameter=query>\nrust\n</parameter>\n</function>";
let mut schemas = ToolSchemas::new();
let mut params = std::collections::HashMap::new();
params.insert("query".to_string(), "string".to_string());
schemas.insert("web_search".to_string(), params);
let (_, name, args) = parse_tool_call_body(body, 0, &schemas).expect("parsed");
assert_eq!(name, "web_search");
assert_eq!(args, r#"{"query":"rust"}"#);
}
#[test]
fn unparseable_tool_call_body_returns_none() {
// Neither JSON nor a `<function=…>` block — caller re-emits as text.
@@ -8542,8 +8963,8 @@ mod tests {
"tools": [{"type": "function", "function": {"name": "x"}}]
}))
.unwrap();
let err =
build_prompt_for_request(Some(bad), &with_tools, &Default::default()).unwrap_err();
let err = build_prompt_for_request(Some(bad), &with_tools, &Default::default(), None)
.unwrap_err();
assert!(matches!(err, InferenceError::TemplateRenderFailed { .. }));
// No tools → falling back is harmless, so it stays Ok.
@@ -8552,6 +8973,6 @@ mod tests {
"messages": [{"role": "user", "content": "hi"}]
}))
.unwrap();
assert!(build_prompt_for_request(Some(bad), &no_tools, &Default::default()).is_ok());
assert!(build_prompt_for_request(Some(bad), &no_tools, &Default::default(), None).is_ok());
}
}
+133
View File
@@ -438,6 +438,139 @@ mod tests {
assert!(out.contains("<|vision_start|>") && out.contains("<|vision_end|>"));
}
/// A file read back through a tool result must reach the model
/// byte-for-byte.
///
/// The reported symptom: the model writes a file correctly, reads it
/// back, and sees errors that are not in the file — consistently,
/// after every write. Anything that mangles `<`, `&`, `"` or
/// newlines on this path would produce exactly that, and would make
/// the model burn its budget reconciling a file against a corrupted
/// view of itself.
#[test]
fn a_file_read_back_through_a_tool_result_is_byte_exact() {
let template = include_str!("../../tests/data/qwen3.8-27b.chat_template.jinja");
// The characters most likely to be mangled, in the shapes a real
// web project contains them.
let file = "<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n <meta charset=\"utf-8\">\n</head>\n<body>\n <script>\n if (a && b || c < d) { g.fillStyle = '#b8905a'; }\n const s = `rgba(${r},${g},0.5)`;\n\n // a comment & an ampersand\n </script>\n</body>\n</html>\n";
let mut tool_extras = serde_json::Map::new();
tool_extras.insert("tool_call_id".into(), json!("call_0"));
let mut assistant_extras = serde_json::Map::new();
assistant_extras.insert(
"tool_calls".into(),
json!([{"id":"call_0","type":"function",
"function":{"name":"read","arguments":"{\"path\":\"/tmp/index.html\"}"}}]),
);
let messages = vec![
user_msg("read the file back"),
ChatMessage {
role: "assistant".into(),
content: MessageContent::Null,
extra: Value::Object(assistant_extras),
},
ChatMessage {
role: "tool".into(),
content: MessageContent::Text(file.to_string()),
extra: Value::Object(tool_extras),
},
user_msg("is it correct?"),
];
let out =
render_chat_template(template, &messages, &Value::Null, &Value::Null).expect("render");
for probe in [
"<!DOCTYPE html>",
"<html lang=\"en\">",
"if (a && b || c < d)",
"g.fillStyle = '#b8905a';",
"`rgba(${r},${g},0.5)`",
"// a comment & an ampersand",
] {
assert!(
out.contains(probe),
"the model would not see {probe:?} verbatim — rendered prompt mangles it"
);
}
// And nothing HTML-escaped anywhere.
for bad in ["&lt;", "&gt;", "&amp;", "&quot;", "&#x27;", "&#39;"] {
assert!(
!out.contains(bad),
"found {bad:?} in the rendered prompt: content is being HTML-escaped"
);
}
}
/// Blank lines inside a replayed tool-call argument must survive
/// rendering intact.
///
/// A model that writes a source file and then re-reads it from its
/// own replayed tool call is comparing what it remembers writing
/// against what the prompt now shows. Any whitespace the pipeline
/// eats makes those disagree, and the model has no way to tell a
/// rendering artifact from a mistake of its own — which is a
/// plausible route into the announce-then-stop stall (#296).
///
/// The JSON string→object normalisation above is the step that
/// could plausibly lose them, since it round-trips the arguments
/// through a parser.
#[test]
fn a_replayed_tool_call_keeps_the_blank_lines_in_its_arguments() {
let template = include_str!("../../tests/data/qwen3.8-27b.chat_template.jinja");
// Blank lines between sections, exactly as a written source file
// carries them.
let file_body = "'use strict';\n\n// ---- constants ----\nconst A = 1;\n\n\nfunction f() {\n return A;\n}\n";
let args = json!({ "path": "/tmp/game.js", "content": file_body });
let mut assistant_extras = serde_json::Map::new();
assistant_extras.insert(
"tool_calls".into(),
json!([{
"id": "call_0", "type": "function",
"function": {
"name": "write",
// The OpenAI wire form: arguments as a JSON *string*.
"arguments": serde_json::to_string(&args).unwrap()
}
}]),
);
let mut tool_extras = serde_json::Map::new();
tool_extras.insert("tool_call_id".into(), json!("call_0"));
let messages = vec![
user_msg("write the file"),
ChatMessage {
role: "assistant".into(),
content: MessageContent::Null,
extra: Value::Object(assistant_extras),
},
ChatMessage {
role: "tool".into(),
content: MessageContent::Text("Successfully wrote 84 bytes".into()),
extra: Value::Object(tool_extras),
},
user_msg("now check it parses"),
];
let out =
render_chat_template(template, &messages, &Value::Null, &Value::Null).expect("render");
assert!(
out.contains(file_body),
"the file body must appear verbatim in the rendered prompt"
);
let blanks_in = file_body.lines().filter(|l| l.is_empty()).count();
assert_eq!(blanks_in, 3, "fixture sanity");
// And specifically the double newline the model would notice.
assert!(
out.contains("'use strict';\n\n// ---- constants ----"),
"blank line after the prologue was eaten"
);
assert!(
out.contains("const A = 1;\n\n\nfunction f()"),
"consecutive blank lines were collapsed"
);
}
/// The second turn of an OpenAI-native agentic loop: the client
/// replays its own assistant turn, which carries `tool_calls` and
/// `"content": null`. That must render as the tool-call block —
+174 -15
View File
@@ -36,6 +36,28 @@ use crate::config::ContextLimitConfig;
/// few turns.
const RATE_EMA_ALPHA: f64 = 0.3;
/// Smallest prompt whose prefill time says anything about throughput.
///
/// A prefill has a fixed per-request cost — tokenize, launch, sync —
/// that dominates a short prompt completely. The deploy's own smoke
/// probe prefills 19 tokens in ~130 ms and reads as **145 tok/s** on a
/// host that measures ~1,400. Folding that in does not make the
/// estimate slightly worse; it replaces it.
///
/// That mattered because the fallback it displaces is the whole
/// cold-start guard: with no sample at all, `derive_limit` uses
/// `bootstrap_prefill_tok_per_sec` (800, ≈96k context at the default
/// 120 s target). One 19-token probe replaced it with 145 tok/s, and
/// the host advertised **17,408** — a sixth of what it can serve —
/// until real traffic happened to warm it (#295).
///
/// 512 excludes the smoke probe and the short auxiliary calls agent
/// harnesses make (observed at 19, 143 and 469 tokens) while admitting
/// ordinary prompts. At ~1,400 tok/s a 512-token prefill runs ~370 ms,
/// so fixed overhead is roughly a tenth of it — informative rather than
/// dominant.
const MIN_PREFILL_SAMPLE_TOKENS: usize = 512;
/// Fold one throughput sample (`tokens` in `elapsed`) into the EMA held in
/// `bits`. No-op for degenerate inputs so a probe request or a clock blip
/// can't poison the average.
@@ -99,7 +121,15 @@ impl ThroughputEma {
/// Fold one prefill measurement (`prompt_tokens` processed in
/// `elapsed`) into the prefill EMA.
///
/// Samples below [`MIN_PREFILL_SAMPLE_TOKENS`] are ignored: they
/// measure fixed per-request cost rather than throughput, and the
/// value they displace — the bootstrap estimate — is better than
/// they are.
pub fn record(&self, prompt_tokens: usize, elapsed: Duration) {
if prompt_tokens < MIN_PREFILL_SAMPLE_TOKENS {
return;
}
fold_rate(&self.prefill_bits, prompt_tokens, elapsed);
}
@@ -253,7 +283,8 @@ const DEFAULT_OUTPUT_CEILING: usize = 32_768;
/// output_ceiling = min(max_output_tokens, max_position_embeddings)
/// vram_ceiling = (free_tightest − activation_headroom − min_free_floor) / kv_bytes_per_token_per_card
/// throughput_ceiling = target_prefill_latency_secs × prefill_tok_per_sec
/// context = min(max_position_embeddings, vram_ceiling, throughput_ceiling) [clamped by `hard_ceiling` if set]
/// kv_ceiling = kv_budget_mb × 1 MiB / kv_bytes_per_token_per_card
/// context = min(max_position_embeddings, vram_ceiling, throughput_ceiling, kv_ceiling) [clamped by `hard_ceiling` if set]
/// input = context − output
/// ```
///
@@ -264,6 +295,23 @@ const DEFAULT_OUTPUT_CEILING: usize = 32_768;
/// `hard_ceiling` is an optional clamp-only backstop
/// (`NEURON_MAX_PROMPT_TOKENS` or a catalogue override); `None` = no clamp.
///
/// `kv_budget_mb` is what admission will actually hand out
/// ([`kv_reservation_mb`](super::candle::kv_reservation_mb) prices every
/// request against it). Without this term the advertisement and the
/// admission gate answer "how long a prompt fits" independently, and
/// they disagree: measured on beast 2026-08-30, a 3689 MiB budget holds
/// 118,048 tokens at 32 KiB/token/card while the advertised context was
/// the model's full 131,072 — so a client that believed the
/// advertisement filled to it and got a hard `413
/// prompt_too_long_for_vram` with no compaction path. `0` means the
/// budget is not published yet (pre-load, or CPU) → no clamp, matching
/// the `free_tightest_mb == 0` sentinel above.
///
/// Note this bounds the *whole* sequence. A caller that intends to
/// request `output_ceiling` output must plan against `context − its own
/// declared output`, not against `input`: `input` is `context − output`
/// for the default reserve, which is the only output size it can know.
///
/// `reasoning`: `input = context − output` keeps a generation reserve
/// below the wall; `output` (the reserve) is a *sub-budget* of context,
/// matching opencode's compaction model.
@@ -272,6 +320,7 @@ pub fn derive_limit(
free_tightest_mb: u64,
prefill_tok_per_sec: f64,
hard_ceiling: Option<usize>,
kv_budget_mb: u64,
cfg: &ContextLimitConfig,
) -> ModelLimit {
let output = cfg.output_reserve_tokens;
@@ -305,10 +354,20 @@ pub fn derive_limit(
};
let throughput_ceiling = (cfg.target_prefill_latency_secs * tok_per_sec).max(0.0) as usize;
// Admission ceiling — what the KV gate will actually grant. Zero is
// the "not published yet" sentinel (pre-load, CPU) and imposes no
// ceiling, like `free_tightest_mb` above.
let kv_ceiling = if kv_budget_mb == 0 || profile.kv_bytes_per_token_per_card == 0 {
usize::MAX
} else {
(kv_budget_mb.saturating_mul(1024 * 1024) / profile.kv_bytes_per_token_per_card) as usize
};
let mut context = profile
.max_position_embeddings
.min(vram_ceiling)
.min(throughput_ceiling);
.min(throughput_ceiling)
.min(kv_ceiling);
if let Some(clamp) = hard_ceiling {
context = context.min(clamp);
}
@@ -325,6 +384,8 @@ pub fn derive_limit(
prefill_tok_per_sec = tok_per_sec,
vram_ceiling,
throughput_ceiling,
kv_budget_mb,
kv_ceiling,
?hard_ceiling,
context,
"derive_limit"
@@ -356,6 +417,60 @@ pub fn derive_limit(
#[cfg(test)]
mod tests {
/// The deploy's smoke probe must not set the advertised context.
///
/// 19 tokens in 130 ms reads as ~145 tok/s on a host that measures
/// ~1,400. Before this guard that one sample displaced the
/// bootstrap estimate and the host advertised 17,408 instead of
/// ~96,000 — until real traffic happened along (#295).
#[test]
fn a_smoke_probe_does_not_poison_the_prefill_rate() {
let ema = ThroughputEma::new();
ema.record(19, Duration::from_millis(130));
assert_eq!(
ema.get(),
None,
"a 19-token prefill must leave the EMA unset so the bootstrap value stands"
);
}
/// The short auxiliary calls agent harnesses make are the same
/// shape as the probe and equally uninformative — observed at 143
/// and 469 tokens against prompts three orders of magnitude larger.
#[test]
fn short_auxiliary_prompts_are_ignored_too() {
let ema = ThroughputEma::new();
for tokens in [143usize, 469] {
ema.record(tokens, Duration::from_millis(200));
}
assert_eq!(ema.get(), None);
}
/// A real prompt still records, and still tracks a shift.
#[test]
fn representative_prefills_still_measure() {
let ema = ThroughputEma::new();
ema.record(1400, Duration::from_secs(1));
let first = ema.get().expect("a 1400-token sample must record");
assert!((first - 1400.0).abs() < 1.0, "got {first}");
// A slower host is tracked, not ignored.
for _ in 0..20 {
ema.record(700, Duration::from_secs(1));
}
let after = ema.get().unwrap();
assert!(after < first && after > 600.0, "got {after}");
}
/// Exactly at the threshold counts — the boundary is inclusive so
/// the constant means what it says.
#[test]
fn the_threshold_itself_is_a_valid_sample() {
let ema = ThroughputEma::new();
ema.record(MIN_PREFILL_SAMPLE_TOKENS, Duration::from_secs(1));
assert!(ema.get().is_some());
}
use super::*;
/// The ceiling a client is told to plan against must not be the
@@ -369,7 +484,7 @@ mod tests {
kv_bytes_per_token_per_card: 65_536,
world_size: 2,
};
let limit = derive_limit(&profile, 0, 0.0, None, &cfg);
let limit = derive_limit(&profile, 0, 0.0, None, 0, &cfg);
assert_eq!(limit.output, cfg.output_reserve_tokens, "reserve unchanged");
assert_eq!(limit.output_ceiling, DEFAULT_OUTPUT_CEILING);
assert!(limit.output_ceiling > limit.output);
@@ -385,10 +500,54 @@ mod tests {
kv_bytes_per_token_per_card: 65_536,
world_size: 1,
};
let limit = derive_limit(&profile, 0, 0.0, None, &cfg);
let limit = derive_limit(&profile, 0, 0.0, None, 0, &cfg);
assert_eq!(limit.output_ceiling, 16_384);
}
/// The advertisement may not promise a context the KV gate will
/// refuse. Numbers are the live beast measurement (2026-08-30): a
/// 3689 MiB budget at 32 KiB/token/card holds 118,048 tokens, while
/// the model's own position limit is 131,072. Before this clamp the
/// node advertised 131,072, pi filled a session to 90,432 prompt
/// tokens believing it, and the turn died on a hard 413.
#[test]
fn the_context_never_exceeds_what_admission_will_grant() {
let cfg = ContextLimitConfig::default();
let profile = ContextProfile {
max_position_embeddings: 131_072,
kv_bytes_per_token_per_card: 32_768,
world_size: 2,
};
// Roomy VRAM and a fast card, so only the KV budget can bind.
let limit = derive_limit(&profile, 60_000, 100_000.0, None, 3689, &cfg);
let context = limit.context;
assert!(
context <= 118_048,
"advertised {context} tokens against a budget holding 118,048"
);
assert_eq!(context, 117_760, "118,048 rounded down to the 1024 grid");
// A caller planning against the advertised ceiling must fit too:
// it subtracts its own declared output from `context`.
let ceiling = limit.output_ceiling;
assert!(context.saturating_sub(ceiling) <= 118_048 - ceiling);
}
/// Zero is "not published yet" (pre-load, or a CPU load), not "no
/// room" — the same sentinel `free_tightest_mb` uses. Collapsing the
/// limit to zero there would unadvertise every model on startup.
#[test]
fn an_unpublished_kv_budget_imposes_no_ceiling() {
let cfg = ContextLimitConfig::default();
let with_budget = derive_limit(&beast_profile(), 9254, 8500.0, None, 3689, &cfg);
let without = derive_limit(&beast_profile(), 9254, 8500.0, None, 0, &cfg);
assert!(
without.context >= with_budget.context,
"the sentinel must not bind tighter than a real budget"
);
assert!(without.context > 0);
}
/// The operator's number wins, which is what makes the default a
/// starting point rather than a guess baked into the binary.
#[test]
@@ -403,7 +562,7 @@ mod tests {
world_size: 2,
};
assert_eq!(
derive_limit(&profile, 0, 0.0, None, &cfg).output_ceiling,
derive_limit(&profile, 0, 0.0, None, 0, &cfg).output_ceiling,
65_536
);
}
@@ -420,10 +579,10 @@ mod tests {
kv_bytes_per_token_per_card: 65_536,
world_size: 2,
};
let roomy = derive_limit(&profile, 60_000, 5_000.0, None, &cfg);
let roomy = derive_limit(&profile, 60_000, 5_000.0, None, 0, &cfg);
// Tight enough that the VRAM term binds — the same shape as the
// fleet advertising 109568 one hour and 131072 the next.
let tight = derive_limit(&profile, 6_000, 5_000.0, None, &cfg);
let tight = derive_limit(&profile, 6_000, 5_000.0, None, 0, &cfg);
assert!(
tight.context < roomy.context,
"the live window should shrink under VRAM pressure"
@@ -459,7 +618,7 @@ mod tests {
// VRAM ceiling on beast pre-#11. VRAM (~9.2 GB free) allows far
// more, max_position_embeddings is 262144, so throughput wins.
let cfg = ContextLimitConfig::default();
let limit = derive_limit(&beast_profile(), 9254, 850.0, None, &cfg);
let limit = derive_limit(&beast_profile(), 9254, 850.0, None, 0, &cfg);
// 120 × 850 = 102000 → rounded down to 1024 → 101376.
assert_eq!(limit.context, 101376);
assert_eq!(limit.output, 8192);
@@ -472,8 +631,8 @@ mod tests {
// Prefix caching (#11) speeds effective prefill → ceiling rises,
// eventually pinned by VRAM / max_position_embeddings.
let cfg = ContextLimitConfig::default();
let slow = derive_limit(&beast_profile(), 9254, 850.0, None, &cfg);
let fast = derive_limit(&beast_profile(), 9254, 8500.0, None, &cfg);
let slow = derive_limit(&beast_profile(), 9254, 850.0, None, 0, &cfg);
let fast = derive_limit(&beast_profile(), 9254, 8500.0, None, 0, &cfg);
assert!(fast.context > slow.context);
}
@@ -481,8 +640,8 @@ mod tests {
fn tighter_vram_lowers_the_limit() {
// Same model, less free VRAM → VRAM ceiling binds below throughput.
let cfg = ContextLimitConfig::default();
let roomy = derive_limit(&beast_profile(), 9254, 8500.0, None, &cfg);
let tight = derive_limit(&beast_profile(), 2600, 8500.0, None, &cfg);
let roomy = derive_limit(&beast_profile(), 9254, 8500.0, None, 0, &cfg);
let tight = derive_limit(&beast_profile(), 2600, 8500.0, None, 0, &cfg);
assert!(tight.context < roomy.context);
}
@@ -490,10 +649,10 @@ mod tests {
fn hard_ceiling_clamps_only_downward() {
let cfg = ContextLimitConfig::default();
// A backstop below the derived value clamps it.
let clamped = derive_limit(&beast_profile(), 9254, 8500.0, Some(49152), &cfg);
let clamped = derive_limit(&beast_profile(), 9254, 8500.0, Some(49152), 0, &cfg);
assert_eq!(clamped.context, 49152);
// A backstop above the derived value is a no-op.
let unclamped = derive_limit(&beast_profile(), 9254, 850.0, Some(200000), &cfg);
let unclamped = derive_limit(&beast_profile(), 9254, 850.0, Some(200000), 0, &cfg);
assert_eq!(unclamped.context, 101376);
}
@@ -545,7 +704,7 @@ mod tests {
world_size: 1,
};
let cfg = ContextLimitConfig::default();
let limit = derive_limit(&profile, 8000, 8500.0, None, &cfg);
let limit = derive_limit(&profile, 8000, 8500.0, None, 0, &cfg);
// max_position_embeddings (32768) binds below throughput (~1.02M).
assert_eq!(limit.context, 32768);
}
+1
View File
@@ -402,6 +402,7 @@ mod tests {
tensor_parallel: tp,
devices: None,
sampling: None,
preserve_thinking: None,
}
}
+115 -96
View File
@@ -13,40 +13,39 @@
//!
//! Two halves live here:
//!
//! - [`ReasoningBudget`] — how many reasoning tokens this request may
//! spend, resolved from whichever control the caller's wire format
//! gave them.
//! - [`ReasoningBudgetConfig`] — the operator's single knob,
//! `answer_reserve_tokens`. The budget itself is derived per request
//! in `candle::requested_reasoning_budget` as
//! `declared_max_tokens − answer_reserve_tokens`, or taken verbatim
//! from `reasoning.max_tokens` when the caller sets one.
//! - [`ReasoningGovernor`] — the per-request state that enforces it, by
//! substituting the model's own `</think>` token when the budget runs
//! out so generation transitions to answering instead of being cut off.
//!
//! **The budget does not depend on the effort level, and must not.**
//! Effort reaches the model as a prompt instruction through its own chat
//! template (`candle::apply_effort_kwarg`, #290); deriving a token cap
//! from it as well is what told Qwen3.8-27B to reason at `xhigh` and
//! then guillotined it at `medium`'s allowance. This module carried a
//! per-rung table for that purpose until 2026-08-30; it had been dead
//! since #290 and is gone. If you are looking for where `low` differs
//! from `xhigh`, it is the rendered prompt, not a number here.
use serde::{Deserialize, Serialize};
/// Per-model mapping from an OpenAI effort level to a token budget.
/// How much of a caller's output budget is held back for the answer.
///
/// OpenAI-shaped clients cannot send a token count — pi-ai's vocabulary
/// is the ladder `off | minimal | low | medium | high | xhigh | max` and
/// nothing else — so the numbers have to live on the server. Operator
/// configurable, because the right values depend on the model and the
/// hardware; a derived default would be a guess baked into the binary.
/// One field. It was a per-effort ladder until 2026-08-30 — see the
/// module doc for why that was the wrong shape and why nothing read it.
/// The name stays `[harness.candle.reasoning_budget]` on the wire so
/// operator configs keep working; unknown keys are ignored, so a stale
/// `medium = 12288` is harmless.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ReasoningBudgetLadder {
pub minimal: usize,
pub low: usize,
pub medium: usize,
pub high: usize,
/// `xhigh` and `max` are only offered by a client when the model
/// declares them; absent here means "not offered", and a request
/// naming one falls back to `high` rather than being rejected.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub xhigh: Option<usize>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub max: Option<usize>,
pub struct ReasoningBudgetConfig {
/// Tokens of the caller's **own** output budget held back for the
/// answer, when the caller names no explicit `reasoning.max_tokens`.
///
/// This is what an OpenAI-shaped client gets instead of the rungs
/// above, and it is deliberately not derived from effort. Effort is
/// Deliberately not derived from effort. Effort is
/// a prompt instruction the model is trained to follow (#290);
/// making it *also* set a guillotine is what told Qwen3.8-27B to
/// reason at `xhigh` and then cut it off at `medium`'s 12,288.
@@ -69,7 +68,7 @@ fn default_answer_reserve() -> usize {
4_096
}
impl Default for ReasoningBudgetLadder {
impl Default for ReasoningBudgetConfig {
/// Rungs sized against observed behaviour rather than round numbers:
/// a long agentic design turn on this class of model runs to ~27k
/// tokens, so `high` sits above that (bounded but rarely binding),
@@ -77,57 +76,11 @@ impl Default for ReasoningBudgetLadder {
/// and `low` at the scale of a focused single-file change.
fn default() -> Self {
Self {
minimal: 1_024,
low: 4_096,
medium: 12_288,
high: 32_768,
xhigh: None,
max: None,
answer_reserve_tokens: default_answer_reserve(),
}
}
}
impl ReasoningBudgetLadder {
/// Resolve an effort level to a token budget.
///
/// `none`/`off` are handled before this — they disable thinking
/// outright via the chat template — so they have no budget here.
/// An unrecognised level resolves to `None` (unbounded) rather than
/// to a guess: silently substituting a budget nobody asked for is
/// the failure mode this whole area has been full of.
pub fn for_effort(&self, effort: &str) -> Option<usize> {
match effort.trim().to_ascii_lowercase().as_str() {
"minimal" => Some(self.minimal),
"low" => Some(self.low),
"medium" => Some(self.medium),
"high" => Some(self.high),
"xhigh" => Some(self.xhigh.unwrap_or(self.high)),
"max" => Some(self.max.or(self.xhigh).unwrap_or(self.high)),
_ => None,
}
}
/// The mapping as a client should see it on `/v1/models` — the rungs
/// this deployment actually offers, so a caller picking `low` knows
/// what it bought instead of guessing (#274).
pub fn advertised(&self) -> Vec<(String, usize)> {
let mut out = vec![
("minimal".to_string(), self.minimal),
("low".to_string(), self.low),
("medium".to_string(), self.medium),
("high".to_string(), self.high),
];
if let Some(v) = self.xhigh {
out.push(("xhigh".to_string(), v));
}
if let Some(v) = self.max {
out.push(("max".to_string(), v));
}
out
}
}
/// Enforces a resolved budget over one request's token stream.
///
/// Fed every sampled token; returns the token that should actually be
@@ -270,36 +223,102 @@ mod tests {
}
assert!(!g.forced(), "content after the block must not be governed");
}
}
/// Whether the longest reasoning rung should be offered on a node
/// running `max_in_flight` concurrent slots.
///
/// Operator-reported (2026-08-29): `xhigh` "hits issues" when this host
/// runs eight slots. The mechanism is not yet isolated, so this is a
/// precaution rather than a diagnosis, and it is written down as one —
/// naming it a fix would stop anyone looking.
///
/// The plausible account: `xhigh` instructs the model to deliberate
/// exhaustively, so a turn can spend tens of thousands of tokens
/// reasoning. KV is a shared pool (#291), and a slot holds its
/// reservation for the request's whole life, so several concurrent
/// `xhigh` turns can hold most of the pool for minutes while shorter
/// requests queue on bytes. That is consistent with "issues" but is not
/// evidence of them; #305's turn is a candidate to instrument.
///
/// **2026-08-30: that account is now contradicted, and the constant
/// below did not hold.** A controlled A/B/C on this host reproduced the
/// defect at `max_in_flight = 2` with `in_flight == 1` — the very
/// configuration this function calls safe — while the `medium` arm at
/// `max_in_flight = 8` was clean. Concurrency does not predict it.
/// Neither does the GatedDeltaNet bf16 state round-trip (#284):
/// `NEURON_GDN_STATE_F32=1` changed neither the defect rate nor its
/// class on a run matched to within 0.2% on the opening generation.
/// What both `xhigh` arms produced was a read against the wrong
/// receiver with the correct one in scope on an adjacent line
/// (`tw.damage` for `def.damage`; `1 - slowPct` for `slowPct / 100`) —
/// the same shape as #305's `<parameter=` for `<function=`. The
/// mechanism remains unisolated; this is still a precaution, and the
/// evidence now says it is very likely guarding the wrong variable.
/// Do not read the number below as a diagnosis.
///
/// Availability is advertised, never enforced. The request path does not
/// reject an unavailable rung — a caller that insists still gets it. The
/// point is to keep a UI from offering an option that will serve someone
/// badly, not to add a refusal path to every existing API client.
pub fn long_rung_available(max_in_flight: usize) -> bool {
max_in_flight <= LONG_RUNG_MAX_IN_FLIGHT
}
/// The slot count above which the longest rung stops being offered.
///
/// Two, because that is the concurrency at which a single deep session
/// still has the pool substantially to itself. A named constant rather
/// than a literal at the call site so the number is findable when the
/// mechanism above is finally isolated and it turns out to be wrong.
pub const LONG_RUNG_MAX_IN_FLIGHT: usize = 2;
/// The rung a deployment considers "longest", if it offers one.
///
/// Taken as the last entry rather than by matching on `"xhigh"`: the
/// rungs come from the model's own template (#290), so the name of the
/// deepest one is the model's to choose and a future model may spell it
/// differently.
pub fn longest_rung(levels: &[String]) -> Option<&String> {
levels.last()
}
#[cfg(test)]
mod availability_tests {
use super::*;
#[test]
fn the_ladder_resolves_the_levels_clients_can_send() {
let l = ReasoningBudgetLadder::default();
assert_eq!(l.for_effort("minimal"), Some(1_024));
assert_eq!(l.for_effort("LOW"), Some(4_096));
assert_eq!(l.for_effort(" medium "), Some(12_288));
assert_eq!(l.for_effort("high"), Some(32_768));
// Not offered by default → falls back to high rather than
// rejecting a level the client was told it could send.
assert_eq!(l.for_effort("xhigh"), Some(32_768));
assert_eq!(l.for_effort("max"), Some(32_768));
// Unknown → unbounded, never a substituted guess.
assert_eq!(l.for_effort("enthusiastic"), None);
assert_eq!(l.for_effort("none"), None);
fn the_long_rung_is_offered_only_at_low_concurrency() {
assert!(long_rung_available(1));
assert!(long_rung_available(2));
assert!(!long_rung_available(3));
assert!(!long_rung_available(8));
}
#[test]
fn the_advertised_ladder_lists_only_what_is_offered() {
let l = ReasoningBudgetLadder::default();
let names: Vec<_> = l.advertised().into_iter().map(|(k, _)| k).collect();
assert_eq!(names, ["minimal", "low", "medium", "high"]);
let l2 = ReasoningBudgetLadder {
xhigh: Some(49_152),
..ReasoningBudgetLadder::default()
};
assert!(
l2.advertised()
.iter()
.any(|(k, v)| k == "xhigh" && *v == 49_152)
);
fn the_longest_rung_is_the_last_one_the_template_offers() {
// Not a match on "xhigh": the ladder is the model's, and the
// deepest rung is whatever it puts last.
let levels = vec!["low".to_string(), "medium".to_string(), "xhigh".to_string()];
assert_eq!(longest_rung(&levels).map(String::as_str), Some("xhigh"));
let other = vec!["brief".to_string(), "exhaustive".to_string()];
assert_eq!(longest_rung(&other).map(String::as_str), Some("exhaustive"));
}
#[test]
fn a_model_with_no_rungs_has_no_longest_one() {
assert_eq!(longest_rung(&[]), None);
}
/// A single-rung model must not have its only option withdrawn: the
/// rule exists to steer a choice, and there is no choice to steer.
#[test]
fn a_single_rung_is_never_withdrawn() {
let levels = vec!["only".to_string()];
assert_eq!(longest_rung(&levels).map(String::as_str), Some("only"));
// The caller is responsible for not withdrawing a lone rung;
// pinned here so the intent survives a refactor of that caller.
assert_eq!(levels.len(), 1);
}
}
+152 -3
View File
@@ -75,6 +75,15 @@ pub struct ModelGenerationDefaults {
pub temperature: Option<f64>,
pub top_p: Option<f64>,
pub top_k: Option<usize>,
/// Not published in `generation_config.json` — this struct is the
/// resolved per-model default layer, and the operator override
/// (#283) lands here so the repetition knobs sit in the same
/// precedence chain as the rest.
pub repeat_penalty: Option<f32>,
pub repeat_last_n: Option<usize>,
pub presence_penalty: Option<f32>,
pub frequency_penalty: Option<f32>,
pub seed: Option<u64>,
}
impl ModelGenerationDefaults {
@@ -155,6 +164,21 @@ impl ModelGenerationDefaults {
if let Some(k) = op.top_k {
self.top_k = Some(k);
}
if let Some(r) = op.repeat_penalty {
self.repeat_penalty = Some(r);
}
if let Some(n) = op.repeat_last_n {
self.repeat_last_n = Some(n);
}
if let Some(v) = op.presence_penalty {
self.presence_penalty = Some(v);
}
if let Some(v) = op.frequency_penalty {
self.frequency_penalty = Some(v);
}
if let Some(v) = op.seed {
self.seed = Some(v);
}
tracing::info!(
model = %model_id,
model_temperature = ?before.temperature,
@@ -163,6 +187,11 @@ impl ModelGenerationDefaults {
temperature = ?self.temperature,
top_p = ?self.top_p,
top_k = ?self.top_k,
repeat_penalty = ?self.repeat_penalty,
repeat_last_n = ?self.repeat_last_n,
presence_penalty = ?self.presence_penalty,
frequency_penalty = ?self.frequency_penalty,
seed = ?self.seed,
"sampling: operator override applied over the model's published defaults (#283)"
);
self
@@ -179,6 +208,8 @@ pub struct RequestedSampling {
pub seed: Option<u64>,
pub repeat_penalty: Option<f32>,
pub repeat_last_n: Option<usize>,
pub presence_penalty: Option<f32>,
pub frequency_penalty: Option<f32>,
}
/// Fully-resolved sampling for one request.
@@ -190,6 +221,9 @@ pub struct SamplingParams {
pub seed: u64,
pub repeat_penalty: f32,
pub repeat_last_n: usize,
/// Sequence-wide penalties, OpenAI semantics. `0.0` disables.
pub presence_penalty: f32,
pub frequency_penalty: f32,
}
impl SamplingParams {
@@ -215,9 +249,23 @@ impl SamplingParams {
// whenever the caller stayed quiet.
top_p: requested.top_p.or(model.top_p),
top_k: requested.top_k.or(model.top_k),
seed: requested.seed.unwrap_or(random_seed),
repeat_penalty: requested.repeat_penalty.unwrap_or(DEFAULT_REPEAT_PENALTY),
repeat_last_n: requested.repeat_last_n.unwrap_or(DEFAULT_REPEAT_LAST_N),
seed: requested.seed.or(model.seed).unwrap_or(random_seed),
repeat_penalty: requested
.repeat_penalty
.or(model.repeat_penalty)
.unwrap_or(DEFAULT_REPEAT_PENALTY),
repeat_last_n: requested
.repeat_last_n
.or(model.repeat_last_n)
.unwrap_or(DEFAULT_REPEAT_LAST_N),
presence_penalty: requested
.presence_penalty
.or(model.presence_penalty)
.unwrap_or(0.0),
frequency_penalty: requested
.frequency_penalty
.or(model.frequency_penalty)
.unwrap_or(0.0),
}
}
@@ -252,6 +300,7 @@ mod tests {
temperature: Some(1.0),
top_p: Some(0.95),
top_k: Some(20),
..Default::default()
}
}
@@ -298,6 +347,7 @@ mod tests {
temperature,
top_p,
top_k,
..Default::default()
}
}
@@ -394,6 +444,103 @@ mod tests {
/// A model with no `generation_config.json` must behave exactly as
/// it did before this landed, so nothing regresses on the models
/// that do not ship one.
/// An operator can pin the seed, because the client that needs it
/// pinned cannot send one: pi's request builder emits no `seed`, so
/// without this a parameter search runs unseeded and sampling
/// variance is indistinguishable from the effect being measured.
#[test]
fn an_operator_can_pin_the_seed() {
let op = cortex_core::harness::SamplingOverride {
seed: Some(424_242),
..Default::default()
};
let defaults = ModelGenerationDefaults::default().with_operator_override("m", Some(&op));
// The random seed argument must be ignored once an operator pinned one.
let p = SamplingParams::resolve(&RequestedSampling::default(), &defaults, 999);
assert_eq!(p.seed, 424_242);
}
/// Precedence holds: a caller that names its own seed keeps it.
#[test]
fn a_requested_seed_outranks_the_operator() {
let op = cortex_core::harness::SamplingOverride {
seed: Some(424_242),
..Default::default()
};
let defaults = ModelGenerationDefaults::default().with_operator_override("m", Some(&op));
let req = RequestedSampling {
seed: Some(7),
..Default::default()
};
assert_eq!(SamplingParams::resolve(&req, &defaults, 999).seed, 7);
}
/// With neither, the per-request random seed still applies — an
/// unset override must not accidentally freeze the fleet's RNG.
#[test]
fn no_pinned_seed_leaves_generation_random() {
let d = ModelGenerationDefaults::default().with_operator_override("m", None);
assert_eq!(
SamplingParams::resolve(&RequestedSampling::default(), &d, 999).seed,
999
);
}
/// The gap this closes: `repeat_last_n` defaults to 64 — candle's
/// chat example value — while a reasoning model's think block runs
/// to tens of thousands of tokens. Anything restated more than 64
/// tokens later is invisible to the penalty, and before this there
/// was no operator route to the knob at all: `SamplingOverride`
/// carried only temperature/top_p/top_k, and OpenAI-shaped clients
/// do not send repetition parameters. The 64 was effectively
/// hardcoded in production.
#[test]
fn an_operator_can_widen_the_repetition_window() {
let op = cortex_core::harness::SamplingOverride {
repeat_penalty: Some(1.05),
repeat_last_n: Some(2048),
presence_penalty: None,
frequency_penalty: None,
..Default::default()
};
let defaults = ModelGenerationDefaults::default().with_operator_override("m", Some(&op));
let p = SamplingParams::resolve(&RequestedSampling::default(), &defaults, 7);
assert_eq!(p.repeat_last_n, 2048);
assert!((p.repeat_penalty - 1.05).abs() < f32::EPSILON);
}
/// Precedence is request > operator > built-in, same as every other
/// sampling field — a caller that named a window keeps it.
#[test]
fn a_requested_repetition_window_outranks_the_operator() {
let op = cortex_core::harness::SamplingOverride {
repeat_last_n: Some(2048),
presence_penalty: None,
frequency_penalty: None,
..Default::default()
};
let defaults = ModelGenerationDefaults::default().with_operator_override("m", Some(&op));
let req = RequestedSampling {
repeat_last_n: Some(128),
presence_penalty: None,
frequency_penalty: None,
..Default::default()
};
assert_eq!(
SamplingParams::resolve(&req, &defaults, 7).repeat_last_n,
128
);
}
/// An operator who sets nothing must not change the constants.
#[test]
fn no_operator_setting_leaves_the_repetition_defaults_alone() {
let d = ModelGenerationDefaults::default().with_operator_override("m", None);
let p = SamplingParams::resolve(&RequestedSampling::default(), &d, 7);
assert_eq!(p.repeat_last_n, DEFAULT_REPEAT_LAST_N);
assert!((p.repeat_penalty - DEFAULT_REPEAT_PENALTY).abs() < f32::EPSILON);
}
#[test]
fn without_model_defaults_the_old_behaviour_is_preserved() {
let p = SamplingParams::resolve(
@@ -495,6 +642,8 @@ mod tests {
&RequestedSampling {
repeat_penalty: Some(1.05),
repeat_last_n: Some(512),
presence_penalty: None,
frequency_penalty: None,
..Default::default()
},
&ModelGenerationDefaults::default(),
+108 -12
View File
@@ -468,11 +468,21 @@ fn usage_chunk(
/// where the same wire object could be assembled differently — the
/// shape of defect #252 and #272 both were.
///
/// Semantics deliberately preserved from the path this replaces:
/// reasoning text is **dropped** from `content` (it is counted in
/// `usage.completion_tokens_details.reasoning_tokens` but never
/// rendered), and `finish_reason` becomes `"tool_calls"` whenever any
/// tool call was parsed.
/// Reasoning is kept out of `content` — it is not the assistant's
/// reply — but it is **returned** on `message.reasoning_content`, the
/// same field the streaming path emits as
/// `choice.delta.reasoning_content`.
///
/// It used to be discarded here (#300). That was inherited from the
/// path this replaced and was wrong on both counts: the tokens are
/// metered in `usage.completion_tokens_details.reasoning_tokens`, so a
/// caller paid for text it could never see; and a client cannot replay
/// what it never received, so on the next turn the model saw its own
/// prior turn as a bare assertion with the reasoning that produced it
/// deleted.
///
/// `finish_reason` becomes `"tool_calls"` whenever any tool call was
/// parsed.
///
/// Two fields that the old path hardcoded to `None` now populate,
/// because the events carry them and the struct already had the slots:
@@ -488,6 +498,7 @@ pub async fn collect_chat_completion(
model_id: String,
) -> Result<ChatCompletionResponse, String> {
let mut content = String::new();
let mut reasoning = String::new();
let mut tool_calls: Vec<serde_json::Value> = Vec::new();
let mut finish: Option<(FinishReason, u32, u32, u32, u32, Option<FinishTiming>)> = None;
@@ -495,8 +506,8 @@ pub async fn collect_chat_completion(
match event {
InferenceEvent::Start => {}
InferenceEvent::TextDelta(t) => content.push_str(&t),
// Counted, never rendered — see the doc comment.
InferenceEvent::ReasoningDelta(_) => {}
// Kept out of `content`, returned on `reasoning_content`.
InferenceEvent::ReasoningDelta(t) => reasoning.push_str(&t),
InferenceEvent::ToolCall {
index,
id: call_id,
@@ -545,11 +556,16 @@ pub async fn collect_chat_completion(
} else {
"tool_calls".to_string()
};
let message_extra = if tool_calls.is_empty() {
serde_json::Value::Object(Default::default())
} else {
json!({ "tool_calls": tool_calls })
};
let mut message_extra = serde_json::Map::new();
if !tool_calls.is_empty() {
message_extra.insert("tool_calls".into(), json!(tool_calls));
}
// Absent rather than empty when the model did not think: a
// non-reasoning model must keep the unchanged message shape.
if !reasoning.is_empty() {
message_extra.insert("reasoning_content".into(), json!(reasoning));
}
let message_extra = serde_json::Value::Object(message_extra);
Ok(ChatCompletionResponse {
id,
@@ -588,6 +604,86 @@ pub async fn collect_chat_completion(
#[cfg(test)]
mod tests {
/// The model's think block must come back on the non-streaming
/// path too (#300).
///
/// It used to be dropped here while the streaming path emitted it
/// as `choice.delta.reasoning_content`. The tokens were metered
/// either way, so a non-streaming caller paid for text it could
/// never see — and, worse, could not replay the model's reasoning
/// on the next turn, leaving the model to face its own prior turn
/// as a bare assertion.
#[tokio::test]
async fn non_streaming_returns_the_models_reasoning() {
let (tx, rx) = mpsc::channel::<InferenceEvent>(8);
tx.send(InferenceEvent::Start).await.unwrap();
tx.send(InferenceEvent::ReasoningDelta(
"weighing the options".into(),
))
.await
.unwrap();
tx.send(InferenceEvent::TextDelta("the answer".into()))
.await
.unwrap();
tx.send(InferenceEvent::Finish {
reason: FinishReason::Stop,
prompt_tokens: 5,
completion_tokens: 7,
reasoning_tokens: 3,
cached_tokens: 0,
timing: None,
})
.await
.unwrap();
drop(tx);
let r = collect_chat_completion(rx, "id".into(), 0, "m".into())
.await
.expect("collect");
let msg = &r.choices[0].message;
assert_eq!(
msg.extra["reasoning_content"], "weighing the options",
"reasoning must be returned, not discarded"
);
// And it must NOT be folded into the visible reply.
assert!(
matches!(&msg.content, MessageContent::Text(t) if t == "the answer"),
"reasoning leaked into content"
);
}
/// A non-reasoning model must keep the unchanged message shape —
/// absent, not an empty string.
#[tokio::test]
async fn no_reasoning_means_no_reasoning_field() {
let (tx, rx) = mpsc::channel::<InferenceEvent>(8);
tx.send(InferenceEvent::Start).await.unwrap();
tx.send(InferenceEvent::TextDelta("plain".into()))
.await
.unwrap();
tx.send(InferenceEvent::Finish {
reason: FinishReason::Stop,
prompt_tokens: 1,
completion_tokens: 1,
reasoning_tokens: 0,
cached_tokens: 0,
timing: None,
})
.await
.unwrap();
drop(tx);
let r = collect_chat_completion(rx, "id".into(), 0, "m".into())
.await
.expect("collect");
assert!(
r.choices[0]
.message
.extra
.get("reasoning_content")
.is_none()
);
}
use super::*;
// ── Non-streaming collector (#285) ────────────────────────────
+405 -10
View File
@@ -176,6 +176,29 @@ pub fn request_to_chat(req: ResponsesRequest) -> Result<ChatCompletionRequest, T
{
attach_reasoning(&mut msg, text);
}
// A `function_call` item belongs to the assistant
// turn that precedes it, not to a turn of its own.
//
// Responses splits one turn across several items —
// `reasoning`, then `message`, then `function_call`
// — while chat carries all three on a single
// assistant message. Pushing each item separately
// produced *consecutive assistant turns*: one that
// deliberates and then ends without acting, and one
// that acts with an empty `<think></think>`.
//
// Neither shape exists in training, and the model
// imitates what it is shown: after four such
// examples in its own replayed context it produced
// a think block and a preamble, then ended the turn
// without the tool call it had just announced
// (#296). Observed on a real session; the model was
// copying our transcript, not misbehaving.
if let Some(prev) = messages.last_mut()
&& merge_tool_call_into(prev, &mut msg)
{
continue;
}
messages.push(msg);
}
}
@@ -196,6 +219,9 @@ pub fn request_to_chat(req: ResponsesRequest) -> Result<ChatCompletionRequest, T
}
}
// Only the newest turn's thinking is context; the rest is noise.
prune_stale_reasoning(&mut messages);
// Carry the caller's extension fields across the hop (#277).
//
// This used to start from an empty map and insert exactly one key,
@@ -239,6 +265,8 @@ pub fn request_to_chat(req: ResponsesRequest) -> Result<ChatCompletionRequest, T
seed: req.seed,
repetition_penalty: req.repetition_penalty,
repeat_last_n: req.repeat_last_n,
presence_penalty: req.presence_penalty,
frequency_penalty: req.frequency_penalty,
max_tokens: req.max_output_tokens,
// Responses spells the cap `max_output_tokens`, already
// resolved onto the legacy field above.
@@ -376,6 +404,105 @@ fn apply_reasoning_effort(extra: &mut serde_json::Map<String, Value>) {
}
}
/// Fold a `function_call`-derived assistant message into the assistant
/// turn it belongs to.
///
/// Returns `true` when the call was merged and the caller should drop
/// its message. Only ever merges a *pure* tool-call message (empty
/// content, no reasoning of its own) into a preceding assistant turn,
/// so a genuine two-turn sequence is never collapsed and a reasoning
/// item that arrived with the call keeps its own turn.
///
/// Consecutive calls accumulate, which is what parallel tool calling
/// looks like on the wire: one assistant turn, several `tool_calls`.
fn merge_tool_call_into(prev: &mut ChatMessage, msg: &mut ChatMessage) -> bool {
if prev.role != "assistant" || msg.role != "assistant" {
return false;
}
// Only a bare tool-call carrier folds in.
if !matches!(&msg.content, MessageContent::Text(t) if t.is_empty()) {
return false;
}
if msg.extra.get("reasoning_content").is_some() {
return false;
}
let Some(incoming) = msg
.extra
.get_mut("tool_calls")
.and_then(|v| v.as_array_mut())
.map(std::mem::take)
else {
return false;
};
// Everything else the item carried must survive the fold, or this
// trades one silent drop for another.
if msg
.extra
.as_object()
.is_some_and(|o| o.keys().any(|k| k != "tool_calls"))
{
return false;
}
let Some(obj) = prev.extra.as_object_mut() else {
return false;
};
match obj.get_mut("tool_calls").and_then(|v| v.as_array_mut()) {
Some(existing) => existing.extend(incoming),
None => {
obj.insert("tool_calls".into(), Value::Array(incoming));
}
}
true
}
/// Keep replayed reasoning on the most recent assistant turn only.
///
/// #277 restored the round-trip, and then over-corrected: a client that
/// replays every turn verbatim — which is what an agentic harness does —
/// had the model's entire thinking history rendered back into the
/// prompt, because Qwen3.8's template defaults `preserve_thinking` to
/// true. Measured on a live session: **62% of the assistant content
/// being replayed was prior think blocks** (95,311 characters of
/// thinking against 467 characters of actual output), and the model's
/// per-turn reasoning collapsed to tens of tokens while it made
/// increasingly shallow passes at the task.
///
/// Reasoning models are trained on histories that carry *final content*,
/// not the deliberation behind it — Qwen's own multi-turn convention
/// strips prior think blocks. But the newest block is different: when a
/// turn was cut off mid-thought and the caller says "continue", that
/// thinking is the work being resumed, which is what #277 exists to
/// preserve.
///
/// So: newest assistant turn keeps its reasoning, everything older
/// loses it. Note the template's own `preserve_thinking: false` is
/// exactly backwards for this — it keeps thinking on turns *after* the
/// last user message (the agentic case, where it is noise) and drops it
/// on turns *before* (the continuation case, where it is the point).
fn prune_stale_reasoning(messages: &mut [ChatMessage]) {
let newest = messages
.iter()
.rposition(|m| m.role == "assistant" && m.extra.get("reasoning_content").is_some());
let Some(newest) = newest else { return };
let mut dropped = 0usize;
for (i, m) in messages.iter_mut().enumerate() {
if i == newest || m.role != "assistant" {
continue;
}
if let Some(obj) = m.extra.as_object_mut()
&& obj.remove("reasoning_content").is_some()
{
dropped += 1;
}
}
if dropped > 0 {
tracing::debug!(
dropped,
"responses: replaying reasoning for the newest assistant turn only"
);
}
}
/// The text of a replayed reasoning item, under either spelling.
///
/// `summary` is what neuron emits (`summary_text` parts) and `content`
@@ -1173,28 +1300,49 @@ fn finish_to_status(reason: FinishReason) -> &'static str {
/// Collect a chat-completions response into a non-streaming
/// [`ResponsesResponse`]. Used by the `/v1/responses` handler when
/// the request doesn't set `stream: true`.
///
/// `reasoning` is the model's think block, when it produced one. It
/// becomes its own output item ahead of the message — the same shape
/// and ordering the streaming projector emits. Omitting it (as this did
/// until #300) meant a non-streaming caller was billed for
/// `reasoning_tokens` it could never see, and could not replay the
/// model's own thinking on the next turn.
pub fn build_response(
meta: &ResponseMeta,
full_text: String,
reason: FinishReason,
usage: Option<ResponsesUsage>,
reasoning: Option<String>,
) -> ResponsesResponse {
let status = finish_to_status(reason).to_string();
let mut output: Vec<ResponsesOutputItem> = Vec::with_capacity(2);
if let Some(text) = reasoning.filter(|t| !t.is_empty()) {
output.push(ResponsesOutputItem::Reasoning {
id: reasoning_item_id(meta),
summary: vec![cortex_core::responses::ResponsesSummaryPart::summary_text(
text,
)],
status: "completed".into(),
});
}
ResponsesResponse {
id: meta.response_id.clone(),
object: "response".into(),
created_at: meta.created_at,
status: status.clone(),
model: meta.model_id.clone(),
output: vec![ResponsesOutputItem::Message {
id: meta.message_item_id.clone(),
role: "assistant".into(),
content: vec![ResponsesOutputContent::OutputText {
text: full_text,
annotations: vec![],
}],
status,
}],
output: {
output.push(ResponsesOutputItem::Message {
id: meta.message_item_id.clone(),
role: "assistant".into(),
content: vec![ResponsesOutputContent::OutputText {
text: full_text,
annotations: vec![],
}],
status,
});
output
},
// Same contract as the streaming shell: a truncated response
// says so, or the caller cannot tell it from a fault.
incomplete_details: matches!(reason, FinishReason::Length)
@@ -1208,6 +1356,173 @@ mod tests {
use super::*;
use cortex_core::openai::MessageContent;
/// The non-streaming Responses body must carry a reasoning item
/// (#300), matching the item the streaming projector emits.
///
/// Observed before the fix: a response reporting
/// `output_tokens_details.reasoning_tokens: 4000` with an empty
/// `output` — 4,000 tokens generated, metered, and invisible.
#[test]
fn non_streaming_response_carries_a_reasoning_item() {
let r = build_response(
&meta(),
"the answer".into(),
FinishReason::Stop,
None,
Some("weighing the options".into()),
);
let v = serde_json::to_value(&r).unwrap();
let items = v["output"].as_array().expect("output array");
assert_eq!(items.len(), 2, "expected a reasoning item and a message");
assert_eq!(items[0]["type"], "reasoning", "reasoning comes first");
assert_eq!(items[0]["summary"][0]["type"], "summary_text");
assert_eq!(items[0]["summary"][0]["text"], "weighing the options");
assert_eq!(items[1]["type"], "message");
assert_eq!(items[1]["content"][0]["text"], "the answer");
}
/// A model that did not think produces no reasoning item — the
/// unchanged single-item shape.
#[test]
fn no_reasoning_means_no_reasoning_item() {
let r = build_response(&meta(), "plain".into(), FinishReason::Stop, None, None);
let v = serde_json::to_value(&r).unwrap();
let items = v["output"].as_array().unwrap();
assert_eq!(items.len(), 1);
assert_eq!(items[0]["type"], "message");
}
/// A file read back must survive the *whole* chain: Responses
/// `function_call_output` -> chat message -> rendered prompt.
///
/// The reported symptom is a model that writes a file correctly,
/// reads it back, and consistently sees errors that are not there.
/// This walks the real path rather than a hand-built chat message,
/// because the translation is the part in between.
#[test]
fn a_file_read_back_survives_the_whole_translation() {
let file = "<!DOCTYPE html>\n<html lang=\"en\">\n\n <script>\n if (a && b || c < d) { s = '#b8905a'; }\n </script>\n</html>\n";
let req: ResponsesRequest = serde_json::from_value(serde_json::json!({
"model": "m",
"input": [
{"type": "message", "role": "user",
"content": [{"type": "input_text", "text": "read it back"}]},
{"type": "function_call", "call_id": "c1", "name": "read",
"arguments": "{\"path\":\"/tmp/index.html\"}"},
// pi sends the tool result as a plain string (its
// `convertToolResultOutput` only produces an array when
// images are involved).
{"type": "function_call_output", "call_id": "c1", "output": file},
{"type": "message", "role": "user",
"content": [{"type": "input_text", "text": "any errors?"}]}
]
}))
.unwrap();
let chat = request_to_chat(req).expect("translate");
// The tool message must carry the file verbatim.
let tool = chat
.messages
.iter()
.find(|m| m.role == "tool")
.expect("a tool message");
let got = match &tool.content {
MessageContent::Text(t) => t.clone(),
other => panic!("tool content is not text: {other:?}"),
};
assert_eq!(got, file, "translation altered the file content");
// And it must survive the render too.
let template = include_str!("../../tests/data/qwen3.8-27b.chat_template.jinja");
let out = crate::harness::chat_template::render_chat_template(
template,
&chat.messages,
&Value::Null,
&Value::Null,
)
.expect("render");
for probe in ["<!DOCTYPE html>", "if (a && b || c < d)", "s = '#b8905a';"] {
assert!(out.contains(probe), "rendered prompt lost {probe:?}");
}
for bad in ["&lt;", "&amp;", "&quot;"] {
assert!(!out.contains(bad), "content HTML-escaped: {bad:?}");
}
}
/// One agentic step must render as ONE assistant turn.
///
/// Responses splits a turn into `reasoning` + `message` +
/// `function_call` items. Emitting a message per item produced
/// *consecutive assistant turns* — one that deliberates and then
/// ends without acting, one that acts with an empty think block.
/// Shown four of those in its own replayed context, the model
/// copied the pattern: it produced a think block and a preamble,
/// then ended the turn without the tool call it had just announced
/// (#296).
#[test]
fn one_agentic_step_becomes_one_assistant_turn() {
let req: ResponsesRequest = serde_json::from_value(serde_json::json!({
"model": "m",
"input": [
{"type": "message", "role": "user",
"content": [{"type": "input_text", "text": "build it"}]},
{"type": "reasoning",
"content": [{"type": "reasoning_text", "text": "planning the file"}]},
{"type": "message", "role": "assistant",
"content": [{"type": "output_text", "text": "Writing index.html:"}]},
{"type": "function_call", "call_id": "c1",
"name": "write", "arguments": "{\"path\":\"a.html\"}"},
{"type": "function_call_output", "call_id": "c1", "output": "wrote 10 bytes"},
{"type": "message", "role": "user",
"content": [{"type": "input_text", "text": "now check it"}]}
]
}))
.unwrap();
let chat = request_to_chat(req).expect("translate");
let roles: Vec<&str> = chat.messages.iter().map(|m| m.role.as_str()).collect();
assert_eq!(
roles,
vec!["user", "assistant", "tool", "user"],
"no two assistant turns may be adjacent: got {roles:?}"
);
let a = &chat.messages[1];
assert!(
a.extra.get("reasoning_content").is_some(),
"reasoning must ride the turn it describes"
);
assert!(
matches!(&a.content, MessageContent::Text(t) if t.contains("Writing index.html")),
"the assistant's own text must survive the fold"
);
let calls = a.extra["tool_calls"].as_array().expect("tool_calls");
assert_eq!(calls.len(), 1);
assert_eq!(calls[0]["function"]["name"], "write");
}
/// Parallel tool calls are several calls on one turn, not several
/// turns.
#[test]
fn consecutive_function_calls_accumulate_on_one_turn() {
let req: ResponsesRequest = serde_json::from_value(serde_json::json!({
"model": "m",
"input": [
{"type": "message", "role": "user",
"content": [{"type": "input_text", "text": "go"}]},
{"type": "message", "role": "assistant",
"content": [{"type": "output_text", "text": "doing both:"}]},
{"type": "function_call", "call_id": "c1", "name": "a", "arguments": "{}"},
{"type": "function_call", "call_id": "c2", "name": "b", "arguments": "{}"}
]
}))
.unwrap();
let chat = request_to_chat(req).expect("translate");
let roles: Vec<&str> = chat.messages.iter().map(|m| m.role.as_str()).collect();
assert_eq!(roles, vec!["user", "assistant"]);
let calls = chat.messages[1].extra["tool_calls"].as_array().unwrap();
assert_eq!(calls.len(), 2, "both calls belong to the same turn");
}
/// Wrap typed items as `input` elements. Most translator tests
/// exercise the typed path; the bare easy-message and unknown-item
/// paths have dedicated tests below.
@@ -1245,6 +1560,8 @@ mod tests {
seed: None,
repetition_penalty: None,
repeat_last_n: None,
presence_penalty: None,
frequency_penalty: None,
previous_response_id: None,
extra: Value::Object(Default::default()),
};
@@ -1271,6 +1588,8 @@ mod tests {
seed: None,
repetition_penalty: None,
repeat_last_n: None,
presence_penalty: None,
frequency_penalty: None,
previous_response_id: None,
extra: Value::Object(Default::default()),
};
@@ -1298,6 +1617,8 @@ mod tests {
seed: None,
repetition_penalty: None,
repeat_last_n: None,
presence_penalty: None,
frequency_penalty: None,
previous_response_id: Some("resp_prev".into()),
extra: Value::Object(Default::default()),
};
@@ -1334,6 +1655,8 @@ mod tests {
seed: None,
repetition_penalty: None,
repeat_last_n: None,
presence_penalty: None,
frequency_penalty: None,
previous_response_id: None,
extra: Value::Object(Default::default()),
};
@@ -1368,6 +1691,8 @@ mod tests {
seed: None,
repetition_penalty: None,
repeat_last_n: None,
presence_penalty: None,
frequency_penalty: None,
previous_response_id: None,
extra: Value::Object(Default::default()),
};
@@ -1417,6 +1742,8 @@ mod tests {
seed: None,
repetition_penalty: None,
repeat_last_n: None,
presence_penalty: None,
frequency_penalty: None,
previous_response_id: None,
extra: Value::Object(Default::default()),
};
@@ -1458,6 +1785,8 @@ mod tests {
seed: None,
repetition_penalty: None,
repeat_last_n: None,
presence_penalty: None,
frequency_penalty: None,
previous_response_id: None,
extra: Value::Object(Default::default()),
};
@@ -1494,6 +1823,8 @@ mod tests {
seed: None,
repetition_penalty: None,
repeat_last_n: None,
presence_penalty: None,
frequency_penalty: None,
previous_response_id: None,
extra: Value::Object(Default::default()),
};
@@ -1603,6 +1934,69 @@ mod tests {
);
}
/// An agentic harness replays every turn verbatim. Rendering all of
/// that thinking back into the prompt made the context mostly
/// self-quotation — 62% of replayed assistant content on a live
/// session — and the model started making shallow passes. Only the
/// newest turn's reasoning is context; the rest is noise the model
/// was never trained to read.
#[test]
fn only_the_newest_turns_reasoning_is_replayed() {
let raw = r#"{
"model": "Qwen/Qwen3.8-27B",
"input": [
{"type": "message", "role": "user", "content": "build it"},
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "old plan"}]},
{"type": "message", "role": "assistant", "content": "wrote index.html"},
{"type": "message", "role": "user", "content": "keep going"},
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "newer plan"}]},
{"type": "message", "role": "assistant", "content": "wrote style.css"}
]
}"#;
let req: ResponsesRequest = serde_json::from_str(raw).expect("parse");
let chat = request_to_chat(req).expect("translate");
let carried: Vec<_> = chat
.messages
.iter()
.filter_map(|m| m.extra.get("reasoning_content").and_then(|v| v.as_str()))
.collect();
assert_eq!(
carried,
["newer plan"],
"older turns must keep their content but lose their thinking"
);
// The turns themselves survive — only the thinking is pruned.
assert_eq!(
chat.messages
.iter()
.filter(|m| m.role == "assistant")
.count(),
2
);
}
/// The continuation case still works: a turn cut off mid-thought is
/// the newest assistant turn, so its reasoning is what survives.
#[test]
fn a_truncated_turn_keeps_its_reasoning_through_a_continue() {
let raw = r#"{
"model": "Qwen/Qwen3.8-27B",
"input": [
{"type": "message", "role": "user", "content": "build it"},
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "24k of design"}]},
{"type": "message", "role": "user", "content": "continue"}
]
}"#;
let req: ResponsesRequest = serde_json::from_str(raw).expect("parse");
let chat = request_to_chat(req).expect("translate");
let carried: Vec<_> = chat
.messages
.iter()
.filter_map(|m| m.extra.get("reasoning_content").and_then(|v| v.as_str()))
.collect();
assert_eq!(carried, ["24k of design"]);
}
/// The control a Responses client actually reaches for (#223): pi-ai
/// emits `reasoning: {"effort": "none"}` for thinking-level *off*.
/// Before this it went nowhere, and a harness that asked for no
@@ -2504,6 +2898,7 @@ mod tests {
output_tokens_details: None,
input_tokens_details: None,
}),
None,
);
assert_eq!(r.status, "completed");
match &r.output[0] {
@@ -2529,7 +2924,7 @@ mod tests {
#[test]
fn build_response_length_yields_incomplete_status() {
let r = build_response(&meta(), "trunc".into(), FinishReason::Length, None);
let r = build_response(&meta(), "trunc".into(), FinishReason::Length, None, None);
assert_eq!(r.status, "incomplete");
}
}
+4
View File
@@ -30,6 +30,7 @@ async fn test_load_default_models_skips_unknown_harness() {
tensor_parallel: None,
devices: None,
sampling: None,
preserve_thinking: None,
},
ModelSpec {
model_id: "model-b".into(),
@@ -38,6 +39,7 @@ async fn test_load_default_models_skips_unknown_harness() {
tensor_parallel: None,
devices: None,
sampling: None,
preserve_thinking: None,
},
];
@@ -98,6 +100,7 @@ async fn test_load_default_models_skipped_on_driver_mismatch() {
tensor_parallel: Some(2),
devices: None,
sampling: None,
preserve_thinking: None,
}];
let activation = ActivationTracker::new(&specs);
let reason = "host NVIDIA driver/library mismatch (userspace NVML 580.159 vs loaded \
@@ -200,6 +203,7 @@ fn qwen_spec() -> ModelSpec {
tensor_parallel: None,
devices: None,
sampling: None,
preserve_thinking: None,
}
}
+1
View File
@@ -54,6 +54,7 @@ async fn test_candle_qwen3_load_unload_lifecycle() {
tensor_parallel: None,
devices: Some(vec![0]),
sampling: None,
preserve_thinking: None,
};
registry
+1
View File
@@ -107,6 +107,7 @@ fn spec(model_id: &str, tp: Option<u32>, quant: Option<&str>) -> ModelSpec {
tensor_parallel: tp,
devices: None,
sampling: None,
preserve_thinking: None,
}
}
+581
View File
@@ -0,0 +1,581 @@
# `qwen4_exp` port spec
Reference specification for `Qwen/Qwen3.8-Flash-Next`, written so that
the decoder can be implemented without re-reading the 131 KB upstream
Python. Every dimension in this document was read from the checkpoint's
own safetensors headers or from the pinned reference source; nothing is
inferred from the model card.
Companion to issue #308 (child A of epic #307).
## Provenance
| what | pinned value |
|---|---|
| reference source | `huggingface/transformers` `src/transformers/models/qwen4_exp/`, commit `fc5c5bde8e656dad91cbf34e61940d984b1c7b91` ("Add Qwen4Exp model", #48337), read at `main` = `5f8ab9bb53ec9e0c9329153d18bd825ff1db80f9` |
| checkpoint | `Qwen/Qwen3.8-Flash-Next`, revision `de4b8e4d43b917e7706784d8bb445c9af86a3540` |
| declared | `model_type: qwen4_exp`, `architectures: [Qwen4ExpForConditionalGeneration]`, `transformers_version: 5.8.0.dev0` |
The checkpoint ships **no remote code**, so the transformers package is
the only authority. Note that `modeling_qwen4_exp.py` has already been
amended once since the model landed (commit `83d46aa2`, "Make gated
delta rule more explicit") — re-diff before trusting this document for
the linear-attention path.
## Measured parameter split
Read from all 131 shard headers (`total_size` 359,999,963,128 B;
1,658 tensors; 1,655 BF16 + 3 I64). This **replaces** the estimate in
#307 — the shape of the estimate was right, the residual bucket was
understated because MTP was counted as dense.
| component | params | share | BF16 | tensors |
|---|---:|---:|---:|---:|
| routed experts | 120,795,955,200 | 67.11% | 241.6 GB | 96 |
| PLE n-gram tables | 51,200,245,760 | 28.44% | 102.4 GB | 128 |
| MTP head | 2,607,150,848 | 1.45% | 5.2 GB | 31 |
| linear attention | 2,086,510,464 | 1.16% | 4.2 GB | 324 |
| hyper-connections | 640,624,640 | 0.36% | 1.3 GB | 387 |
| `embed_tokens` | 635,699,200 | 0.35% | 1.3 GB | 1 |
| `lm_head` | 635,699,200 | 0.35% | 1.3 GB | 1 |
| full attention | 597,694,464 | 0.33% | 1.2 GB | 72 |
| vision tower | 448,931,056 | 0.25% | 0.9 GB | 333 |
| shared experts | 236,052,480 | 0.13% | 0.5 GB | 192 |
| MoE routers | 62,914,560 | 0.03% | 0.1 GB | 48 |
| PLE consumption (conv1d, projections, norms) | 32,839,683 | 0.02% | 0.1 GB | 7 |
| QSA indexers | 19,663,872 | 0.01% | 0.04 GB | 36 |
| **total** | **179,999,981,459** | | **360.0 GB** | **1,658** |
The offload plan in #307 survives contact with the measurement:
- **sparse-access, offloadable: 171.996 B (95.55%)** — routed experts
(10 of 512 per token) plus PLE tables (a few rows per token, one
layer).
- **always-resident text path: 4.94 B (2.74%)** — everything above
minus MTP and vision. 9.9 GB at BF16.
- MTP (2.61 B) is only resident if speculative decoding is on (#313);
vision (0.45 B) only if the vision tower is loaded (#314).
## Skeleton
```
hidden_size 2560 num_hidden_layers 48
head_dim 256 num_attention_heads 24
num_key_value_heads 2 vocab_size 248,320
max_position_embeddings 262,144
layer_types 36 linear_attention + 12 full_attention,
full_attention_interval 4 → layers 3,7,11,…,47
num_experts 512 num_experts_per_tok 10
moe_intermediate_size 640 shared_expert_intermediate_size 640
norm_topk_prob true (default; absent from config.json)
hc_count 4 hc_lowrank 320
ple_layer_ids [2] (one-indexed → zero-indexed layer 1)
rms_norm_eps 1e-6 hidden_act silu
output_gate_type sigmoid mamba_ssm_dtype float32
```
`layer_types` is authoritative; `full_attention_interval` is a hint. The
same convention as `arch/qwen3_5/`.
### Top-level dataflow
```
input_ids ──► embed_tokens ──► x [B,T,2560]
│
repeat(1,1,hc_count) ──► h [B,T,10240] 4 identical streams
│
┌────────────────────────▼───────────────────────────────┐
│ for layer in 0..48: │
│ if layer == 1: h = h + PLE(h, input_ids) │ [B,T,10240]
│ h = decoder_layer(h) │
└────────────────────────┬───────────────────────────────┘
│
hyper_connection_mixer(h) (use_combine=False)
│
▼ [B,T,2560]
lm_head
```
Two things to note, because both differ from every arch we serve:
1. **The inter-layer residual is 10240 wide, not 2560.** Four parallel
residual streams, initialised as four copies of the token embedding.
2. **There is no final `model.norm`.** The `hc_norm` inside
`hyper_connection_mixer` is the only normalisation before `lm_head`.
Do not go looking for a missing tensor — the checkpoint has
`hyper_connection_mixer.{hc_norm,input_mix_weight_down,input_mix_weight_up}`
and no `norm.weight`.
### Decoder layer
There is **no `input_layernorm` and no `post_attention_layernorm`**. The
hyper-connection block absorbs both roles: it normalises, mixes the four
streams down to one 2560 vector for the sublayer, and scatters the
sublayer's output back across the four streams.
```
h [B,T,10240]
│
├─ (layer 1 only) h = h + PLE(h, ple_input_ids)
│
├─ (x, h_saved, inj) = attn_hyper_connection(h) x:[.,2560] inj:[.,4]
│ x = linear_attn(x) | x = self_attn(x) → [.,2560]
│ h = h_saved + (x ⊗ inj).flatten(-2) → [.,10240]
│
└─ (x, h_saved, inj) = mlp_hyper_connection(h)
x = sparse_moe(x) → [.,2560]
h = h_saved + (x ⊗ inj).flatten(-2) → [.,10240]
```
`x ⊗ inj` is the outer product `x.unsqueeze(-2) * inj.unsqueeze(-1)`,
giving `[.,4,2560]` — the sublayer output scaled by a per-stream weight
before being added into each stream.
## Component specs
### 1. Hyper-connections (`Qwen4ExpTextGatedResidual`)
Tensors, per instance:
```
hc_norm.weight (10240,)
input_mix_weight_down.weight (320, 10240)
input_mix_weight_up.weight (10240, 320)
block_inject_weight.weight (4, 10240) # absent on the final mixer
```
Instances: `layers.{i}.attn_hyper_connection`,
`layers.{i}.mlp_hyper_connection` for all 48 layers, plus
`model.language_model.hyper_connection_mixer` and
`mtp.hyper_connection_mixer` (both `use_combine=False`, i.e. no
`block_inject_weight`). 387 + 3 tensors.
Forward, exactly:
```
hn = hc_norm(h) # grouped RMSNorm, see §8
w = silu(input_mix_weight_down(hn) / hc_count) # [.,320]
w = sigmoid(input_mix_weight_up(w)) # [.,10240]
w = w.unflatten(-1, (4, 2560))
x = (w * hn.unflatten(-1, (4, 2560))).mean(dim=-2) # [.,2560]
inj = 2 * sigmoid(block_inject_weight(hn) / hc_count) # [.,4]
return x, h, inj # note: h, NOT hn
```
Three details that are easy to get wrong and produce fluent-but-worse
output rather than a crash:
- The **division by `hc_count` (4)** happens inside both gates, before
the nonlinearity.
- The mix uses `.mean()` over the four streams, not `.sum()`.
- The residual returned for the skip connection is the **un-normalised**
`h`. Only the mixing and injection paths see `hn`.
### 2. PLE — per-layer embedding over hashed n-grams
Applied at zero-indexed **layer 1** (`ple_layer_ids: [2]` is
one-indexed; upstream computes
`config.ple_layer_ids.index(layer_idx + 1)`, and the checkpoint
confirms it — the tensors live at
`model.language_model.layers.1.ple.*`).
#### Table geometry
```
ngram_size 3 heads_per_ngram 8
ngram_heads = (ngram_size - 1) * heads_per_ngram = 16
ple_embed_dim 2560 → head_dim_per_ngram = 2560 / 16 = 160
ngram_vocab_size_base 20,000,000
make_ngram_vocab_size_divisible_by 128
split_ngram_parts 128
```
Head *h* (0..15) gets vocab size = the (h+1)-th prime strictly greater
than 19,999,999; offsets are the running cumulative sum. Total padded up
to a multiple of 128 gives **320,001,536 rows × 160 dims**, sharded as
128 × `(2500012, 160)` tensors named
`ple_embedding.ngram_embedding.shard_{0..127}.weight`. The shards are a
**flat split of the concatenated table**, not a per-head split —
concatenate along dim 0 in shard-index order and the global row index
from the hash indexes straight into it.
Do **not** reimplement the prime search or the splitmix64 multiplier
derivation. All three derived quantities ship in the checkpoint as I64
buffers and should simply be loaded:
```
ple_embedding.ngram_heads_offsets (16,)
ple_embedding.ngram_heads_vocab_sizes (16,)
ple_embedding.layer_multipliers (3,)
```
#### Hashing
Let `tok[s]` be the input ids shifted right by `s` (s = 0,1,2) and
`m = layer_multipliers`. For n-gram order *n* ∈ {2, 3}, occupying heads
`[(n-2)*8, (n-2)*8 + 8)`:
```
mixed = tok[0] * m[0]
for p in 1..n: mixed ^= tok[p] * m[p] # XOR, int64
row[head] = (mixed mod vocab_sizes[head]) + offsets[head]
```
so each token produces 16 row indices, gathers 16 × 160 values and
flattens to a 2560-wide embedding. Arithmetic is int64 and the
multipliers are ~10^13, so the products overflow int32 by a wide margin
— this must be done in 64-bit.
There is **no unigram head**: with `ngram_size = 3` the loop runs
n = 2,3 only. `layer_multipliers` has three entries because the trigram
needs three shift multipliers.
#### Shifting is segment-aware
`_shift_right_ignore_eos` does not shift across an EOS token. Within
each EOS-delimited segment, positions closer to the segment start than
`shift` get filled with `eos_token_id` (248044) instead of borrowing
the previous document's tokens. Additionally
`Qwen4ExpTextModel.forward` replaces padded positions in
`ple_input_ids` with EOS before the layer is called. Getting this wrong
is invisible on a single-turn prompt and wrong on anything batched or
multi-document.
#### Consumption
```
key_proj.weight (10240, 2560) value_proj.weight (2560, 2560)
norm_key.weight (10240,) norm_query.weight (10240,)
norm_conv.weight (10240,) conv1d.weight (10240, 1, 4)
```
```
e = ngram_lookup(input_ids) # [.,2560]
k = norm_key(key_proj(e)).unflatten(-1, (4, 2560))
v = value_proj(e) # [.,2560]
q = norm_query(h).unflatten(-1, (4, 2560)) # h is the 10240 stream
g = (k * q).sum(-1, keepdim=True) / sqrt(2560) # [.,4,1]
g = sign(g) * sqrt(clamp_min(|g|, 1e-6)) # signed sqrt
gv = sigmoid(g) * v.unsqueeze(-2) # [.,4,2560]
out = gv.flatten(-2) + short_conv(norm_conv(gv.flatten(-2)))
```
`short_conv` is a **depthwise, dilated** Conv1d: `groups = 10240`,
`kernel_size = ple_conv_kernel_size = 4`, **`dilation = ngram_size = 3`**,
no bias, followed by SiLU. Its receptive field is
`(4 - 1) * 3 = 9` positions of left context, which is also the cached
conv-state length. The dilation is the part most likely to be dropped
by accident — it is not a plain causal conv.
The three norms are grouped RMSNorm (§8) over 10240 in groups of 2560.
#### Cache state
PLE needs two pieces of per-request state, both stored upstream as
extra "conv states" on layer 1:
| slot | shape | contents |
|---|---|---|
| `conv_states[1]` | `(B, 10240, 9)` | short-conv left context |
| `conv_states[2]` | `(B, 2)` | previous `ngram_size - 1` input **ids**, EOS-initialised |
The second is token ids, not activations. Reset both when a request's
KV cache is cleared.
### 3. QSA sparse attention indexer
On the 12 full-attention layers and the MTP layer.
```
indexer_n_heads 4 indexer_kv_heads 1 indexer_head_dim 128
indexer_budget 2048 indexer_compress_ratio 4
block_topk = indexer_budget / indexer_compress_ratio = 512
index_qk_proj.weight (640, 2560) # 4*128 query + 1*128 key, fused
q_layernorm.weight (128,)
k_layernorm.weight (128,)
```
`indexer_budget` is in **tokens**; 512 *blocks* of 4 tokens are
selected. `index_qk_proj` is a fused projection: the first 512 output
channels are the 4 query heads, the last 128 are the single index key.
Per query position:
1. `q = q_layernorm(q); q = RoPE(q, pos)` — RoPE at the query's own
position, using the same partial-rotary cos/sin as the main
attention (64 of 128 dims rotated).
2. The raw index key (**pre-norm, pre-RoPE**) is appended to a
**separate indexer KV cache**, one head × 128 dims.
3. Take the causally-visible key positions. Group them into
`floor(n_visible / 4)` complete blocks of 4 **consecutive visible
positions**.
4. Pool: `mean` over the 4 keys **in float32**, then `k_layernorm`, then
RoPE at the **first token position of the block**.
5. Score: `relu(q @ pooled_k^T).sum(over the 4 index heads) / sqrt(128)`.
The ReLU is before the head sum — a signed sum is a different model.
6. Select the top `min(512, n_blocks)` blocks; expand back to their
4 token positions each.
7. **Unconditionally append the tail** — the `n_visible mod 4` positions
that did not form a complete block are always attended, never
scored.
8. The selection becomes a mask that is ANDed (bool) or added (float)
onto the causal mask before the main attention.
Consequences for the port:
- **The full KV cache is still materialised for every position.** #307's
24 KiB/token arithmetic holds. QSA bounds the *attention compute*, not
the cache.
- **There is a second cache.** 12 layers × 1 head × 128 dims. At BF16
that is 3 KiB/token on top of the main 24 KiB/token — #307's KV table
omitted it. llama.cpp's own accounting on a 2-GPU box reports
6144 MB main + 2304 MB indexer at the full 262,144 window, i.e. the
indexer is a real ~27% surcharge, not a rounding error.
- Below the budget (fewer than 2048 visible tokens) selection is
effectively a no-op: every complete block gets selected and the tail
is appended, so the result is dense causal attention. **Short-prompt
parity against a dense implementation is therefore a valid test**, and
is the cheapest correctness gate available.
- The reference is a Python double loop over batch × query position. It
is a correctness oracle only — a usable implementation must batch the
block pooling and score in one pass, and gather rather than build a
dense `[T, kv_len]` mask.
### 4. Full attention
```
q_proj.weight (12288, 2560) # 24 heads × 256 × 2 — query and gate
k_proj.weight (512, 2560) # 2 kv heads × 256
v_proj.weight (512, 2560)
o_proj.weight (2560, 6144)
q_norm.weight (256,) k_norm.weight (256,)
```
Mechanically identical to `arch/qwen3_5/full_attn.rs`: GQA 24:2, per-head
RMSNorm on q and k, a sigmoid output gate carried in the widened
`q_proj`, scaling `head_dim^-0.5`.
The gate split is **per head, not a flat halving**: reshape to
`(..., n_heads, 2 * head_dim)` and chunk on the last axis, so within
each head the first 256 channels are the query and the last 256 are the
gate. Splitting the 12288 vector in half is a different and wrong
permutation.
The gate is applied to the attention output **before** `o_proj`:
`attn_out = attn_out * sigmoid(gate)`.
### 5. Linear attention (GatedDeltaNet)
```
linear_num_key_heads 16 linear_key_head_dim 128
linear_num_value_heads 48 linear_value_head_dim 128
linear_conv_kernel_dim 4 → conv_dim = 16*128*2 + 48*128 = 10240
in_proj_qkv.weight (10240, 2560) in_proj_z.weight (6144, 2560)
in_proj_a.weight (48, 2560) in_proj_b.weight (48, 2560)
conv1d.weight (10240, 1, 4) A_log (48,) dt_bias (48,)
norm.weight (128,) out_proj.weight (2560, 6144)
```
**Tensor names and layout are identical to Qwen3.6's split (non-fused)
form**, which `arch/qwen3_5/linear_attn.rs` already loads — it branches
on the presence of `in_proj_qkvz.weight` and takes the split path
otherwise. Same delta rule, same conv, same state shapes.
One real difference: the output gated RMSNorm's activation is
`config.output_gate_type or config.hidden_act`, and this config sets
**`output_gate_type: sigmoid`**. Our `Qwen3_5RmsNormGated` hardcodes
SiLU. This must become configurable, or the linear-attention output is
wrong on all 36 layers.
`mamba_ssm_dtype: float32` — upstream carries the recurrent state in
f32. We default the equivalent GatedDeltaNet state to a bf16 round-trip
(#284, `NEURON_GDN_STATE_F32` off). **A port must not silently inherit
our default against upstream's explicit choice**: default this arch to
f32 state and measure the cost, rather than assuming #284's tradeoff
transfers.
### 6. MoE
```
mlp.gate.weight (512, 2560)
mlp.experts.gate_up_proj (512, 1280, 2560)
mlp.experts.down_proj (512, 2560, 640)
mlp.shared_expert.{gate,up}_proj.weight (640, 2560)
mlp.shared_expert.down_proj.weight (2560, 640)
mlp.shared_expert_gate.weight (1, 2560)
```
```
shared = sigmoid(shared_expert_gate(x)) * shared_expert(x)
probs = softmax(gate(x), dim=-1, dtype=f32) # over all 512
w, idx = topk(probs, 10)
w = w / w.sum(-1, keepdim=True) # norm_topk_prob = true
out = sum_k w_k * expert_{idx_k}(x) + shared
```
Routing arithmetic matches `arch/qwen3_5/moe.rs` exactly. The **storage
layout does not**: this checkpoint ships experts as fused 3D tensors,
where ours loads per-expert `mlp.experts.{i}.{gate,up,down}_proj`. In
`gate_up_proj` the first 640 output rows are the gate and the last 640
are the up projection. The fused layout is the better one for a grouped
GEMM; the loader needs a new branch, not a new algorithm.
Sparsity is extreme: 10 of 512, `moe_intermediate_size` only 640.
Per token, 10 × (1280 + 2560) × 2560 ≈ 98.3 M params ≈ 2.36 B across
48 layers — the number that drives every placement decision in #309.
### 7. Rotary — interleaved M-RoPE
```
rope_theta 1e7 partial_rotary_factor 0.25 mrope_section [11,11,10]
rotary_dim = head_dim * 0.25 = 64 → 32 inverse frequencies
```
`sum(mrope_section) = 32` = the number of frequencies. The interleave
assigns frequency index *i* to axis T, H or W:
```
freqs = freqs[0] # start from temporal
freqs[..., 1:33:3] = freqs_H[..., 1:33:3] # 11 indices: 1,4,…,31
freqs[..., 2:30:3] = freqs_W[..., 2:30:3] # 10 indices: 2,5,…,29
```
i.e. the layout is `T,H,W,T,H,W,…` with temporal keeping the remainder,
rather than three contiguous chunks. Rotation is `rotate_half` over the
**first 64 of 256** head dims; the remaining 192 pass through unrotated.
**This is byte-for-byte the configuration `arch/qwen3_5/rope.rs`
already implements** (`head_dim 256`, `partial_rotary_factor 0.25`,
`mrope_section [11,11,10]`, `mrope_interleaved true`) — the module's
own tests pin exactly these values. Verify the index sets against the
slices above, but expect this item to be free.
`rope_theta` differs between the main model (1e7) and the MTP layer's
config block (also 1e7 here — `mtp.rope_theta: 10000000`); do not
assume they always agree.
### 8. Norms
Two conventions coexist in the same file, and mixing them up is silent:
| variant | scale | used by |
|---|---|---|
| `Qwen4ExpTextRMSNorm` | `(1.0 + weight)` | everything except the GDN output |
| `Qwen4ExpTextRMSNormGated` | `weight` | GatedDeltaNet output gate only |
`arch/qwen3_5/rmsnorm.rs` already implements both with these exact
conventions — this is a solved problem in our tree.
**What is new is `group_size`.** `Qwen4ExpTextRMSNorm(dim, group_size)`
reshapes the last axis to `(-1, group_size)`, normalises each group
independently, and flattens back. Every 10240-wide norm in this model
(`hc_norm`, `norm_key`, `norm_query`, `norm_conv`,
`mtp.pre_fc_norm_hidden`) uses `group_size = 2560`, i.e. four
independent RMS normalisations, not one over 10240. A single norm over
the full 10240 vector compiles, runs, and is wrong.
Variance is computed in f32 and the `(1 + w)` shift is applied in f32
before the multiply — same reasoning as the note already in
`rmsnorm.rs`.
### 9. MTP head
```
mtp.pre_fc_norm_embedding.weight (2560,) # plain RMSNorm
mtp.pre_fc_norm_hidden.weight (10240,) # grouped, group_size 2560
mtp.fc_embedding.weight (2560, 2560)
mtp.fc_hidden.weight (2560, 2560)
mtp.hyper_connection_mixer.* # use_combine = False
mtp.layers.0.* # a complete full_attention
# decoder layer: 512-expert
# MoE, shared expert, QSA
# indexer, both hyper-
# connections
```
`mtp_num_hidden_layers: 1`, `mtp.hybrid: true`, `mtp.layer_types:
["full_attention"]`, `mtp_use_dedicated_embeddings: false` (so it reuses
`embed_tokens` and `lm_head`).
**Transformers does not implement the MTP head.**
`Qwen4ExpPreTrainedModel._keys_to_ignore_on_load_unexpected = [r"^mtp.*"]`
— the weights are dropped on load. So for #313 the upstream reference is
*not* usable, and the authorities are instead llama.cpp PR #27836
(`--spec-type draft-mtp`) and vLLM's `speculative-config
{"method": "mtp"}` path. The tensor names above and the two `fc_*`
projections make the intended wiring clear (project the next token's
embedding and the accepted hidden state, sum, run one decoder layer,
reuse `lm_head`), but the accept rule and the proposal depth must come
from one of those two implementations, not from this document.
`mtp.layers.0` carries its own full 512-expert MoE — 2.61 B params,
5.2 GB at BF16. It is not free to keep resident and its placement is
part of #310's problem.
### 10. Vision tower
```
depth 27 hidden_size 1152 intermediate_size 4304 num_heads 16
patch_size 16 spatial_merge_size 2 temporal_patch_size 2
out_hidden_size 2560 num_position_embeddings 2304
hidden_act gelu_pytorch_tanh deepstack_visual_indexes []
```
448.9 M params. Smaller than the 27B's tower and with deepstack merging
disabled. Out of scope for the text path (#314); `language_model_only`
is a config knob.
## Reuse boundary against `arch/qwen3_5/`
| piece | verdict |
|---|---|
| `rope.rs` | **reuse unchanged.** Identical `head_dim`/`partial_rotary_factor`/`mrope_section`/`mrope_interleaved`. Verify the interleave index sets, expect no change. |
| `rmsnorm.rs` | **extend.** Both `(1+w)` and gated conventions already correct. Add `group_size`; make the gated variant's activation configurable (sigmoid here, SiLU in Qwen3.6). |
| `linear_attn.rs` | **reuse, two deltas.** Names and layout already match the split form. Switch the output-gate activation to sigmoid; default the recurrent state to f32 per `mamba_ssm_dtype`. |
| `full_attn.rs` | **extend.** Same GQA + q/k norm + widened-`q_proj` output gate. Add the QSA indexer and its separate cache. |
| `moe.rs` | **extend loader only.** Routing arithmetic identical; add a fused-3D `experts.{gate_up,down}_proj` branch beside the per-expert one. |
| `mlp.rs` | **reuse** for the shared expert. |
| `decoder.rs` | **rewrite.** Hyper-connections replace both layernorms and the residual is 4× wide. |
| `mod.rs` (config, load, forward) | **new sibling arch.** Config field vocabulary overlaps heavily; the model-level plumbing (10240 stream, no final norm, PLE at layer 1, MTP) does not. |
| `snapshot.rs` | **extend.** Batched decode needs snapshot support for the new per-request state: PLE's two conv slots and the QSA indexer cache, on top of the existing conv + recurrent state. |
| `vision.rs` | **likely reuse with a new config.** Deferred (#314). |
Net: the linear-attention half of the model — 36 of 48 layers and the
piece that took the most work in `qwen3_5` — carries over close to
unchanged. The genuinely new maths is hyper-connections (§1), PLE (§2)
and QSA (§3).
## Correctness gates, in the order they should be built
Each of these fails loudly, unlike the model as a whole, which fails
fluently.
1. **Grouped RMSNorm** against a hand-computed 4×2560 case.
2. **Hyper-connection block** in isolation: feed a known `h`, check
`x`, `inj` and the reconstructed `h` against a torch trace. Do this
before stacking anything on it — a wrong stream mix still produces
plausible text.
3. **PLE hashing** against the reference for a short id sequence,
including one that crosses an EOS, in int64.
4. **QSA below budget** — a prompt under 2048 tokens must give
bit-comparable logits to dense causal attention through the same
layer.
5. **QSA above budget** against the reference double loop on a small
synthetic case (short `indexer_budget`, so the selection is actually
exercised).
6. Full-model logits parity on a short prompt against a reference trace.
## Open questions this document does not settle
- **The accept rule and proposal depth for the MTP head** (§9). Needs
llama.cpp #27836 or vLLM, not transformers.
- **Whether QSA can engage flash-attn at all.** The selection is a
gather, our flavour builds are per-sm, and flash-attn is only wired on
some (#95). Measure; do not assume.
- **Whether the f32 recurrent state (§5) costs anything measurable**
here, given 36 linear layers rather than Qwen3.6's proportion.
- **How `kv_budget_mb` should account for the indexer cache** (§3) — it
is a second, differently-shaped allocation and `derive_limit` does not
know about it. Feeds #310 and #315.
+1 -1
View File
@@ -35,7 +35,7 @@ default_endpoint = "helexa"
name = "helexa"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.6-27B"
default_model = "Qwen/Qwen3.8-27B"
max_tokens = 8192
# Compaction kicks in when the rolling history grows past this token
# budget. Set to your model's context window. Disable by removing
+1 -1
View File
@@ -23,7 +23,7 @@ listen = "0.0.0.0:8088"
#
# [aliases]
# "helexa/small" = "Qwen/Qwen3-8B"
# "helexa/balanced" = "Qwen/Qwen3.6-27B"
# "helexa/balanced" = "Qwen/Qwen3.8-27B"
# -- Downstream cortexes -------------------------------------------------
# Each [[cortexes]] entry is an operator-run cortex the router may dispatch
+37
View File
@@ -0,0 +1,37 @@
# ─────────────────────────────────────────────────────────────────────
# MANAGED BY CI — .gitea/workflows/deploy.yml is the only writer.
#
# Edits made directly to the deployed copy on gallumbits are reverted by
# the next deploy, and until then the fleet runs something no commit
# describes. Change it here, commit, let the pipeline ship it.
# ─────────────────────────────────────────────────────────────────────
# helexa-router — federation data plane on gallumbits.
#
# Tracked despite living beside the secret-bearing service configs,
# because it holds no secrets: a listen address, the product-tier
# aliases, and a mesh endpoint. Being untracked cost an outage on
# 2026-08-27 — `helexa/balanced` still resolved to Qwen/Qwen3.6-27B
# after that model had been retired from the catalogue, so every
# authenticated web-chat turn evicted beast's resident flagship and
# cold-loaded a model nobody had asked for. Nothing could diff the two
# files, so nothing did.
#
# `script/check-alias-targets.py` now fails CI if any alias below names
# a model that models.toml does not define.
[router]
listen = "0.0.0.0:8088"
poll_interval_secs = 10
region = "eu-central"
# Product tiers → real model ids (#166). The SPA sends the tier names.
[aliases]
"helexa/small" = "Qwen/Qwen3-8B"
"helexa/balanced" = "Qwen/Qwen3.8-27B"
# hanzalova operator cortex, plaintext over the WireGuard mesh (the mesh
# is the trust boundary; no TLS pin needed for http://).
[[cortexes]]
name = "hanzalova"
endpoint = "http://hanzalova.internal:31313"
region = "eu-central"
@@ -44,6 +44,70 @@ change.
VRAM fragmentation has had time to accumulate. It is a signal to restart
that neuron, not an automatic action.
## secrets.toml
Credentials do not belong in the file CI deploys. Put them in a
`secrets.toml` beside the main config and the service will merge it
last, so its values win:
```
/etc/cortex/cortex.toml structure — deployed by CI, safe to diff
/etc/cortex/secrets.toml credentials — yours, root-only, never in git
```
```toml
# /etc/cortex/secrets.toml (root:cortex 0640)
[[entitlements.keys]]
key = "sk-…"
account_id = "acct-a"
key_id = "a"
[upstream]
bearer = "…"
```
`helexa-upstream` and `helexa-angels` read a `secrets.toml` beside their
own configs the same way — the DB URL, JWT secret and SMTP URL belong
there.
**An absent `secrets.toml` is not an error.** A single-file deployment
keeps working exactly as before, so this is opt-in per host and can be
adopted one service at a time.
### Why bother
Because a config CI cannot write is a config no check can defend, and
the cost of that is not hypothetical. `helexa-router.toml` was kept out
of git for sitting beside secret-bearing files. Nothing could diff it
against the catalogue, so its `helexa/balanced` alias went on pointing
at a model that had been retired — and since both models were
deliberately given equal residency priority so they could displace each
other, every authenticated chat turn evicted the resident flagship to
cold-load the retired one. Four of those overlapped and the node ended
up holding 32 GB of GPU memory with no model loaded and nothing
serving.
Splitting secrets from structure is what lets the structure be
version-controlled, reviewed and checked. The ordering matters too: the
operator's file merges last, so a deploy can never replace a live
credential with a placeholder.
### Moving existing secrets
One-time, per host, and reversible — the service reads both files, so
you can move a value and restart before moving the next:
1. Create `secrets.toml` beside the config, owned `root:<service>` mode
`0640` (`0600` where the service runs as root).
2. Move the credential blocks across. Leave everything else in place.
3. Restart the service and confirm it still authenticates — for cortex,
a request with a real key should be billed rather than treated as
anonymous.
4. Delete the moved lines from the main file only once step 3 passes.
Until step 4 the value exists in both files, which is safe: the
operator's copy wins.
## models.toml
The catalogue says what each model needs and where it may run:
@@ -78,7 +142,7 @@ Clients should not hardcode model ids. Aliases let them ask for a role:
[aliases]
"helexa/small" = "Qwen/Qwen3-1.7B"
"helexa/balanced" = "Qwen/Qwen3-8B"
"helexa/large" = "Qwen/Qwen3.6-27B"
"helexa/large" = "Qwen/Qwen3.8-27B"
"helexa/image" = "Tongyi-MAI/Z-Image-Turbo"
```
@@ -58,6 +58,32 @@ Two things that bite:
- **Cold loads are disk-bound.** A 50 GB model off spinning rust is a
very different experience from the same model on NVMe.
### Where weights come from
By default a bare `org/name` id is fetched from HuggingFace. Operators
running a mirror, or serving models from more than one place, can
declare sources explicitly:
```toml
[harness.candle]
default_source = "huggingface"
[harness.candle.sources.huggingface]
endpoint = "https://huggingface.co"
auth_env = "HF_TOKEN" # env var name, never the token itself
cache_dir = "/archive/llm" # optional; per-source cache
```
`auth_env` names an environment variable rather than holding a token, so
credentials stay out of the config file — worth keeping to even when the
file is not obviously shared, because config is exactly what gets copied
between hosts.
Give a source its own `cache_dir` when two sources can serve the same
`org/name`. The cache tree is keyed on the model id alone, so sharing a
directory between sources that disagree about what `org/name` means will
collide on disk.
## Default models
Models listed as defaults load when the daemon activates, so the host
@@ -66,7 +92,7 @@ load cost:
```toml
[[default_models]]
model_id = "Qwen/Qwen3.6-27B"
model_id = "Qwen/Qwen3.8-27B"
harness = "candle"
quant = "q6k"
tensor_parallel = 2
@@ -91,10 +117,17 @@ Each loaded model has bounded admission control:
[harness.candle.admission]
max_in_flight = 8 # concurrent requests actually running
max_queue_depth = 8 # waiting before rejection
max_wait_secs = 30 # how long a queued request waits
max_wait_secs = 30 # how long a queued request waits for a slot
kv_max_wait_secs = 300 # how long it waits for enough KV budget
max_per_principal = 2 # fair share, so one caller cannot monopolise
```
`kv_max_wait_secs` is deliberately much longer than `max_wait_secs`.
The first waits for a slot to turn over; the second waits for a whole
long-context sequence to finish, which on a 27B can be minutes. A
request that needs KV budget nobody has yet is not a request to reject
in thirty seconds.
Over-capacity requests are rejected promptly with `429`/`503` and a
`Retry-After` rather than being allowed to queue invisibly. A fast
rejection a client can retry is far better than a request that hangs
@@ -102,8 +135,34 @@ until something times out.
`max_in_flight` above 1 enables batched decode on architectures that
support it, so concurrent requests share a decode step instead of
serialising. Raise it on big cards, leave it at 1 on small ones — the
KV cache for several concurrent long contexts is itself substantial.
serialising.
**`max_in_flight` does not partition the KV budget.** KV is a shared
pool, and each request reserves what its own prompt actually needs when
it is admitted — which is why `kv_max_wait_secs` above exists at all. A
request larger than what is currently free waits for bytes to come back
rather than being handed a fixed 1/N slice, and a single long session
can legitimately hold most of the pool.
So the slot count and the context window are separate questions. What
bounds a busy moment is the pool: divide `kv_budget_mb` by the model's
KV cost per token (`GET /health` reports both) to see how many tokens
are available to share. Many ordinary chat turns fit at once; a few very
long agentic sessions will queue on bytes.
Size `max_in_flight` against the concurrency you actually want to serve
concurrently, not against context. Anything above it queues on admission
with a `Retry-After` rather than being refused, so setting it too low
converts capacity you have into latency your callers feel. Setting it
very high mainly costs batch-step efficiency, since the engine pads to
the widest active sequence.
Measure before assuming you need a high number — but measure the traffic
you will have, not the traffic you had. On one production host, seven
days of spans showed 89.6% of busy time at a single request in flight;
that host's chat tier was at the time being routed to a *different*
model, and the figure stopped describing it the moment that was
corrected.
`max_per_principal` is what stops one busy client consuming the whole
model.
@@ -150,7 +209,8 @@ reservation is doing its job, not that the model is overloaded.
neuron refuses work it cannot finish rather than crashing partway:
- a **static floor** of free VRAM below which prefill is refused
(`NEURON_MIN_FREE_VRAM_MB`, default 1500)
(`NEURON_MIN_FREE_VRAM_MB`, or `min_free_floor_mb` under
`[harness.candle.context_limit]`; default 1500)
- a **length-aware check** that estimates the KV cache the request will
actually need and refuses if it will not fit
@@ -159,15 +219,207 @@ can pass the static floor and still be impossible, because KV grows with
prompt *and* generation length. See [context
limits](/docs/operating/context-limits) for how the numbers relate.
`activation_headroom_mb` reserves VRAM per card for prefill activations
on top of the weights and the KV cache, before the context ceiling is
computed. Raise it if prefill OOMs on a host whose steady-state
occupancy looks fine — activations are transient and do not show up in
the resident figure.
A host that cannot currently serve a model reports that on `/health`,
and cortex stops routing to it — an unservable replica is not a routing
target.
## Image models
```toml
[harness.candle.image]
te_device = "cpu" # where the text encoder runs
te_resident = true # keep it loaded between generations
max_dim = 2048 # resolution ceiling per side
```
The text encoder is the placement decision. On CPU the ~8 GB encoder
never touches the GPU — prompt encoding is one short forward and the
features are tiny — which is what lets a 24 GB card serve a BF16 DiT at
all. Move it to `cuda` only on a card with room to spare.
`te_resident` trades memory for latency: resident costs ~16 GB of system
RAM on CPU (or ~8 GB of VRAM on CUDA) and saves a few seconds per
request. CPU defaults to resident, CUDA to rebuilding per request, which
are the right defaults for where each is scarce.
`max_dim` rejects oversized requests **before admission**, so an
impossible resolution costs nothing. That matters more than it sounds:
an OOM on the device poisons the worker context, so refusing early is
the difference between one rejected request and a model that needs
restarting.
## Prompt caps
`NEURON_MAX_PROMPT_TOKENS` sets a hard ceiling; larger prompts are
rejected before any GPU work, so a runaway client costs nothing.
## Sampling
Every model publishes its own sampling defaults in
`generation_config.json`, and a caller can override them per request.
Between those two sits an operator override, for the cases where the
model's published defaults are wrong for how you serve it:
```toml
[[default_models]]
model_id = "Qwen/Qwen3.8-27B"
[default_models.sampling]
temperature = 0.6
presence_penalty = 1.0
```
Precedence is **request > operator > model > built-in**. Anything you
leave unset changes nothing, so an empty table behaves exactly as no
table at all.
Set the same values in the gateway's `models.toml` under
`[models.sampling]`. A model that is cold-loaded on demand gets its
sampling from the catalogue, not from this file, so the two must agree
or the same model samples differently depending on how it arrived —
`script/check-config-consistency.py` fails the deploy if they diverge.
### Which knob for which symptom
| symptom | knob |
|---|---|
| Structurally malformed output — truncated JSON, unbalanced braces | `temperature` |
| The think block restates the same passage over and over | `presence_penalty` |
| Common tokens over-represented across a long answer | `frequency_penalty` |
| Short immediate loops — "no, no, no…" | `repeat_penalty` |
| You need two runs to be comparable | `seed` |
**`temperature`** is the blunt one. Qwen3.8-27B publishes `1.0`, which
measured a 20% structural-defect rate on long agentic turns against 0/60
at `0.6`; the fleet therefore overrides it. Note that `0.6` is also what
Qwen recommend for their thinking variants, so this is not fighting the
model.
**`presence_penalty`** and **`frequency_penalty`** are scored over the
*whole* generated sequence. They are the ones that matter for reasoning
models, and Qwen name `presence_penalty` (0–2, 1.5 when severe) as the
remedy for the endless repetition their models fall into during long
thinking. Both default to `0.0`, which is off.
**`repeat_penalty`** (default `1.1`) and **`repeat_last_n`** (default
`64`) are the older, *windowed* form: the penalty only sees the last
`repeat_last_n` tokens. That is fine for chat and close to useless for a
think block — a 20,000-token deliberation restating a passage 5,000
tokens later is entirely outside a 64-token window. Measured across
recorded sessions, **86% of think blocks over 60k characters contained
verbatim repeated sentences**, one of them repeating 76% of its own
content.
Widening the window is the obvious move and the wrong one: the penalty
is token-level, and source code legitimately repeats tokens constantly
(`const`, `function`, brace runs), so a wide window at a meaningful
penalty degrades exactly the output you care about. Reach for
`presence_penalty` first — what recurs is passages, not tokens.
**`seed`** pins the sampler's RNG for every request to this model. It
exists because parameter search needs a control and the clients that
would otherwise pin it often cannot. Leaving it set is a product
decision, not just a testing one: every identical prompt then returns an
identical answer to every caller, which is predictable and cacheable but
removes the variation a user may expect from retrying.
## Reasoning
```toml
[default_models.sampling]
# nothing here — reasoning settings live in their own places, below
```
**`preserve_thinking`** controls whether prior turns keep their think
blocks when the conversation is re-rendered. It is the model's own chat
template control, not ours:
```toml
[[default_models]]
preserve_thinking = false # keep reasoning only for the turn in progress
```
Unset is the default and means the template decides. On Qwen3.8-27B that
is `true` — every prior turn's reasoning is replayed. Full replay is
what makes prompt growth track *total* completion rather than visible
output: one measured session produced 28,672 reasoning tokens on turn 1
and carried all of them into turn 2's 31,269-token prompt, at ~22 s of
prefill on every turn thereafter.
Setting it to `false` is a legitimate experiment, not a recommendation.
Whether narrowing replay *helps* the model is unmeasured. Set it in both
`models.toml` and this file, like sampling, and the value is reported on
`/v1/models` and logged at load so a session can be attributed to the
arm it ran under.
**`reasoning_budget.answer_reserve_tokens`** (default `4096`) holds back
part of the caller's output budget for the answer:
```toml
[harness.candle.reasoning_budget]
answer_reserve_tokens = 4096
```
A reasoning model with no stopping criterion fills whatever budget it is
given. Measured on Qwen3.8-27B: given 16,384 tokens it used 16,384;
given 32,768 it used 32,768 — and both times returned no answer at all.
The reserve bounds the think block at `max_tokens - answer_reserve` so
the rest survives for the reply.
Deliberately *not* derived from reasoning effort. Effort is a prompt
instruction the model is trained to follow; making it also set a
guillotine is how a model came to be told to reason at `xhigh` and then
cut off at `medium`'s budget. Set `0` to disable, restoring unbounded
reasoning.
The trade to know: when the reserve fires, the model is interrupted
mid-deliberation and its answer phase tends to state an *intention*
rather than a conclusion. That is better than returning nothing, but it
is not free.
## The prefix cache
Snapshots of previously-served prefixes, so a follow-up turn re-uses the
KV cache instead of re-prefilling the whole conversation:
```toml
[harness.candle.prefix_cache]
enabled = true
budget_mb = 2048 # snapshot bytes per loaded model
max_entries = 8 # live snapshots per model, regardless of budget
```
**Reach for `budget_mb` when `cached_tokens` stops growing mid-session.**
That is the symptom: the prompt keeps growing, the cached prefix does
not, and every turn re-prefills the difference. Observed on a 1 GiB
default — `cached_tokens` pinned at 2,016 for ten consecutive turns
while the prompt reached 52,869, so ~50k tokens were re-prefilled every turn
at roughly 37 s each, against ~20 s of actual decode.
The number is not free, and the direction of the trade is the opposite
of what it looks like. **The snapshot budget is reserved before the KV
budget, not from what is left over.** On a 2×32 GB host serving a 27B,
measured 1:1 — the token column is the pool all concurrent requests
share, not a per-slot allowance:
| `budget_mb` | KV budget | tokens in the pool | holds a 53k snapshot |
|---|---|---|---|
| 1024 | 4713 MB | 150,800 | no |
| 2048 | 3689 MB | 118,000 | yes |
| 3072 | 2665 MB | 85,300 | yes, but a session cannot reach 53k |
So raising it too far starves the KV cache and shortens how long a
single session can run — the model can no longer reach the context whose
snapshot the larger cache was meant to hold. Size it against the deepest
session you actually serve, and check `kv_budget_mb` in the journal
after a reload.
## Checking on it
```sh
@@ -24,7 +24,7 @@ confined anywhere.
```toml
[[models]]
id = "Qwen/Qwen3.6-27B"
id = "Qwen/Qwen3.8-27B"
harness = "candle"
min_devices = 2
min_device_vram_mb = 24000
@@ -32,6 +32,43 @@ daemon now does automatically.
| `limit.input` | same | compaction trigger — opencode compacts to keep the prompt at/under this |
| `limit.output` | same | generation reserve opencode leaves below the wall |
### Tuning the derived limit
Since #67 the neuron computes `limit` itself (see below), and three
`[harness.candle.context_limit]` settings shape that computation:
```toml
[harness.candle.context_limit]
target_prefill_latency_secs = 120 # longest acceptable prefill per turn
bootstrap_prefill_tok_per_sec = 800 # cold-start estimate until measured
output_reserve_tokens = 8192 # what a request gets when it names no cap
max_output_tokens = 32768 # the largest output a request may name
```
**`target_prefill_latency_secs`** is the one that usually binds. The
throughput ceiling is `target_prefill_latency_secs × measured
prefill tok/s`, so on a host measuring ~1,400 tok/s a 120-second target
advertises about 168k — above most hard ceilings, in which case it stops
mattering. On a slower host, or one whose recent traffic has been small
prompts, it is what caps the window.
**`bootstrap_prefill_tok_per_sec`** applies until the model has measured
its own rate, and stops the advertised window collapsing on a
freshly-restarted host. Prefill samples below 512 tokens are ignored
when computing that rate: a short prompt is dominated by fixed
per-request cost and says nothing about throughput — the deploy's own
19-token smoke probe reads as ~145 tok/s on a host that measures ~1,400.
So a host that has served only small requests keeps the bootstrap
estimate rather than adopting a misleading one, and no warm-up is
needed after a restart.
**`output_reserve_tokens`** is what a request that names no cap
receives. **`max_output_tokens`** is the ceiling advertised to clients
as `max_output_tokens`, and it is the *ceiling*, not the reserve —
publishing the reserve tells every client that reads the advertisement
to cap itself there, which on a reasoning model can be less than a
single think block.
## How they must relate
For a single model on a single neuron, all of these must hold:
@@ -60,9 +97,9 @@ Notes:
lands, treat the VRAM-safe ceiling in rule 5 as a hard limit you set
`NEURON_MAX_PROMPT_TOKENS` below, not something the daemon enforces.
## VRAM cost of context (Qwen3.6-27B on beast)
## VRAM cost of context (Qwen3.8-27B on beast)
Qwen3.6-27B is a hybrid linear-attention model: of its 64 layers only
Qwen3.8-27B is a hybrid linear-attention model: of its 64 layers only
every 4th is full-attention (`full_attention_interval = 4` → **16**
full-attn layers); the rest are `linear_attention` with constant-size
recurrent state. KV cache grows **only** on the 16 full-attn layers
@@ -131,7 +168,7 @@ value wins over a hand override of the same variable. Use `local.conf`
only for genuinely host-local, transient experiments — and remember a
later deploy will re-assert `model.conf`.
**opencode** — `opencode.json`, `provider.models."Qwen/Qwen3.6-27B".limit`:
**opencode** — `opencode.json`, `provider.models."Qwen/Qwen3.8-27B".limit`:
```json
{ "context": 131072, "input": 122880, "output": 8192 }
+451 -1
View File
@@ -36,7 +36,8 @@
"globals": "^16.5.0",
"typescript": "~5.9.3",
"typescript-eslint": "^8.46.4",
"vite": "^7.2.0"
"vite": "^7.2.0",
"vitest": "^3.2.7"
}
},
"node_modules/@babel/code-frame": {
@@ -1814,6 +1815,17 @@
"@swc/counter": "^0.1.3"
}
},
"node_modules/@types/chai": {
"version": "5.2.3",
"resolved": "https://registry.npmjs.org/@types/chai/-/chai-5.2.3.tgz",
"integrity": "sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/deep-eql": "*",
"assertion-error": "^2.0.1"
}
},
"node_modules/@types/debug": {
"version": "4.1.13",
"resolved": "https://registry.npmjs.org/@types/debug/-/debug-4.1.13.tgz",
@@ -1823,6 +1835,13 @@
"@types/ms": "*"
}
},
"node_modules/@types/deep-eql": {
"version": "4.0.2",
"resolved": "https://registry.npmjs.org/@types/deep-eql/-/deep-eql-4.0.2.tgz",
"integrity": "sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==",
"dev": true,
"license": "MIT"
},
"node_modules/@types/estree": {
"version": "1.0.9",
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz",
@@ -2243,6 +2262,121 @@
"vite": "^4 || ^5 || ^6 || ^7 || ^8"
}
},
"node_modules/@vitest/expect": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-3.2.7.tgz",
"integrity": "sha512-E8eBXaKibuvH2pSZErOjdVb5vF4PbKYcrnluBTYxEk1l/VhhwZg1kZQsdtjq+CsF5CFydf2Rdkz7jDHKSisi3w==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/chai": "^5.2.2",
"@vitest/spy": "3.2.7",
"@vitest/utils": "3.2.7",
"chai": "^5.2.0",
"tinyrainbow": "^2.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/mocker": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-3.2.7.tgz",
"integrity": "sha512-Trr0hYO9CM3Wj6ksWHRhK9IZpIY6wTMO5u/MqXurMxT57sWBaOPEtP3Oq60ihZuh5JsiagKfz95OcxdEP6dBrA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/spy": "3.2.7",
"estree-walker": "^3.0.3",
"magic-string": "^0.30.17"
},
"funding": {
"url": "https://opencollective.com/vitest"
},
"peerDependencies": {
"msw": "^2.4.9",
"vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0"
},
"peerDependenciesMeta": {
"msw": {
"optional": true
},
"vite": {
"optional": true
}
}
},
"node_modules/@vitest/pretty-format": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-3.2.7.tgz",
"integrity": "sha512-KUHlwqVu0sRlhCdyPdQ/wBoTfRahjUky1MubOmYw9fWfIZy1gNoHpuaaQBPAaMaVYdQYHJLurzj8ECCj5OwTqA==",
"dev": true,
"license": "MIT",
"dependencies": {
"tinyrainbow": "^2.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/runner": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-3.2.7.tgz",
"integrity": "sha512-sB9y4ovltoQP+WaUPwmSxO9WIg9Ig694Di5PalVPsYHklAdE027mehpWF2SQSVq+k6sFgaivbTjTJwZLSHbedA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/utils": "3.2.7",
"pathe": "^2.0.3",
"strip-literal": "^3.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/snapshot": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-3.2.7.tgz",
"integrity": "sha512-7C+MwShwtBSI5Buwoyg3s/iY1eHL9PKAf+O1wVh/TdnjXUtkoL/9YQtre90i4MtNXM6edP1wJ2zOBpfCyhIS7g==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "3.2.7",
"magic-string": "^0.30.17",
"pathe": "^2.0.3"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/spy": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-3.2.7.tgz",
"integrity": "sha512-Q2eQGI6d2L/hBtZ0qNuKcAGid68XK6cv1xsoaIma6PaJhHPoqcEJhYpXZ/5myCMqkNgtP6UKuBhbc0nHKnrkuQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"tinyspy": "^4.0.3"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/utils": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-3.2.7.tgz",
"integrity": "sha512-x6BDOd7dyo3PFLY3I9/HJ25X/6OurhGXk2/B9gOZNPF7XDVjeBK4k01lQE5uvDpbuheErh91qYuE1E2OEjK3Rw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "3.2.7",
"loupe": "^3.1.4",
"tinyrainbow": "^2.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/acorn": {
"version": "8.17.0",
"resolved": "https://registry.npmjs.org/acorn/-/acorn-8.17.0.tgz",
@@ -2318,6 +2452,16 @@
"node": ">=10"
}
},
"node_modules/assertion-error": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-2.0.1.tgz",
"integrity": "sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=12"
}
},
"node_modules/bail": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/bail/-/bail-2.0.2.tgz",
@@ -2412,6 +2556,16 @@
"node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7"
}
},
"node_modules/cac": {
"version": "6.7.14",
"resolved": "https://registry.npmjs.org/cac/-/cac-6.7.14.tgz",
"integrity": "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=8"
}
},
"node_modules/callsites": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/callsites/-/callsites-3.1.0.tgz",
@@ -2453,6 +2607,23 @@
"url": "https://github.com/sponsors/wooorm"
}
},
"node_modules/chai": {
"version": "5.3.3",
"resolved": "https://registry.npmjs.org/chai/-/chai-5.3.3.tgz",
"integrity": "sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==",
"dev": true,
"license": "MIT",
"dependencies": {
"assertion-error": "^2.0.1",
"check-error": "^2.1.1",
"deep-eql": "^5.0.1",
"loupe": "^3.1.0",
"pathval": "^2.0.0"
},
"engines": {
"node": ">=18"
}
},
"node_modules/chalk": {
"version": "4.1.2",
"resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz",
@@ -2510,6 +2681,16 @@
"url": "https://github.com/sponsors/wooorm"
}
},
"node_modules/check-error": {
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/check-error/-/check-error-2.1.3.tgz",
"integrity": "sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 16"
}
},
"node_modules/classnames": {
"version": "2.5.1",
"resolved": "https://registry.npmjs.org/classnames/-/classnames-2.5.1.tgz",
@@ -2633,6 +2814,16 @@
"url": "https://github.com/sponsors/wooorm"
}
},
"node_modules/deep-eql": {
"version": "5.0.2",
"resolved": "https://registry.npmjs.org/deep-eql/-/deep-eql-5.0.2.tgz",
"integrity": "sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=6"
}
},
"node_modules/deep-is": {
"version": "0.1.4",
"resolved": "https://registry.npmjs.org/deep-is/-/deep-is-0.1.4.tgz",
@@ -2695,6 +2886,13 @@
"dev": true,
"license": "ISC"
},
"node_modules/es-module-lexer": {
"version": "1.7.0",
"resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-1.7.0.tgz",
"integrity": "sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==",
"dev": true,
"license": "MIT"
},
"node_modules/esbuild": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.27.7.tgz",
@@ -2944,6 +3142,16 @@
"url": "https://opencollective.com/unified"
}
},
"node_modules/estree-walker": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-3.0.3.tgz",
"integrity": "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/estree": "^1.0.0"
}
},
"node_modules/esutils": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/esutils/-/esutils-2.0.3.tgz",
@@ -2954,6 +3162,16 @@
"node": ">=0.10.0"
}
},
"node_modules/expect-type": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.4.0.tgz",
"integrity": "sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==",
"dev": true,
"license": "Apache-2.0",
"engines": {
"node": ">=12.0.0"
}
},
"node_modules/extend": {
"version": "3.0.2",
"resolved": "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz",
@@ -3539,6 +3757,13 @@
"loose-envify": "cli.js"
}
},
"node_modules/loupe": {
"version": "3.2.1",
"resolved": "https://registry.npmjs.org/loupe/-/loupe-3.2.1.tgz",
"integrity": "sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==",
"dev": true,
"license": "MIT"
},
"node_modules/lowlight": {
"version": "3.3.0",
"resolved": "https://registry.npmjs.org/lowlight/-/lowlight-3.3.0.tgz",
@@ -3564,6 +3789,16 @@
"yallist": "^3.0.2"
}
},
"node_modules/magic-string": {
"version": "0.30.21",
"resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz",
"integrity": "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@jridgewell/sourcemap-codec": "^1.5.5"
}
},
"node_modules/markdown-table": {
"version": "3.0.4",
"resolved": "https://registry.npmjs.org/markdown-table/-/markdown-table-3.0.4.tgz",
@@ -4591,6 +4826,23 @@
"node": ">=8"
}
},
"node_modules/pathe": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz",
"integrity": "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==",
"dev": true,
"license": "MIT"
},
"node_modules/pathval": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/pathval/-/pathval-2.0.1.tgz",
"integrity": "sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 14.16"
}
},
"node_modules/picocolors": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
@@ -5096,6 +5348,13 @@
"node": ">=8"
}
},
"node_modules/siginfo": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/siginfo/-/siginfo-2.0.0.tgz",
"integrity": "sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==",
"dev": true,
"license": "ISC"
},
"node_modules/source-map-js": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
@@ -5116,6 +5375,20 @@
"url": "https://github.com/sponsors/wooorm"
}
},
"node_modules/stackback": {
"version": "0.0.2",
"resolved": "https://registry.npmjs.org/stackback/-/stackback-0.0.2.tgz",
"integrity": "sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==",
"dev": true,
"license": "MIT"
},
"node_modules/std-env": {
"version": "3.10.0",
"resolved": "https://registry.npmjs.org/std-env/-/std-env-3.10.0.tgz",
"integrity": "sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==",
"dev": true,
"license": "MIT"
},
"node_modules/stringify-entities": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/stringify-entities/-/stringify-entities-4.0.4.tgz",
@@ -5143,6 +5416,26 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/strip-literal": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/strip-literal/-/strip-literal-3.1.0.tgz",
"integrity": "sha512-8r3mkIM/2+PpjHoOtiAW8Rg3jJLHaV7xPwG+YRGrv6FP0wwk/toTpATxWYOW0BKdWwl82VT2tFYi5DlROa0Mxg==",
"dev": true,
"license": "MIT",
"dependencies": {
"js-tokens": "^9.0.1"
},
"funding": {
"url": "https://github.com/sponsors/antfu"
}
},
"node_modules/strip-literal/node_modules/js-tokens": {
"version": "9.0.1",
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-9.0.1.tgz",
"integrity": "sha512-mxa9E9ITFOt0ban3j6L5MpjwegGz6lBQmM1IJkWeBZGcMxto50+eWdjC/52xDbS2vy0k7vIMK0Fe2wfL9OQSpQ==",
"dev": true,
"license": "MIT"
},
"node_modules/style-to-js": {
"version": "1.1.21",
"resolved": "https://registry.npmjs.org/style-to-js/-/style-to-js-1.1.21.tgz",
@@ -5174,6 +5467,20 @@
"node": ">=8"
}
},
"node_modules/tinybench": {
"version": "2.9.0",
"resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz",
"integrity": "sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==",
"dev": true,
"license": "MIT"
},
"node_modules/tinyexec": {
"version": "0.3.2",
"resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-0.3.2.tgz",
"integrity": "sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==",
"dev": true,
"license": "MIT"
},
"node_modules/tinyglobby": {
"version": "0.2.17",
"resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.17.tgz",
@@ -5191,6 +5498,36 @@
"url": "https://github.com/sponsors/SuperchupuDev"
}
},
"node_modules/tinypool": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/tinypool/-/tinypool-1.1.1.tgz",
"integrity": "sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==",
"dev": true,
"license": "MIT",
"engines": {
"node": "^18.0.0 || >=20.0.0"
}
},
"node_modules/tinyrainbow": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-2.0.0.tgz",
"integrity": "sha512-op4nsTR47R6p0vMUUoYl/a+ljLFVtlfaXkLQmqfLR1qHma1h/ysYk4hEXZ880bf2CYgTskvTa/e196Vd5dDQXw==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/tinyspy": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/tinyspy/-/tinyspy-4.0.4.tgz",
"integrity": "sha512-azl+t0z7pw/z958Gy9svOTuzqIk6xq+NSheJzn5MMWtWTFywIacg2wUlzKFGtt3cthx0r2SxMK0yzJOR0IES7Q==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/trim-lines": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/trim-lines/-/trim-lines-3.0.1.tgz",
@@ -5557,6 +5894,102 @@
}
}
},
"node_modules/vite-node": {
"version": "3.2.4",
"resolved": "https://registry.npmjs.org/vite-node/-/vite-node-3.2.4.tgz",
"integrity": "sha512-EbKSKh+bh1E1IFxeO0pg1n4dvoOTt0UDiXMd/qn++r98+jPO1xtJilvXldeuQ8giIB5IkpjCgMleHMNEsGH6pg==",
"dev": true,
"license": "MIT",
"dependencies": {
"cac": "^6.7.14",
"debug": "^4.4.1",
"es-module-lexer": "^1.7.0",
"pathe": "^2.0.3",
"vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0"
},
"bin": {
"vite-node": "vite-node.mjs"
},
"engines": {
"node": "^18.0.0 || ^20.0.0 || >=22.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/vitest": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-3.2.7.tgz",
"integrity": "sha512-KrxIJ62Fd89gfysR4WotlgZABiz2dqFPgqGzX7s+CwsqLFomRH7777ZcrOD6+WVAh7khPQP41A+BKbpcJFrdEg==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/chai": "^5.2.2",
"@vitest/expect": "3.2.7",
"@vitest/mocker": "3.2.7",
"@vitest/pretty-format": "^3.2.7",
"@vitest/runner": "3.2.7",
"@vitest/snapshot": "3.2.7",
"@vitest/spy": "3.2.7",
"@vitest/utils": "3.2.7",
"chai": "^5.2.0",
"debug": "^4.4.1",
"expect-type": "^1.2.1",
"magic-string": "^0.30.17",
"pathe": "^2.0.3",
"picomatch": "^4.0.2",
"std-env": "^3.9.0",
"tinybench": "^2.9.0",
"tinyexec": "^0.3.2",
"tinyglobby": "^0.2.14",
"tinypool": "^1.1.1",
"tinyrainbow": "^2.0.0",
"vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0",
"vite-node": "3.2.4",
"why-is-node-running": "^2.3.0"
},
"bin": {
"vitest": "vitest.mjs"
},
"engines": {
"node": "^18.0.0 || ^20.0.0 || >=22.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
},
"peerDependencies": {
"@edge-runtime/vm": "*",
"@types/debug": "^4.1.12",
"@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0",
"@vitest/browser": "3.2.7",
"@vitest/ui": "3.2.7",
"happy-dom": "*",
"jsdom": "*"
},
"peerDependenciesMeta": {
"@edge-runtime/vm": {
"optional": true
},
"@types/debug": {
"optional": true
},
"@types/node": {
"optional": true
},
"@vitest/browser": {
"optional": true
},
"@vitest/ui": {
"optional": true
},
"happy-dom": {
"optional": true
},
"jsdom": {
"optional": true
}
}
},
"node_modules/void-elements": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/void-elements/-/void-elements-3.1.0.tgz",
@@ -5591,6 +6024,23 @@
"node": ">= 8"
}
},
"node_modules/why-is-node-running": {
"version": "2.3.0",
"resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-2.3.0.tgz",
"integrity": "sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==",
"dev": true,
"license": "MIT",
"dependencies": {
"siginfo": "^2.0.0",
"stackback": "0.0.2"
},
"bin": {
"why-is-node-running": "cli.js"
},
"engines": {
"node": ">=8"
}
},
"node_modules/word-wrap": {
"version": "1.2.5",
"resolved": "https://registry.npmjs.org/word-wrap/-/word-wrap-1.2.5.tgz",
+4 -1
View File
@@ -9,6 +9,8 @@
"preview": "vite preview",
"lint": "eslint .",
"typecheck": "tsc -b",
"test": "vitest run",
"test:watch": "vitest",
"i18n:check": "node ./scripts/check-i18n-keys.mjs",
"i18n:meta": "node ./scripts/check-i18n-metadata.mjs",
"i18n:lang-labels": "node ./scripts/check-i18n-lang-labels.mjs",
@@ -43,6 +45,7 @@
"globals": "^16.5.0",
"typescript": "~5.9.3",
"typescript-eslint": "^8.46.4",
"vite": "^7.2.0"
"vite": "^7.2.0",
"vitest": "^3.2.7"
}
}
+37 -1
View File
@@ -58,7 +58,9 @@ if (!fs.existsSync(SSR_ENTRY)) {
fail(`${path.relative(ROOT, SSR_ENTRY)} is missing — run the SSR build first`);
}
const { renderDoc, slugs } = await import(url.pathToFileURL(SSR_ENTRY).href);
const { renderDoc, slugs, indexes } = await import(
url.pathToFileURL(SSR_ENTRY).href
);
const template = fs.readFileSync(path.join(DIST, "index.html"), "utf8");
// The shell must contain an empty #root for the markup to go into. If
@@ -117,6 +119,40 @@ for (const slug of slugs()) {
written.push({ slug, canonical });
}
// Index pages for every directory the loop above created. Without these
// nginx answers 403 for /docs and each section: the directory exists, so
// `try_files $uri/` matches, and with no index inside and autoindex off
// there is nothing to serve and no fallthrough to the SPA shell.
for (const idx of indexes()) {
const slugPath = idx.path ? `/docs/${idx.path}` : "/docs";
const html = withHead(
template.replace(ROOT_DIV, `<div id="root">${idx.html}</div>`),
idx.title,
undefined,
`${SITE}${slugPath}`,
);
const dir = idx.path
? path.join(DIST, "docs", ...idx.path.split("/"))
: path.join(DIST, "docs");
fs.mkdirSync(dir, { recursive: true });
fs.writeFileSync(path.join(dir, "index.html"), html);
}
// Nothing the prerenderer creates may be left without an index — that is
// the 403 above, and it is silent at build time.
const missing = [];
(function walk(dir) {
if (!fs.existsSync(dir)) return;
const entries = fs.readdirSync(dir, { withFileTypes: true });
if (!entries.some((e) => e.isFile() && e.name === "index.html")) {
missing.push(path.relative(DIST, dir));
}
for (const e of entries) if (e.isDirectory()) walk(path.join(dir, e.name));
})(path.join(DIST, "docs"));
if (missing.length) {
fail(`directories with no index.html would 403: ${missing.join(", ")}`);
}
// A sitemap so the pages are discoverable without relying on a crawler
// walking the sidebar — only once they are meant to be found.
if (INDEXABLE) {
+120
View File
@@ -2063,3 +2063,123 @@ html[dir="rtl"] .hx-landing-composer textarea {
display: none;
}
}
/* Assistant working — reasoning and tool calls (#TurnBlocks).
*
* Deliberately quieter than the answer: muted text, no bubble of its
* own, hairline separators. The whole point is that it can be present
* without competing, so it reads as an annotation on the reply rather
* than as more reply. */
.hx-blocks {
display: flex;
flex-direction: column;
gap: 0.25rem;
margin-bottom: 0.5rem;
}
.hx-block {
border-left: 2px solid var(--color-border-subtle);
padding-left: 0.5rem;
}
.hx-block-open {
border-left-color: var(--color-accent-soft);
}
.hx-block-head {
display: flex;
align-items: center;
gap: 0.35rem;
width: 100%;
padding: 0.15rem 0;
background: none;
border: 0;
color: var(--color-text-muted);
font-size: 0.78rem;
text-align: start;
cursor: pointer;
}
.hx-block-head:hover {
color: var(--color-text);
}
.hx-block-caret {
flex: none;
transition: transform 0.12s ease;
}
.hx-block-caret-open {
transform: rotate(90deg);
}
.hx-block-label {
flex: none;
font-weight: 600;
}
.hx-block-running {
flex: none;
opacity: 0.7;
font-style: italic;
}
/* The peek is the first thing to give up space — it is a hint, and a
* wrapped one would cost the header its single-line shape. */
.hx-block-peek {
flex: 1 1 auto;
min-width: 0;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
opacity: 0.6;
}
.hx-block-body {
/* Capped and scrollable: a long think block would otherwise push the
* answer off-screen, which defeats the point of showing it. */
max-height: 16rem;
overflow-y: auto;
margin: 0.15rem 0 0.35rem;
padding-inline-end: 0.4rem;
color: var(--color-text-muted);
font-size: 0.82rem;
line-height: 1.45;
white-space: pre-wrap;
overflow-wrap: anywhere;
}
/* Reasoning toggle in the composer. Sits with the send button rather
* than in a settings panel: it changes what the next message does, so it
* belongs where the message is written. */
.hx-thinking-toggle {
display: inline-flex;
align-items: center;
gap: 0.35rem;
margin: 0;
padding: 0 0.5rem;
color: var(--color-text-muted);
font-size: 0.78rem;
white-space: nowrap;
cursor: pointer;
user-select: none;
}
.hx-thinking-toggle:hover {
color: var(--color-text);
}
.hx-thinking-select {
background: transparent;
color: inherit;
border: 1px solid var(--color-border-subtle);
border-radius: 0.35rem;
padding: 0.1rem 0.25rem;
font-size: 0.78rem;
cursor: pointer;
}
.hx-thinking-select:focus-visible {
outline: 2px solid var(--color-accent);
outline-offset: 1px;
}
/* Options inherit the page background in some engines and end up
* unreadable on a dark theme; set both explicitly. */
.hx-thinking-select option {
background: var(--color-bg-elevated);
color: var(--color-text);
}
/* Narrow screens: the composer is already tight, and the word is the
* expendable half — the select still reads as a control. */
@media (max-width: 30rem) {
.hx-thinking-label {
display: none;
}
}
+129
View File
@@ -0,0 +1,129 @@
import { useState } from "react";
import { useTranslation } from "react-i18next";
import { FaChevronRight, FaBrain, FaWrench } from "react-icons/fa6";
import type { TurnBlock } from "../data/db";
import { isOpen } from "../lib/turnBlockState";
/**
* The assistant's working — reasoning and tool calls — shown above its
* answer.
*
* Two things are in tension. All of this should be *available*, because a
* model that reasons for two minutes and then answers has done work worth
* inspecting, and a tool call the user cannot see is a claim they cannot
* check. But none of it is what they asked for, so it must not bury the
* answer.
*
* The resolution is that visibility follows activity:
*
* * While a block is the live one — still streaming, no answer text yet
* — it is open. That span is otherwise dead air; a spinner would say
* "working", this says what it is working *on*.
* * The moment answer text starts, every block collapses to a one-line
* header. The thing the user asked for takes the space.
* * Completed turns load collapsed, so scrollback stays readable.
*
* A block the user opens or closes by hand stays that way. Explicit
* intent outranks the automatic rule, including for the live block —
* someone who closes a noisy reasoning stream should not have it reopen
* itself on the next token.
*/
export function TurnBlocks({
blocks,
streaming,
answerStarted,
}: {
blocks: TurnBlock[];
/** This turn is still being generated. */
streaming: boolean;
/** Answer text has begun arriving, so the working is now history. */
answerStarted: boolean;
}) {
const { t } = useTranslation();
// Only blocks the user has touched appear here; everything else follows
// the automatic rule. Keyed by index, which is stable because blocks
// are only ever appended within a turn.
const [overrides, setOverrides] = useState<Record<number, boolean>>({});
if (blocks.length === 0) return null;
return (
<div className="hx-blocks">
{blocks.map((b, i) => {
const open = isOpen(i, blocks.length, streaming, answerStarted, overrides[i]);
const body = b.kind === "reasoning" ? b.text : blockToolBody(b);
return (
<div key={i} className={`hx-block${open ? " hx-block-open" : ""}`}>
<button
type="button"
className="hx-block-head"
aria-expanded={open}
onClick={() => setOverrides((o) => ({ ...o, [i]: !open }))}
>
<FaChevronRight
className={`hx-block-caret${open ? " hx-block-caret-open" : ""}`}
size={10}
aria-hidden="true"
/>
{b.kind === "reasoning" ? (
<FaBrain size={11} aria-hidden="true" />
) : (
<FaWrench size={11} aria-hidden="true" />
)}
<span className="hx-block-label">
{b.kind === "reasoning"
? t("chat:blockReasoning")
: t("chat:blockTool", { name: b.name })}
</span>
{/* A pending tool call is the one case where the header
* alone should say something is happening: its body is
* empty until the result lands, so an open block would
* otherwise look like a stall. */}
{b.kind === "tool" && b.result === undefined && streaming && (
<span className="hx-block-running">{t("chat:blockRunning")}</span>
)}
{!open && (
<span className="hx-block-peek">{peek(body)}</span>
)}
</button>
{open && <div className="hx-block-body">{body}</div>}
</div>
);
})}
</div>
);
}
/** Arguments, then the result once it arrives. */
function blockToolBody(b: Extract<TurnBlock, { kind: "tool" }>): string {
const args = prettyArgs(b.args);
return b.result === undefined ? args : `${args}\n\n${b.result}`;
}
/**
* Tool arguments as sent, pretty-printed when they are JSON.
*
* Models emit malformed arguments often enough that this must never
* throw — a UI that blanks because the model produced bad JSON hides the
* very thing worth seeing.
*/
function prettyArgs(raw: string): string {
try {
return JSON.stringify(JSON.parse(raw), null, 2);
} catch {
return raw;
}
}
/**
* The tail of a collapsed block, so a closed row still carries a hint of
* its content.
*
* The tail rather than the head: while a block streams, the end is where
* the model currently is. A head-anchored preview freezes on the first
* few words and stops telling you anything.
*/
function peek(text: string): string {
const flat = text.replace(/\s+/g, " ").trim();
return flat.length <= 80 ? flat : `…${flat.slice(-79)}`;
}
+27
View File
@@ -39,6 +39,23 @@ export interface MessageSource {
url: string;
}
/**
* Work the assistant did on the way to an answer: a stretch of reasoning,
* or one tool call and its result.
*
* Kept as an ordered list rather than two separate fields because the
* order is the story — a turn can reason, call a tool, reason about what
* came back, and call another. Flattening that into "the reasoning" and
* "the tool calls" would lose which thought preceded which call.
*
* `text` and `result` grow while streaming, so a block is also its own
* progress indicator: a reasoning block with text and no following block
* is what the model is doing *right now*.
*/
export type TurnBlock =
| { kind: "reasoning"; text: string }
| { kind: "tool"; name: string; args: string; result?: string };
export interface Message {
id: string;
conversationId: string;
@@ -50,6 +67,16 @@ export interface Message {
promptTokens?: number;
completionTokens?: number;
sources?: MessageSource[];
/**
* Reasoning and tool calls, in the order they happened. Absent on user
* turns and on assistant turns that answered directly.
*
* Not indexed, so no schema version bump: Dexie stores the whole
* object and only the declared indexes constrain it. Messages written
* before this field existed simply have none, which renders as an
* answer with no working shown — correct, since none was captured.
*/
blocks?: TurnBlock[];
}
/**
+14 -1
View File
@@ -10,6 +10,7 @@ import {
type Message,
type MessageRole,
type Project,
type TurnBlock,
} from "./db";
function uuid(): string {
@@ -138,10 +139,22 @@ export async function setMessageContent(id: string, content: string): Promise<vo
await db.messages.update(id, { content });
}
/**
* Write the whole block list as an absolute snapshot.
*
* Same discipline as `setMessageContent`: the caller owns the array and
* writes it entire, because a read-modify-write append races itself when
* deltas arrive faster than the IndexedDB round-trip — which is how the
* content path dropped tokens before it was written this way.
*/
export async function setMessageBlocks(id: string, blocks: TurnBlock[]): Promise<void> {
await db.messages.update(id, { blocks });
}
export async function finalizeMessage(
id: string,
patch: Partial<
Pick<Message, "status" | "errorCode" | "promptTokens" | "completionTokens" | "sources">
Pick<Message, "status" | "errorCode" | "promptTokens" | "completionTokens" | "sources" | "blocks">
>,
): Promise<void> {
await db.messages.update(id, patch);
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "{{host}} እየተነበበ ነው…",
"copyCode": "ኮድ ቅዳ",
"copiedCode": "ተቀድቷል",
"anonNearLimit": "ወደ የስም-አልባ ተጠቃሚ ገደብ እየተቃረቡ ነው። ነፃ ድልድል ለማግኘት ይመዝገቡ።"
"anonNearLimit": "ወደ የስም-አልባ ተጠቃሚ ገደብ እየተቃረቡ ነው። ነፃ ድልድል ለማግኘት ይመዝገቡ።",
"blockReasoning": "በማሰብ ላይ",
"blockTool": "{{name}}",
"blockRunning": "በማስኬድ ላይ…",
"showThinking": "ማመዛዘን",
"showThinkingHint": "ሞዴሉ ከመመለሱ በፊት እንዲያስብ ጠይቅ፣ እና ማመዛዘኑን አሳይ",
"effortOff": "ጠፍቷል",
"effort_low": "ዝቅተኛ",
"effort_medium": "መካከለኛ",
"effort_xhigh": "በጣም ከፍተኛ"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "تجري قراءة {{host}}…",
"copyCode": "نسخ الكود",
"copiedCode": "تم النسخ",
"anonNearLimit": "أوشكت على بلوغ حد الاستخدام المجهول. أنشئ حسابًا للحصول على مخصّص مجاني."
"anonNearLimit": "أوشكت على بلوغ حد الاستخدام المجهول. أنشئ حسابًا للحصول على مخصّص مجاني.",
"blockReasoning": "جارٍ التفكير",
"blockTool": "{{name}}",
"blockRunning": "قيد التشغيل…",
"showThinking": "التفكير",
"showThinkingHint": "اطلب من النموذج التفكير قبل الإجابة، واعرض تفكيره",
"effortOff": "مغلق",
"effort_low": "منخفض",
"effort_medium": "متوسط",
"effort_xhigh": "مرتفع جدًا"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "четене на {{host}}…",
"copyCode": "Копиране на кода",
"copiedCode": "Копирано",
"anonNearLimit": "Наближавате анонимния лимит. Регистрирайте се за безплатна квота."
"anonNearLimit": "Наближавате анонимния лимит. Регистрирайте се за безплатна квота.",
"blockReasoning": "Разсъждаване",
"blockTool": "{{name}}",
"blockRunning": "изпълнява се…",
"showThinking": "Разсъждение",
"showThinkingHint": "Моделът да мисли преди да отговори и да показва разсъжденията си",
"effortOff": "Изкл.",
"effort_low": "Ниско",
"effort_medium": "Средно",
"effort_xhigh": "Много високо"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "s'està llegint {{host}}…",
"copyCode": "Copia el codi",
"copiedCode": "Copiat",
"anonNearLimit": "Us acosteu al límit anònim. Registreu-vos per obtenir una assignació gratuïta."
"anonNearLimit": "Us acosteu al límit anònim. Registreu-vos per obtenir una assignació gratuïta.",
"blockReasoning": "Pensant",
"blockTool": "{{name}}",
"blockRunning": "en execució…",
"showThinking": "Raonament",
"showThinkingHint": "Demana al model que pensi abans de respondre i mostra'n el raonament",
"effortOff": "Desactivat",
"effort_low": "Baix",
"effort_medium": "Mitjà",
"effort_xhigh": "Molt alt"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "čtení {{host}}…",
"copyCode": "Kopírovat kód",
"copiedCode": "Zkopírováno",
"anonNearLimit": "Blížíte se k anonymnímu limitu. Zaregistrujte se a získejte bezplatný příděl."
"anonNearLimit": "Blížíte se k anonymnímu limitu. Zaregistrujte se a získejte bezplatný příděl.",
"blockReasoning": "Přemýšlení",
"blockTool": "{{name}}",
"blockRunning": "probíhá…",
"showThinking": "Uvažování",
"showThinkingHint": "Ať model před odpovědí přemýšlí a ukáže své uvažování",
"effortOff": "Vypnuto",
"effort_low": "Nízké",
"effort_medium": "Střední",
"effort_xhigh": "Velmi vysoké"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "læser {{host}}…",
"copyCode": "Kopiér kode",
"copiedCode": "Kopieret",
"anonNearLimit": "Du nærmer dig den anonyme grænse. Opret en konto og få en gratis tildeling."
"anonNearLimit": "Du nærmer dig den anonyme grænse. Opret en konto og få en gratis tildeling.",
"blockReasoning": "Tænker",
"blockTool": "{{name}}",
"blockRunning": "kører…",
"showThinking": "Ræsonnement",
"showThinkingHint": "Bed modellen tænke før den svarer, og vis dens ræsonnement",
"effortOff": "Fra",
"effort_low": "Lav",
"effort_medium": "Mellem",
"effort_xhigh": "Meget høj"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "{{host}} wird gelesen…",
"copyCode": "Code kopieren",
"copiedCode": "Kopiert",
"anonNearLimit": "Sie nähern sich dem anonymen Limit. Registrieren Sie sich für ein kostenloses Kontingent."
"anonNearLimit": "Sie nähern sich dem anonymen Limit. Registrieren Sie sich für ein kostenloses Kontingent.",
"blockReasoning": "Denkt nach",
"blockTool": "{{name}}",
"blockRunning": "läuft…",
"showThinking": "Denkprozess",
"showThinkingHint": "Das Modell vor der Antwort nachdenken lassen und den Denkprozess anzeigen",
"effortOff": "Aus",
"effort_low": "Niedrig",
"effort_medium": "Mittel",
"effort_xhigh": "Sehr hoch"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "ανάγνωση του {{host}}…",
"copyCode": "Αντιγραφή κώδικα",
"copiedCode": "Αντιγράφηκε",
"anonNearLimit": "Πλησιάζετε το όριο ανώνυμης χρήσης. Εγγραφείτε για μια δωρεάν κατανομή."
"anonNearLimit": "Πλησιάζετε το όριο ανώνυμης χρήσης. Εγγραφείτε για μια δωρεάν κατανομή.",
"blockReasoning": "Σκέφτεται",
"blockTool": "{{name}}",
"blockRunning": "εκτελείται…",
"showThinking": "Συλλογισμός",
"showThinkingHint": "Ζήτα από το μοντέλο να σκεφτεί πριν απαντήσει και δείξε τον συλλογισμό του",
"effortOff": "Ανενεργό",
"effort_low": "Χαμηλό",
"effort_medium": "Μέτριο",
"effort_xhigh": "Πολύ υψηλό"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "reading {{host}}…",
"copyCode": "Copy code",
"copiedCode": "Copied",
"anonNearLimit": "You're close to the anonymous limit. Sign up for a free allocation."
"anonNearLimit": "You're close to the anonymous limit. Sign up for a free allocation.",
"blockReasoning": "Thinking",
"blockTool": "{{name}}",
"blockRunning": "running…",
"showThinking": "Reasoning",
"showThinkingHint": "Ask the model to think before answering, and show its reasoning",
"effortOff": "Off",
"effort_low": "Low",
"effort_medium": "Medium",
"effort_xhigh": "Extra high"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "leyendo {{host}}…",
"copyCode": "Copiar código",
"copiedCode": "Copiado",
"anonNearLimit": "Estás cerca del límite anónimo. Regístrate para obtener una asignación gratuita."
"anonNearLimit": "Estás cerca del límite anónimo. Regístrate para obtener una asignación gratuita.",
"blockReasoning": "Pensando",
"blockTool": "{{name}}",
"blockRunning": "ejecutando…",
"showThinking": "Razonamiento",
"showThinkingHint": "Pide al modelo que piense antes de responder y muestra su razonamiento",
"effortOff": "Desactivado",
"effort_low": "Bajo",
"effort_medium": "Medio",
"effort_xhigh": "Muy alto"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "loetakse {{host}}…",
"copyCode": "Kopeeri kood",
"copiedCode": "Kopeeritud",
"anonNearLimit": "Oled lähenemas anonüümse kasutuse piirile. Registreeru tasuta eraldise saamiseks."
"anonNearLimit": "Oled lähenemas anonüümse kasutuse piirile. Registreeru tasuta eraldise saamiseks.",
"blockReasoning": "Mõtleb",
"blockTool": "{{name}}",
"blockRunning": "töötab…",
"showThinking": "Arutlus",
"showThinkingHint": "Palu mudelil enne vastamist mõelda ja näita tema arutlust",
"effortOff": "Väljas",
"effort_low": "Madal",
"effort_medium": "Keskmine",
"effort_xhigh": "Väga kõrge"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "در حال خواندن {{host}}…",
"copyCode": "کپی کد",
"copiedCode": "کپی شد",
"anonNearLimit": "به محدودیت کاربران ناشناس نزدیک می‌شوید. برای دریافت سهمیه رایگان ثبت‌نام کنید."
"anonNearLimit": "به محدودیت کاربران ناشناس نزدیک می‌شوید. برای دریافت سهمیه رایگان ثبت‌نام کنید.",
"blockReasoning": "در حال تفکر",
"blockTool": "{{name}}",
"blockRunning": "در حال اجرا…",
"showThinking": "استدلال",
"showThinkingHint": "از مدل بخواهید پیش از پاسخ فکر کند و استدلالش را نشان دهد",
"effortOff": "خاموش",
"effort_low": "کم",
"effort_medium": "متوسط",
"effort_xhigh": "خیلی زیاد"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "luetaan {{host}}…",
"copyCode": "Kopioi koodi",
"copiedCode": "Kopioitu",
"anonNearLimit": "Lähestyt anonyymin käytön rajaa. Rekisteröidy saadaksesi ilmaisen kiintiön."
"anonNearLimit": "Lähestyt anonyymin käytön rajaa. Rekisteröidy saadaksesi ilmaisen kiintiön.",
"blockReasoning": "Ajattelee",
"blockTool": "{{name}}",
"blockRunning": "suoritetaan…",
"showThinking": "Päättely",
"showThinkingHint": "Pyydä mallia ajattelemaan ennen vastaamista ja näytä sen päättely",
"effortOff": "Pois",
"effort_low": "Matala",
"effort_medium": "Keskitaso",
"effort_xhigh": "Erittäin korkea"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "lecture de {{host}}…",
"copyCode": "Copier le code",
"copiedCode": "Copié",
"anonNearLimit": "Vous approchez de la limite anonyme. Inscrivez-vous pour obtenir une allocation gratuite."
"anonNearLimit": "Vous approchez de la limite anonyme. Inscrivez-vous pour obtenir une allocation gratuite.",
"blockReasoning": "Réflexion",
"blockTool": "{{name}}",
"blockRunning": "en cours…",
"showThinking": "Raisonnement",
"showThinkingHint": "Demander au modèle de réfléchir avant de répondre et afficher son raisonnement",
"effortOff": "Désactivé",
"effort_low": "Faible",
"effort_medium": "Moyen",
"effort_xhigh": "Très élevé"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "ag léamh {{host}}…",
"copyCode": "Cóipeáil an cód",
"copiedCode": "Cóipeáilte",
"anonNearLimit": "Tá tú ag druidim leis an teorainn gan ainm. Cláraigh le haghaidh leithdháileadh saor in aisce."
"anonNearLimit": "Tá tú ag druidim leis an teorainn gan ainm. Cláraigh le haghaidh leithdháileadh saor in aisce.",
"blockReasoning": "Ag machnamh",
"blockTool": "{{name}}",
"blockRunning": "ar siúl…",
"showThinking": "Réasúnaíocht",
"showThinkingHint": "Iarr ar an tsamhail machnamh a dhéanamh sula bhfreagraíonn sí, agus taispeáin é",
"effortOff": "As",
"effort_low": "Íseal",
"effort_medium": "Meánach",
"effort_xhigh": "An-ard"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "ana karanta {{host}}…",
"copyCode": "Kwafi lambar",
"copiedCode": "An kwafa",
"anonNearLimit": "Ka kusa kai iyakar amfani ba tare da shiga ba. Yi rajista don samun rabo kyauta."
"anonNearLimit": "Ka kusa kai iyakar amfani ba tare da shiga ba. Yi rajista don samun rabo kyauta.",
"blockReasoning": "Yana tunani",
"blockTool": "{{name}}",
"blockRunning": "yana gudana…",
"showThinking": "Tunani",
"showThinkingHint": "Ka nemi samfurin ya yi tunani kafin amsa, kuma a nuna tunaninsa",
"effortOff": "A kashe",
"effort_low": "Ƙasa",
"effort_medium": "Matsakaici",
"effort_xhigh": "Mai tsayi sosai"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "מתבצעת קריאה של {{host}}…",
"copyCode": "העתקת הקוד",
"copiedCode": "הועתק",
"anonNearLimit": "אתם מתקרבים למגבלת השימוש האנונימי. הירשמו לקבלת הקצאה חינם."
"anonNearLimit": "אתם מתקרבים למגבלת השימוש האנונימי. הירשמו לקבלת הקצאה חינם.",
"blockReasoning": "חושב",
"blockTool": "{{name}}",
"blockRunning": "פועל…",
"showThinking": "חשיבה",
"showThinkingHint": "בקש מהמודל לחשוב לפני שהוא עונה, והצג את החשיבה שלו",
"effortOff": "כבוי",
"effort_low": "נמוך",
"effort_medium": "בינוני",
"effort_xhigh": "גבוה מאוד"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "čitanje {{host}}…",
"copyCode": "Kopiraj kôd",
"copiedCode": "Kopirano",
"anonNearLimit": "Približavate se anonimnom limitu. Registrirajte se za besplatnu kvotu."
"anonNearLimit": "Približavate se anonimnom limitu. Registrirajte se za besplatnu kvotu.",
"blockReasoning": "Razmišljanje",
"blockTool": "{{name}}",
"blockRunning": "izvodi se…",
"showThinking": "Zaključivanje",
"showThinkingHint": "Zatraži da model razmisli prije odgovora i prikaži njegovo zaključivanje",
"effortOff": "Isključeno",
"effort_low": "Nisko",
"effort_medium": "Srednje",
"effort_xhigh": "Vrlo visoko"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "{{host}} olvasása…",
"copyCode": "Kód másolása",
"copiedCode": "Másolva",
"anonNearLimit": "Közeledsz az anonim limithez. Regisztrálj ingyenes keretért."
"anonNearLimit": "Közeledsz az anonim limithez. Regisztrálj ingyenes keretért.",
"blockReasoning": "Gondolkodik",
"blockTool": "{{name}}",
"blockRunning": "fut…",
"showThinking": "Gondolatmenet",
"showThinkingHint": "Kérd meg a modellt, hogy gondolkodjon válasz előtt, és mutasd a gondolatmenetét",
"effortOff": "Ki",
"effort_low": "Alacsony",
"effort_medium": "Közepes",
"effort_xhigh": "Nagyon magas"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "lettura di {{host}} in corso…",
"copyCode": "Copia il codice",
"copiedCode": "Copiato",
"anonNearLimit": "Ti stai avvicinando al limite anonimo. Registrati per un’allocazione gratuita."
"anonNearLimit": "Ti stai avvicinando al limite anonimo. Registrati per un’allocazione gratuita.",
"blockReasoning": "Sto pensando",
"blockTool": "{{name}}",
"blockRunning": "in esecuzione…",
"showThinking": "Ragionamento",
"showThinkingHint": "Chiedi al modello di ragionare prima di rispondere e mostra il ragionamento",
"effortOff": "Disattivato",
"effort_low": "Basso",
"effort_medium": "Medio",
"effort_xhigh": "Molto alto"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "მიმდინარეობს {{host}}-ის კითხვა…",
"copyCode": "კოდის კოპირება",
"copiedCode": "დაკოპირდა",
"anonNearLimit": "უახლოვდები ანონიმურ ლიმიტს. დარეგისტრირდი უფასო კვოტისთვის."
"anonNearLimit": "უახლოვდები ანონიმურ ლიმიტს. დარეგისტრირდი უფასო კვოტისთვის.",
"blockReasoning": "ფიქრობს",
"blockTool": "{{name}}",
"blockRunning": "მიმდინარეობს…",
"showThinking": "მსჯელობა",
"showThinkingHint": "სთხოვე მოდელს იფიქროს პასუხამდე და აჩვენე მისი მსჯელობა",
"effortOff": "გამორთული",
"effort_low": "დაბალი",
"effort_medium": "საშუალო",
"effort_xhigh": "ძალიან მაღალი"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "{{host}} оқылуда…",
"copyCode": "Кодты көшіру",
"copiedCode": "Көшірілді",
"anonNearLimit": "Анонимді шекке жақындап қалдыңыз. Тегін квота алу үшін тіркеліңіз."
"anonNearLimit": "Анонимді шекке жақындап қалдыңыз. Тегін квота алу үшін тіркеліңіз.",
"blockReasoning": "Ойлануда",
"blockTool": "{{name}}",
"blockRunning": "орындалуда…",
"showThinking": "Пайымдау",
"showThinkingHint": "Жауап бермес бұрын модель ойлансын және пайымдауын көрсетсін",
"effortOff": "Өшірулі",
"effort_low": "Төмен",
"effort_medium": "Орташа",
"effort_xhigh": "Өте жоғары"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "skaitoma {{host}}…",
"copyCode": "Kopijuoti kodą",
"copiedCode": "Nukopijuota",
"anonNearLimit": "Artėjate prie anoniminio naudojimo ribos. Užsiregistruokite ir gaukite nemokamą kvotą."
"anonNearLimit": "Artėjate prie anoniminio naudojimo ribos. Užsiregistruokite ir gaukite nemokamą kvotą.",
"blockReasoning": "Mąsto",
"blockTool": "{{name}}",
"blockRunning": "vykdoma…",
"showThinking": "Samprotavimas",
"showThinkingHint": "Paprašyk modelio pamąstyti prieš atsakant ir rodyk jo samprotavimą",
"effortOff": "Išjungta",
"effort_low": "Žemas",
"effort_medium": "Vidutinis",
"effort_xhigh": "Labai aukštas"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "lasa {{host}}…",
"copyCode": "Kopēt kodu",
"copiedCode": "Nokopēts",
"anonNearLimit": "Tuvojaties anonīmās lietošanas limitam. Reģistrējieties bezmaksas piešķīrumam."
"anonNearLimit": "Tuvojaties anonīmās lietošanas limitam. Reģistrējieties bezmaksas piešķīrumam.",
"blockReasoning": "Domā",
"blockTool": "{{name}}",
"blockRunning": "notiek…",
"showThinking": "Spriešana",
"showThinkingHint": "Lūdz modelim padomāt pirms atbildes un rādi tā spriešanu",
"effortOff": "Izslēgts",
"effort_low": "Zems",
"effort_medium": "Vidējs",
"effort_xhigh": "Ļoti augsts"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "كنقراو {{host}}…",
"copyCode": "نسخ الكود",
"copiedCode": "تنسخ",
"anonNearLimit": "قربتي توصل للحد ديال الاستعمال بلا حساب. سجّل باش تاخد حصة بلاش."
"anonNearLimit": "قربتي توصل للحد ديال الاستعمال بلا حساب. سجّل باش تاخد حصة بلاش.",
"blockReasoning": "كيفكر",
"blockTool": "{{name}}",
"blockRunning": "كيخدم…",
"showThinking": "التفكير",
"showThinkingHint": "طلب من الموديل يفكر قبل ما يجاوب، وبيّن التفكير ديالو",
"effortOff": "مطفي",
"effort_low": "ضعيف",
"effort_medium": "متوسط",
"effort_xhigh": "عالي بزاف"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "qed naqra {{host}}…",
"copyCode": "Ikkopja l-kodiċi",
"copiedCode": "Ikkupjat",
"anonNearLimit": "Qed toqrob lejn il-limitu anonimu. Irreġistra għal allokazzjoni b'xejn."
"anonNearLimit": "Qed toqrob lejn il-limitu anonimu. Irreġistra għal allokazzjoni b'xejn.",
"blockReasoning": "Qed jaħseb",
"blockTool": "{{name}}",
"blockRunning": "għaddej…",
"showThinking": "Raġunament",
"showThinkingHint": "Itlob lill-mudell jaħseb qabel iwieġeb, u uri r-raġunament tiegħu",
"effortOff": "Mitfi",
"effort_low": "Baxx",
"effort_medium": "Medju",
"effort_xhigh": "Għoli ħafna"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "lezen van {{host}}…",
"copyCode": "Code kopiëren",
"copiedCode": "Gekopieerd",
"anonNearLimit": "Je nadert de anonieme limiet. Registreer je voor een gratis tegoed."
"anonNearLimit": "Je nadert de anonieme limiet. Registreer je voor een gratis tegoed.",
"blockReasoning": "Denkt na",
"blockTool": "{{name}}",
"blockRunning": "bezig…",
"showThinking": "Redenering",
"showThinkingHint": "Laat het model nadenken vóór het antwoordt en toon de redenering",
"effortOff": "Uit",
"effort_low": "Laag",
"effort_medium": "Gemiddeld",
"effort_xhigh": "Zeer hoog"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "leser {{host}}…",
"copyCode": "Kopier kode",
"copiedCode": "Kopiert",
"anonNearLimit": "Du nærmer deg den anonyme grensen. Registrer deg og få en gratis tildeling."
"anonNearLimit": "Du nærmer deg den anonyme grensen. Registrer deg og få en gratis tildeling.",
"blockReasoning": "Tenker",
"blockTool": "{{name}}",
"blockRunning": "kjører…",
"showThinking": "Resonnement",
"showThinkingHint": "Be modellen tenke før den svarer, og vis resonnementet",
"effortOff": "Av",
"effort_low": "Lav",
"effort_medium": "Middels",
"effort_xhigh": "Svært høy"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "czytanie {{host}}…",
"copyCode": "Kopiuj kod",
"copiedCode": "Skopiowano",
"anonNearLimit": "Zbliżasz się do limitu anonimowego użycia. Zarejestruj się, aby otrzymać bezpłatny przydział."
"anonNearLimit": "Zbliżasz się do limitu anonimowego użycia. Zarejestruj się, aby otrzymać bezpłatny przydział.",
"blockReasoning": "Myślenie",
"blockTool": "{{name}}",
"blockRunning": "w toku…",
"showThinking": "Rozumowanie",
"showThinkingHint": "Poproś model, by pomyślał przed odpowiedzią, i pokaż jego rozumowanie",
"effortOff": "Wyłączone",
"effort_low": "Niskie",
"effort_medium": "Średnie",
"effort_xhigh": "Bardzo wysokie"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "a ler {{host}}…",
"copyCode": "Copiar código",
"copiedCode": "Copiado",
"anonNearLimit": "Está a aproximar-se do limite anónimo. Registe-se para obter uma alocação gratuita."
"anonNearLimit": "Está a aproximar-se do limite anónimo. Registe-se para obter uma alocação gratuita.",
"blockReasoning": "A pensar",
"blockTool": "{{name}}",
"blockRunning": "em execução…",
"showThinking": "Raciocínio",
"showThinkingHint": "Pedir ao modelo que pense antes de responder e mostrar o raciocínio",
"effortOff": "Desativado",
"effort_low": "Baixo",
"effort_medium": "Médio",
"effort_xhigh": "Muito alto"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "se citește {{host}}…",
"copyCode": "Copiază codul",
"copiedCode": "Copiat",
"anonNearLimit": "Te apropii de limita pentru utilizatori anonimi. Creează un cont pentru o alocare gratuită."
"anonNearLimit": "Te apropii de limita pentru utilizatori anonimi. Creează un cont pentru o alocare gratuită.",
"blockReasoning": "Gândește",
"blockTool": "{{name}}",
"blockRunning": "în curs…",
"showThinking": "Raționament",
"showThinkingHint": "Cere modelului să gândească înainte de a răspunde și arată raționamentul",
"effortOff": "Oprit",
"effort_low": "Scăzut",
"effort_medium": "Mediu",
"effort_xhigh": "Foarte ridicat"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "читаем {{host}}…",
"copyCode": "Копировать код",
"copiedCode": "Скопировано",
"anonNearLimit": "Вы приближаетесь к лимиту анонимного доступа. Зарегистрируйтесь и получите бесплатную квоту."
"anonNearLimit": "Вы приближаетесь к лимиту анонимного доступа. Зарегистрируйтесь и получите бесплатную квоту.",
"blockReasoning": "Размышление",
"blockTool": "{{name}}",
"blockRunning": "выполняется…",
"showThinking": "Рассуждение",
"showThinkingHint": "Просить модель подумать перед ответом и показывать её рассуждение",
"effortOff": "Выкл.",
"effort_low": "Низкое",
"effort_medium": "Среднее",
"effort_xhigh": "Очень высокое"
}
+10 -1
View File
@@ -29,5 +29,14 @@
"reading": "čítam {{host}}…",
"copyCode": "Kopírovať kód",
"copiedCode": "Skopírované",
"anonNearLimit": "Blížite sa k anonymnému limitu. Zaregistrujte sa a získajte bezplatný prídel."
"anonNearLimit": "Blížite sa k anonymnému limitu. Zaregistrujte sa a získajte bezplatný prídel.",
"blockReasoning": "Premýšľa",
"blockTool": "{{name}}",
"blockRunning": "prebieha…",
"showThinking": "Uvažovanie",
"showThinkingHint": "Nech model pred odpoveďou premýšľa a ukáže svoje uvažovanie",
"effortOff": "Vypnuté",
"effort_low": "Nízke",
"effort_medium": "Stredné",
"effort_xhigh": "Veľmi vysoké"
}

Some files were not shown because too many files have changed in this diff Show More