Compare commits

...
Author SHA1 Message Date
grenade 37afba40ab fix(toolcall): recover a call that tagged its function as a parameter
CI / Config consistency (push) Successful in 9s
CI / Classify changes (push) Successful in 21s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m10s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m29s
CI / Test (push) Successful in 9m32s
CI / CUDA type-check (push) Successful in 18m51s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Reported from helexa.ai on 2026-08-29. Asked "what's the best hardware to
run qwen 3.8 flash next on?", the user's answer was a screenful of raw
markup:

    <tool_call> <parameter=web_search> <parameter=query>
    "Qwen3.8 Flash Next" hardware GPU </parameter> </function> </tool_call>

The model tagged the function itself as a parameter — there is no
`<function=` anywhere in that block. `parse_qwen_xml_tool_call` opens
with `body.split("<function=").nth(1)?`, so it returned None, and the
caller did what it does with an unparseable block: re-emitted it as
text. That text was the whole reply.

The journal shows the rest of the shape. Rounds one to four each finished
`ToolCalls` and ran real searches — 17 sources were collected — and the
fifth, tool-less round produced 35 tokens which were this malformed call.
So the turn spent five model passes and never produced a sentence.

Now a leading `<parameter=X>` is promoted to the function name when X is
a tool the request actually offered, and that pseudo-parameter is
excluded from the arguments.

The known-tool guard is the whole safety of it: without it any
`<parameter=x>` mints a function call, and a model rambling in angle
brackets starts invoking tools. `does_not_invent_a_function_from_an_unknown_parameter`
pins it, and was checked by deleting the guard — it fails, as it should.

Tested against the verbatim string from the report, plus a well-formed
call to prove the recovery does not disturb the normal path.

This makes the call parse, which is what lets the UI render it as a
search block rather than markup. It does not fix the deeper problem that
an unparseable block still reaches the user as prose — filed separately.
2026-08-29 13:47:42 +03:00
grenade 8d32f4e9aa feat(chat): a reasoning toggle, because the server only thinks when asked
The reasoning UI in 940124b0 would have displayed nothing. neuron
couples reasoning *generation* to reasoning *surfacing* (api.rs:359): a
caller that does not send `x-include-thinking: true` gets
`enable_thinking: false`, and the model produces no reasoning at all.
The web chat sent no such header, which is why every request in the
reported session logged `reasoning_tokens=0` — including an earlier
1,866-token answer.

That coupling is right and stays: an undisplayed think block still eats
the output budget, which is the failure it was added to prevent. The
client just has to opt in.

A checkbox rather than always-on, per the operator: the chat interface is
a showcase for integrators, and both paths are worth demonstrating — the
reasoning path, and the clean-output path every vanilla OpenAI client
gets. It switches real server behaviour, not a rendering preference.

Verified through the live public router before building anything on it:
the same request with the header returns `reasoning_tokens=30` where it
returned 0 without. `x-include-thinking` survives helexa-stream's
forwarder (only `host` and `content-length` are dropped) and the router's
`CorsLayer::permissive()` admits it through preflight — a custom header
from a browser needs that, and helexa-acp never exercised it.

Preference persists in localStorage, wrapped in try/catch on both sides:
private windows and blocked site data throw on access, and losing a
convenience must not take the page down.

Translated into all 42 locales.

Refs #304.
2026-08-29 13:45:23 +03:00
grenade 22ce46cb47 test(web): add vitest and gate on it, and fix the bug that found
CI / CUDA type-check (push) Blocked by required conditions
CI / Config consistency (push) Successful in 10s
CI / Classify changes (push) Successful in 21s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m22s
CI / Format (push) Successful in 15s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 10m46s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
helexa.ai had no test runner: lint, typecheck, i18n parity and build all
gated, but nothing executed a line of application logic. Closes the first
half of #304.

Gating from the first commit was the operator's call, and the reasoning
is better than warn-first: with no existing suite there is nothing to
stabilise, so a warning period only creates a window in which a vacuous
test lands and gets normalised. A suite nobody trusts is worse than none,
because it gets cited.

**It immediately earned its keep.** Extracting `useChat`'s tool-result
matching into a testable function exposed that it was backwards. Results
arrive in call order, and the inline version scanned from the newest end
— so with two `web_search` calls in one turn, the first query's results
were filed under the second query's heading. Nothing would have reported
it: no error, no empty block, just a confident answer under the wrong
question. Now FIFO, with `pairs repeated calls to one tool in order`
pinning it.

That is the argument for this issue in one defect. The bug was written,
reviewed and shipped by me hours earlier; it became visible only when the
rule had to be stated somewhere it could be run.

25 tests across three modules:

- `turnBlockState` — the eight visibility behaviours, ported from the
  throwaway harness and now importing the real module rather than
  re-declaring the rule, which is what the harness's first version did
  and which would pass against any implementation.
- `turnBlocks` — reasoning coalescing, think → act → think ordering, and
  the result-matching cases above.
- `chatClient` — SSE framing against a real `ReadableStream`: a frame cut
  mid-JSON across two reads, several frames in one read, `[DONE]`
  terminating, a malformed frame not killing the stream, comments and
  `event:` lines ignored, and reasoning never leaking into content.

Each of the three core behaviours was mutation-checked — the buggy
newest-first scan, blocks staying open past the first answer token, and
reasoning routed to `onDelta` — and every mutation is caught. A test that
cannot fail is not a test.

Also gated in deploy.yml's website build, alongside i18n:check and for
the same reason: pushes to main skip ci.yml, so that is the only check a
merge itself passes through.

No jsdom yet and no coverage threshold. The pure modules are most of the
value; a threshold picked against an empty codebase is arbitrary and
rewards tests written to satisfy it.

Refs #304.
2026-08-29 13:27:38 +03:00
grenade 940124b06f feat(chat): show the assistant's working, collapsed until it matters
CI / Config consistency (push) Successful in 9s
CI / Classify changes (push) Successful in 19s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m6s
CI / CUDA type-check (push) Has been skipped
CI / Format (push) Has been skipped
CI / Clippy (push) Has been skipped
CI / Test (push) Has been skipped
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The web chat threw reasoning away. neuron emits it as
`choice.delta.reasoning_content` — deliberately not as `content`, so
naïve clients still render clean output — and this client had no handler,
so on a hard prompt the UI sat silent for however long the model thought.
Tool calls were equally invisible: the status line said "searching…" and
nothing recorded what was searched or what came back.

Both are now captured as an ordered `blocks` list on the message, and
rendered above the answer.

Ordered rather than "the reasoning" plus "the tool calls" because the
order is the story: a turn can reason, call a tool, reason about what
came back, and call another. Two flat fields lose which thought preceded
which call.

Visibility follows activity:

  * the live block — still streaming, no answer text yet — is open,
    because that span is otherwise dead air, and this says what the model
    is working on rather than merely that it is working;
  * the moment answer text starts, everything collapses to one-line
    headers, so the thing that was asked for takes the space;
  * completed turns load collapsed;
  * an explicit click outranks all of it, in both directions. Closing a
    noisy reasoning stream must not spring back open on the next token.

That rule lives in `lib/turnBlockState.ts`, free of React imports, so it
can be read and exercised on its own — as an inline expression the only
way to check it was to click around. Verified against the built module
across all eight behaviours above, including the two override cases.

Collapsed rows carry a tail-anchored peek rather than a head-anchored
one: while a block streams, the end is where the model currently is, and
a head preview freezes on the first few words. Bodies are capped and
scrollable, since a long think block would otherwise push the answer
off-screen and defeat the point.

Storage needed no Dexie version bump — `blocks` is not indexed, and Dexie
stores whole objects. Older messages simply have none, which renders as
an answer with no working shown: correct, since none was captured.

All three strings are translated into all 42 locales; `i18n:check`
enforces parity and would have failed on English-only, which I had
assumed `fallbackLng` covered. `blockTool` is pure interpolation and is
identical everywhere.
2026-08-29 13:14:59 +03:00
grenade 5208b18784 Merge branch 'fix/router-sync-restarts-only-on-settings-change'
build-prerelease / Resolve version stamps + change detection (push) Successful in 19s
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-08-29 10:34:48 +03:00
grenade 4b3bf433bb fix(deploy): restart the router for a settings change, not a comment
CI / Classify changes (push) Successful in 21s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Config consistency (push) Successful in 9s
CI / Format (push) Successful in 14s
CI / Clippy (push) Successful in 2m30s
CI / Test (push) Successful in 11m16s
CI / CUDA type-check (push) Successful in 27m27s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The sync step's first real run restarted helexa-router, and the only
difference was the MANAGED BY CI header I had just added to the tracked
copy. Parsed settings were identical. So as written, editing a comment
drops whatever federation requests are in flight.

Byte equality and settings equality answer different questions. The bytes
decide whether to SYNC; the parsed settings decide whether to RESTART.
The router reads its aliases at startup, so a comment is not a reason to
interrupt anyone.

`script/toml-settings-equal.py` makes the second comparison, as a script
rather than an inline heredoc — a heredoc inside a YAML block scalar
needs its terminator at column 0, which breaks out of the block, and the
first attempt at this did exactly that. Verified on all three cases: the
comment-only diff that caused the restart returns "settings identical";
Wednesday's alias diff returns "settings differ in: aliases"; a missing
or unparseable file returns changed, so being wrong costs one restart
rather than leaving a service on settings nobody deployed.

Needs a scoped `cat` grant to read the deployed copy back. It is
deliberately excluded from the step's fail-hard preflight: without it the
read yields nothing, the comparison says "changed", and the step restarts
on any byte difference — the previous behaviour. A host part-way through
a sudoers rollout degrades rather than fails.
2026-08-29 10:05:28 +03:00
grenade 98df7de7e5 Merge branch 'feat/ci-owns-the-router-config'
build-prerelease / Resolve version stamps + change detection (push) Successful in 22s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m45s
build-prerelease / Build neuron-blackwell (push) Successful in 17m35s
build-prerelease / Build helexa-bench binary (push) Successful in 1m53s
build-prerelease / Build cortex binary (push) Successful in 2m23s
build-prerelease / Build neuron-ampere (push) Successful in 2m41s
build-prerelease / Build helexa-angels binary (push) Successful in 2m48s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m51s
build-prerelease / Test (push) Successful in 9m31s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package cortex RPM (push) Successful in 26s
build-prerelease / Package helexa-upstream RPM (push) Successful in 27s
build-prerelease / Package helexa-bench RPM (push) Successful in 31s
build-prerelease / Package helexa-angels RPM (push) Successful in 28s
build-prerelease / Build helexa-router binary (push) Successful in 1m34s
build-prerelease / Package helexa-router RPM (push) Successful in 20s
build-prerelease / Build neuron-ada (push) Successful in 2m34s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m10s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m10s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 3m56s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 29s
2026-08-29 09:20:31 +03:00
grenade b89ab1fafb feat(config): an operator-owned secrets layer, so CI can own the rest
CI / Classify changes (push) Successful in 18s
CI / Config consistency (push) Successful in 8s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m27s
CI / Format (push) Successful in 12s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 15m51s
CI / CUDA type-check (push) Successful in 21m37s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The remaining #287 Gap 1 files could not be version-controlled because
they mix structure with credentials, and the proposed fix — render them
in CI from Gitea secrets — makes the pipeline a credential store. It also
is not needed: the secrets already exist on the hosts and work. What CI
needs is to own the structure without touching them.

cortex, helexa-upstream and helexa-angels now merge a `secrets.toml`
sibling after the main config and before env overrides. The main file
holds structure and can be deployed by CI; `secrets.toml` holds
credentials, is written only by the operator, and is never read, written
or diffed by the pipeline.

Merging last is the safety property, not an implementation detail: the
operator's value wins, so a deploy can never replace a live credential
with a placeholder. Pinned by `the_operators_value_wins_over_the_deployed_one`.

An absent secrets.toml is not an error — figment treats a missing
`Toml::file` as an empty source — so every existing single-file
deployment keeps working untouched and adoption is per host, per service.
That is the property most worth having: if it regressed, the change would
break the fleet on rollout rather than on adoption, so
`an_absent_secrets_file_is_not_an_error` asserts it directly.

A second TOML layer rather than env vars because
`[[entitlements.keys]]` is an array of tables, which `CORTEX_*`
`__`-split env overrides cannot express. The layer then carries every
secret uniformly instead of splitting the mechanism by field shape.

Docs cover the migration as four steps that are safe to stop between,
since a value present in both files resolves to the operator's.

This is the general form of the lesson from 2026-08-27: config CI cannot
write is config no check can defend. `helexa-router.toml` was excluded
from git for sitting beside secret-bearing files, and its stale alias
took beast down. The fix is to stop mixing the two, not to hand the
pipeline the credentials.

Refs #287.
2026-08-29 08:55:06 +03:00
grenade 2956e3843c feat(deploy): CI owns the router config, and aliases must name real models
CI / Test (push) Blocked by required conditions
CI / CUDA type-check (push) Blocked by required conditions
CI / Classify changes (push) Successful in 17s
CI / Config consistency (push) Successful in 9s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m31s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
`helexa-router.toml` was gitignored beside the secret-bearing service
configs, and it holds no secrets: a listen address, poll interval, region,
the product-tier aliases, and a mesh endpoint. It was excluded by
association.

That exclusion cost an outage. On 2026-08-27 the router still aliased
`helexa/balanced` to Qwen/Qwen3.6-27B after that model had been retired
from the catalogue. Both 27Bs sat at equal `residency_priority` — the
right design, so an A/B could displace either way — so every
authenticated web-chat turn evicted beast's resident flagship and
cold-loaded the retired one, ~70 s of TP layer loading each way. Four
overlapped and left the node holding 32 GB on one card with no model
registered and nothing serving, while `activation.state` said "ready".

No bad code was involved. Two files described one routing decision and
nothing could diff them, because one of them was not in git.

So:

- `helexa-router.toml` is tracked and carries the MANAGED BY CI banner.
- `deploy-gallumbits` syncs it, digest-gated, restarting the router only
  on change (it reads aliases at startup; a needless restart drops
  in-flight federation requests). The step sits after the package work
  and preflights its own sudoers grants, per the two ways that went
  wrong this week.
- `script/infra-setup.sh` no longer writes it. Two writers for one file
  is how a fleet ends up running something no commit describes.
- `asset/sudoers.d/gallumbits-host.conf` gains three scoped grants. That
  host had no rsync grant at all, which is why #287 lists this as
  blocked; the grants must be installed by a hand-run infra-setup before
  the step can pass, and it says so when they are not.
- `script/check-alias-targets.py` fails CI when any alias names a model
  the catalogue does not define, in both ci.yml and deploy.yml. Verified
  against Wednesday's exact config: it fails, and names Qwen/Qwen3.8-27B
  as the intended target.

The check is deliberately narrow — it asks whether a target exists, never
which model a tier ought to point at. That is a product decision, and a
check that guessed would be wrong more often than the operator.

The genuinely secret-bearing configs (cortex.toml, helexa-upstream.toml,
helexa-angels.toml) are untouched and still hand-synced. They need
rendering from Gitea secrets, which is the rest of #287 Gap 1.

Refs #287.
2026-08-29 08:47:26 +03:00
grenade c43f4bc74f Merge branch 'fix/infra-setup-ships-ratelimit-to-gateway'
build-prerelease / Resolve version stamps + change detection (push) Successful in 19s
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-08-29 08:42:03 +03:00
grenade cd0211a864 Merge branch 'fix/nginx-preflight-checks-every-grant' 2026-08-29 08:42:03 +03:00
grenade 65011311f2 fix(infra-setup): ship helexa-ratelimit.conf to the gateway, before the vhost
CI / Classify changes (push) Successful in 15s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Config consistency (push) Successful in 10s
CI / Format (push) Successful in 9s
CI / CUDA type-check (push) Successful in 22m16s
CI / Clippy (push) Successful in 2m38s
CI / Test (push) Successful in 7m34s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
helexa-ratelimit.conf was installed on the web and angels hosts but never
on the gateway, where helexa.internal.conf lives and references the names
it defines — the limit_req_zones and, since the search-logging change,
the `helexa_noquery` log format. The gateway's copy has no such format:
`grep -c helexa_noquery` returns 0 there against 4 uses in the vhost.

The ordering in this block turns that into a live hazard rather than a
caught mistake. The sites-enabled symlink is created BEFORE `nginx -t`
runs, so a config that fails validation is already wired in. The running
nginx survives (it is never reloaded), but the next reload or reboot —
by anything, including cert renewal via step@bench.timer, enabled three
lines later — fails on it.

Found while about to run this script to install the two sudoers grants a
deploy step needs. It would have pushed a vhost the host could not parse.

deploy.yml's deploy-nginx job already ships the pair in this order for
exactly this reason; this brings the hand-run path in line with it.
2026-08-27 23:31:09 +03:00
grenade 9b0b32a43b Merge branch 'fix/restore-max-in-flight-8'
build-prerelease / Resolve version stamps + change detection (push) Successful in 19s
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build cortex binary (push) Successful in 2m16s
build-prerelease / Package cortex RPM (push) Successful in 25s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m52s
build-prerelease / Test (push) Successful in 9m42s
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Successful in 1m28s
build-prerelease / Package helexa-router RPM (push) Successful in 18s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 14s
2026-08-27 23:27:54 +03:00
grenade bc7f4a301d fix(deploy): the nginx preflight must check every grant, not a representative one
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 11s
CI / CUDA type-check (push) Successful in 22m19s
CI / Clippy (push) Successful in 2m42s
CI / Test (push) Successful in 7m59s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
It tested `sudo -n nginx -t` and nothing else. That grant had existed for
some time, so the check passed on a host whose sudoers predated the two
helexa-ratelimit.conf entries added in the very commit that introduced
the check — and the job then failed on an opaque rsync denial, seconds
after reporting the permissions were fine.

That is worse than having no preflight. A check that names a remedy while
probing the wrong permission actively misleads: the operator reads "the
sudoers are OK" and looks elsewhere.

Verified against the live host, where the split is real:

    GRANTED rsync sites-available:  OK      MISSING rsync ratelimit:      FAIL
    GRANTED ln -sfn:                OK      MISSING sha256sum ratelimit:  FAIL
    GRANTED systemctl reload:       OK

Now probes all seven. sha256sum, `nginx -t` and `systemctl` are safe to
invoke directly; rsync and ln have no harmless form that still matches
their sudoers pattern, so those use `sudo -nl`, which answers whether a
command is permitted without running it.

The failure message also now says that having run script/infra-setup.sh
before is not enough — grants accumulate in that file as deploy steps
grow, and a host keeps whatever revision it last saw. That was the actual
situation here.
2026-08-27 23:27:40 +03:00
grenade 318dc5f4e1 Merge branch 'fix/retire-qwen36-27b'
build-prerelease / Lint (fmt + clippy) (push) Blocked by required conditions
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Build cortex binary (push) Blocked by required conditions
build-prerelease / Build helexa-router binary (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 20s
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been cancelled
build-prerelease / Build neuron-ada (push) Has been cancelled
build-prerelease / Package cortex RPM (push) Has been cancelled
build-prerelease / Package helexa-router RPM (push) Has been cancelled
build-prerelease / Package helexa-tools RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-08-27 23:25:55 +03:00
grenade 21eefe7e45 fix(beast): restore max_in_flight = 8 — it never divided the KV budget
CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 16s
CI / Format (push) Has been skipped
CI / Clippy (push) Has been skipped
CI / Test (push) Has been skipped
CI / CUDA type-check (push) Has been skipped
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m47s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
I lowered this to 2 earlier today (#291) on the stated grounds that
"max_in_flight is both the admission bound and the batch engine's slot
count, and the KV budget is divided by it at load". The second half is
false, and has been since eca42510 (2026-08-15) made admission reserve
KV per request:

    admission.enter_with_kv(principal,
        kv_reservation_mb(prompt_len, requested_output, profile, cfg))

`kv_budget` is a semaphore of MiB permits; a request takes what its own
prompt needs and waits for bytes when the pool is short — which is what
`kv_max_wait_secs` is for. Nothing in the tree divides that budget by
the slot count, and the engine's `max_slots` is only the batch width.
The docs said both things twelve lines apart: `kv_max_wait_secs` was
already documented as "how long it waits for enough KV budget".

The measurement in #291 stands as data — seven days of spans, 89.6% of
busy time at one request in flight — but two things undercut the
conclusion. It cannot show that eight slots starve anything, because
they do not. And it described traffic that no longer exists: while it
was taken the router's `helexa/balanced` alias pointed web chat at a
second 27B (3b0b3a22), so per-model concurrency was split across two
models busy evicting each other. Both public tiers now resolve here, so
chat concurrency lands on this model.

Also corrects the operating docs, which carried the same claim, and the
prefix-cache trade table, whose "tokens per slot" column had silently
divided the pool by two. It is a shared pool, so the honest column is
what all concurrent requests draw from: 118,000 tokens at budget_mb =
2048, not 59,000 each.

Refs #291, #257.
2026-08-27 23:24:53 +03:00
grenade 3b0b3a2227 fix(catalogue): retire Qwen3.6-27B — a stale alias was evicting the flagship
CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 15s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m7s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 10m39s
CI / CUDA type-check (push) Successful in 19m28s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
beast served nothing for ~50 minutes on 2026-08-27 with 32 GB held on one
card, the other empty, `models: []`, and `activation.state: "ready"`.

The driver was the router's alias table. The web chat's authenticated
default is `helexa/balanced` (Chat.tsx:35), and on gallumbits that alias
still resolved to Qwen/Qwen3.6-27B while beast's resident model is the
3.8. The two sat at equal `residency_priority` deliberately, so they
displaced each other in both directions — meaning every authenticated
web-chat turn evicted the flagship and cold-loaded the 3.6 (~70 s of TP
layer loading), and the next flagship request swapped it straight back.
At 21:27–21:33 four of those overlapped and left the leader holding
accumulated shards with nothing registered.

Mutual displacement is not the bug; #99 wants it and the equal ranks were
right for the A/B they were set up for (#251). The bug is that the A/B
was over and the loser stayed catalogued, so a stale alias pointed real
user traffic at a model whose only remaining job was to evict the one
people were actually being served.

Retires the 3.6 from models.toml and repoints every operator-facing
mention at the 3.8: the docs under helexa.ai/content/docs/2-operating/,
the example catalogues and router/acp configs, and the helexa-acp README
(which told readers to set a default_model that would now 404).

Deliberately NOT renamed: doc/benchmarks.md and
crates/neuron/tests/fixtures/numerical/ record measurements and a parity
snapshot actually taken against the 3.6 — rewriting those would falsify
the record. Test fixtures elsewhere keep the old string too; it is an
arbitrary model id in those, and the churn would bury the real diff.

The live alias on gallumbits was corrected and the router restarted
ahead of this commit, since it was the active cause; helexa-router.toml
is gitignored, so that file is still hand-synced (#287).
2026-08-27 22:34:40 +03:00
grenade 2d1a4ebb2f Merge branch 'fix/nginx-sync-must-not-block-cortex-deploy'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-08-27 13:24:03 +03:00
grenade d87327c3a2 fix(deploy): the nginx sync must be its own job, not a step that gates the fleet
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Config consistency (push) Successful in 6s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 6m43s
CI / CUDA type-check (push) Successful in 20m55s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Run 1284 proved the first attempt wrong, and worse than predicted. The
sync step hard-fails until an operator runs script/infra-setup.sh to
install its sudoers grant; a failed step fails its job; and
`deploy-neurons` has `needs: [deploy-cortex]`. So a missing nginx
permission skipped the rollout of all three neuron flavours. The fleet
sat two commits behind on eb783c3 while the run reported nothing worse
than "deploy-cortex: failure" — the neuron jobs were *skipped*, which
reads like "nothing to do".

Moving the step later within deploy-cortex, as the previous commit did,
does not fix this. The job still ends red and the dependents still skip.
Order was never the problem.

`deploy-nginx` now stands alone with no `needs`, so its failure is a
statement about the vhost and nothing else. It still fails hard —
drift between the repo copy and the live file is a real defect and
should be red — but it can no longer take the fleet with it.

#287 names this trap: adding a sync step before its sudoers grant exists
breaks the deploy. Having written that down and then done it anyway, the
narrower rule is now recorded beside the job: a step whose prerequisite
is installed by hand does not belong in a job that other jobs depend on.

Refs #287.
2026-08-27 13:02:09 +03:00
grenade a17b3acb70 fix(deploy): the nginx sync must not hold the cortex binary hostage
CI / CUDA type-check (push) Blocked by required conditions
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Config consistency (push) Successful in 16s
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m31s
CI / Test (push) Successful in 6m40s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
The step I added in c6c6fc8a sat before the cortex package update, and
it hard-fails when `gitea_ci` lacks the nginx sudoers grant — which is
installed by hand, by `script/infra-setup.sh`, and has not been run yet.
So until an operator runs it, an unrelated permission blocks the gateway
binary from deploying at all.

That is the exact trap #287 names: adding the sync step before its
sudoers grant exists breaks every deploy. Writing it down and then
walking into it the same day is worth the comment now beside the step.

Moved after `Deploy cortex`, with the reasoning recorded so it does not
drift back. The hard failure stays — a job that could not sync the vhost
should be red, because it did not do everything it promised — but it now
fails after the package work rather than instead of it.
2026-08-27 12:36:47 +03:00
grenade c6c6fc8a50 Merge branch 'feat/nginx-search-alias-and-sync'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m12s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 3m6s
build-prerelease / Build neuron-ada (push) Successful in 3m16s
build-prerelease / Test (push) Successful in 9m39s
build-prerelease / Build neuron-blackwell (push) Successful in 17m3s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m2s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m8s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m6s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 26s
2026-08-27 12:33:49 +03:00
grenade 192030dff5 Merge branch 'fix/wire-surface-in-neuron-spans'
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Build neuron-blackwell (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 14s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m1s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 2m47s
build-prerelease / Build neuron-ada (push) Has been cancelled
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-08-27 12:26:01 +03:00
grenade 24b8de7f82 Merge branch 'fix/295-ignore-tiny-prefill-samples' 2026-08-27 12:26:01 +03:00
grenade 432f8009ce feat(nginx): accept POST on search, and keep queries out of the logs
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 14s
CI / Format (push) Successful in 8s
CI / Clippy (push) Successful in 2m33s
CI / CUDA type-check (push) Successful in 20m21s
CI / Test (push) Successful in 7m4s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
**Queries were being logged.** The `main` log format records
`$request` — method, path *and query string* — so every search a user
typed was written to /var/log/nginx/access.log beside their IP, for
authenticated and anonymous visitors alike:

    10.6.0.193 - - [27/Aug/2026] "GET /tools/web_search?q=bansko+weather&format=json HTTP/2.0" 200

A `helexa_noquery` format logs `$request_method $uri` instead, so the
operational signal survives — who, when, which endpoint, what status,
how big, how long — and the query does not. Rate-limit debugging and
abuse detection need counts, statuses and addresses, never search terms.

Applied to `/tools/fetch` as well: its `url=` parameter is the page a
user is having read to them, which is as revealing as a search query and
was equally in the clear.

**POST is now accepted** on both search locations. It was GET-only
because the SPA only GETs — least privilege by default rather than a
considered restriction. SearXNG accepts both, some clients POST, and a
POST keeps the query out of any intermediary's request line, which is
the same concern the log format addresses. `client_max_body_size 8k`
bounds it: a search query is small, and without a bound this becomes a
way to push arbitrary bytes at the upstream.

The log format lives in helexa-ratelimit.conf because `log_format` is
http-context only, and that file is already the http-context include for
this feature. That creates a dependency the deploy has to respect: the
vhost now references a name defined there, so the sync ships
helexa-ratelimit.conf *first*. Shipping the vhost alone would fail
`nginx -t` on an unknown log format — caught by the gate, but leaving
the two files out of step.

Both files validated together by installing them into the live paths on
hanzalova, running `nginx -t`, and restoring with a verified diff.
2026-08-27 12:10:48 +03:00
grenade 98ba45cbb4 feat(nginx): serve SearXNG's own path, and deploy the vhost from CI
CI / Test (push) Blocked by required conditions
CI / CUDA type-check (push) Blocked by required conditions
CI / Config consistency (push) Successful in 10s
CI / Classify changes (push) Successful in 20s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 11s
CI / Clippy (push) Successful in 2m38s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Three coupled changes; the first is useless without the second.

**`location = /search`.** `/tools/web_search` is our name for the
SearXNG upstream and a generic SearXNG client cannot guess it — they
take a base URL and append `/search`, because that is what every
instance serves. Measured on the live edge:

    /tools/web_search?q=…&format=json  -> application/json, SearXNG output
    /search?q=…&format=json            -> text/html (the SPA catch-all)

So such a client gets HTTP 200 with an HTML body and fails at JSON
parsing — which sends people looking for a bug in their own parser. A
404 would at least be honest. The new block is an alias, not a laxer
door: same upstream, same `helexa_tools` rate-limit zone, same
`limit_except GET`.

**The deploy now syncs the vhost.** `asset/nginx/*.conf` has been
version-controlled from the start but only ever installed by
`script/infra-setup.sh`, a one-time run-by-hand script — so the repo
copy and the live file could diverge with nothing to detect it. That is
the #252 shape: two sources of truth for one thing and nothing forcing
agreement. Digest-compared, `nginx -t` gates the reload, and a config
nginx rejects leaves the previous one serving.

**sites-enabled becomes a symlink.** On this host it was a *regular
file*, not a link to sites-available — two independent copies, of which
nginx reads one (`conf.d/sites-enabled.conf` includes
`sites-enabled/*.conf` and nothing includes sites-available). An
operator editing "the config" would reasonably edit sites-available,
watch `nginx -t` pass, reload, and see nothing change. The sync
(re)points the link as its mechanism rather than as a separate chore, so
the layout cannot drift back.

Validated by installing the candidate into sites-enabled on the real
host, running `nginx -t`, and restoring — `nginx -t` does not reload, so
the running config was never at risk. Worth noting the first attempt
tested sites-available and proved nothing, because nginx does not
include it; that is the same trap the symlink change exists to close.

**Requires one manual step before the next deploy.** The nginx sudoers
entries are added to `asset/sudoers.d/cortex-host.conf`, which is
installed by `script/infra-setup.sh`. Until that is re-run once, the new
deploy step fails — deliberately, with a preflight that names the remedy
rather than surfacing an opaque rsync permission error.
2026-08-27 12:05:35 +03:00
grenade 522682282c fix(observability): record which API surface a request arrived on
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 9m12s
CI / CUDA type-check (push) Successful in 28m29s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
neuron translates `/v1/responses` into a chat request at the door
(`openai_responses::request_to_chat`) and every line after that point is
emitted by the shared chat path. So the journal showed

    tp_chat_stream{req_id=… model=…}: chat_completion (stream): done

for every request, whatever the caller used. That is not a missing
detail; it asserts something false. It cost a real investigation into
why a Responses-configured client "was using chat completions" — it was
not, and nothing in neuron's journal could have said so.

The `tp_chat_stream` span even carried a comment claiming the journal
"can be reconstructed regardless of which API surface the client hit",
while recording nothing that would allow it.

Both spans now carry `wire=openai-chat` or `wire=openai-responses`,
threaded from the four handler entry points alongside the `principal`
that already made the same trip. The strings are deliberately the ones
`cortex-gateway`'s `log_inbound` already emits, so `journalctl | grep
openai-responses` spans both services instead of stopping at the
gateway.

A `WireSurface` enum rather than a `&'static str` parameter: with four
call sites feeding two shared inference paths, a bare string is one
copy-paste away from a span that lies in a new way.

The value is on the span, so every line already being emitted gains it —
finish lines with `reasoning_tokens` / `cached_tokens` / `finish_reason`,
rendered-prompt traces, admission rejections. Those are where the useful
per-turn data lives, and none of it could be attributed to a surface
before.

Note the TP call site is `#[cfg(feature = "cuda")]` and therefore
invisible to a local build: the CPU build was green with that one
argument missing. Caught by grepping the call sites rather than trusting
`cargo build`, which is the second time today that gate has mattered.
2026-08-27 11:35:41 +03:00
grenade 176ce47349 fix(context): ignore prefill samples too small to measure throughput
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 16s
CI / Format (push) Successful in 9s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m22s
CI / Clippy (push) Successful in 2m37s
CI / Test (push) Successful in 6m50s
CI / CUDA type-check (push) Successful in 28m29s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
A restart collapsed the advertised context to a sixth of what the host
can serve, and stayed there until real traffic happened along. beast,
minutes after a deploy:

    prefill_tok_per_sec=145.16  throughput_ceiling=17419  -> context=17408
    limit: {"context": 17408, "input": 9216, ...}

against a `vram_ceiling` of 182,560 and a hard ceiling of 131,072.

The cause is not a missing guard — the guard exists and is displaced.
`derive_limit` falls back to `bootstrap_prefill_tok_per_sec` (800,
≈96k context) only when the EMA has *no* sample, and the deploy's own
smoke probe supplies one: 19 tokens in ~130 ms, which is almost entirely
fixed per-request cost and reads as 145 tok/s on a host that measures
~1,400. One probe replaced a conservative estimate with a wrong one.

`fold_rate` already intended to prevent this — "No-op for degenerate
inputs so a probe request or a clock blip can't poison the average" —
but only guarded zero tokens and zero elapsed, not the small-prompt case
that actually occurs.

Prefill samples below 512 tokens are now ignored. That excludes the
smoke probe and the short auxiliary calls agent harnesses make (observed
at 19, 143 and 469 tokens) while admitting ordinary prompts: at
~1,400 tok/s a 512-token prefill runs ~370 ms, so fixed overhead is
about a tenth of it rather than all of it.

Fixed here rather than by warming the host on deploy, which would have
made one caller's experience better once and left the next restart to
rediscover it.

Prefill only. `record_decode` has the same shape of flaw — a two-token
completion is equally overhead-dominated — but it feeds the #137 metrics
rather than the advertised limit, so it is left for a change that can be
attributed on its own.

Four tests: the 19-token probe leaves the EMA unset so the bootstrap
value stands, the 143/469-token auxiliary calls likewise, a
representative prefill still records and still tracks a shift, and the
threshold itself counts.

Docs updated: the operating page no longer tells operators to warm a
restarted host by hand.
2026-08-27 10:45:21 +03:00
grenade 00acd283b6 docs(operating): document every operator setting a neuron exposes
build-prerelease / Resolve version stamps + change detection (push) Successful in 16s
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
`config.rs` exposed 35 operator-settable keys and 17 appeared nowhere in
the operating docs — including every one this week's defects turned on.
An operator had no way to discover them, and #297 opened because two of
them hid real bugs: `prefix_cache.budget_mb` (the #292 cache plateau)
and `bootstrap_prefill_tok_per_sec` (which exists to prevent #295's
cold-start collapse and is defeated by the first tiny request).

Now 0 of 35 are undocumented.

Written as guidance rather than a key dump. Each lever says what it
controls, what the default is *and why*, and — the part that makes a
reference usable — **what symptom would make you reach for it**.
`budget_mb` is useless as "snapshot byte budget"; it is useful as "if
`cached_tokens` stops growing mid-session, this is why".

New sections in 03-neurons.md:

- **Sampling.** The full precedence chain (request > operator > model >
  built-in), a symptom-to-knob table, and why `presence_penalty` is the
  right lever for a repeating think block while widening `repeat_last_n`
  is the wrong one — the penalty is token-level and source code repeats
  tokens legitimately. Carries the measured finding that 86% of think
  blocks over 60k characters contained verbatim repeated sentences.
- **Reasoning.** `preserve_thinking` framed as the model's own template
  control and an unmeasured experiment rather than a recommendation, and
  `answer_reserve_tokens` with the measurement that motivated it (given
  16,384 tokens the model used 16,384 and returned no answer) plus the
  trade it introduces — a reserve that fires interrupts deliberation and
  the answer tends to state an intention rather than a conclusion.
- **The prefix cache.** Including the counter-intuitive direction: the
  snapshot budget is reserved *before* the KV budget, so raising it too
  far starves KV and shortens the sessions whose snapshots the larger
  cache was meant to hold. Table of the measured 1:1 trade.
- **Image models.** Why the text encoder defaults to CPU, what
  `te_resident` costs on each device, and why `max_dim` rejects before
  admission — a device OOM poisons the worker context, so refusing early
  is the difference between one rejected request and a restart.
- **Where weights come from.** `sources`, `default_source`, `auth_env`
  (an env var name, never a token) and per-source `cache_dir`, with the
  collision that shared caches cause.

05-context-limits.md gains the three settings that shape the derived
limit, and notes that `bootstrap_prefill_tok_per_sec` only applies when
the EMA has no sample at all — so a host that restarted recently should
be warmed before its `/v1/models` is trusted.

Two corrections while in here: the concurrency example advertised
`max_in_flight = 8`, which #291 showed divides the KV budget eight ways
and leaves a single long session unable to use the advertised context;
the example and its guidance now reflect the measured production
setting, with the request-span data behind it.
2026-08-27 10:35:34 +03:00
grenade eb783c345e Merge branch 'fix/300-anthropic-streaming-thinking'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Build helexa-bench binary (push) Successful in 2m8s
build-prerelease / Build neuron-ada (push) Successful in 2m27s
build-prerelease / Build cortex binary (push) Successful in 2m28s
build-prerelease / Build neuron-ampere (push) Successful in 2m33s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m32s
build-prerelease / Package helexa-bench RPM (push) Successful in 24s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m58s
build-prerelease / Test (push) Successful in 7m56s
build-prerelease / Build neuron-blackwell (push) Successful in 21m49s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Successful in 24s
build-prerelease / Package cortex RPM (push) Successful in 28s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m9s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m15s
build-prerelease / Build helexa-router binary (push) Successful in 1m29s
build-prerelease / Package helexa-router RPM (push) Successful in 17s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m13s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 28s
2026-08-27 10:04:24 +03:00
83 changed files with 3103 additions and 163 deletions
+12
View File
@@ -177,6 +177,12 @@ jobs:
steps:
- uses: actions/checkout@v4
- run: python3 script/check-config-consistency.py
# Two files describe one routing decision and nothing forced them
# to agree. On 2026-08-27 the router still aliased
# `helexa/balanced` to a model retired from the catalogue, so
# every web-chat turn evicted beast's resident flagship to
# cold-load it. Same shape as the check above (#252/#287).
- run: python3 script/check-alias-targets.py
web:
name: Web (lint + typecheck + i18n + build)
@@ -199,6 +205,12 @@ jobs:
# reached main exactly that way.
- run: npm run lint
- run: npm run typecheck
# Behaviour, not just types. Gating from the first commit was the
# deliberate call (#304): with no existing suite there is nothing to
# stabilise, so a warn-first period would only create a window in
# which a vacuous test lands and gets normalised — and a suite
# nobody trusts is worse than none, because it gets cited.
- run: npm test
# 42 locales with strict key parity is a promise that breaks
# quietly. Adding an English string without translating it must fail
# here, not in review, and certainly not in front of somebody
+261
View File
@@ -59,6 +59,12 @@ jobs:
steps:
- uses: actions/checkout@v4
- run: python3 script/check-config-consistency.py
# Two files describe one routing decision and nothing forced them
# to agree. On 2026-08-27 the router still aliased
# `helexa/balanced` to a model retired from the catalogue, so
# every web-chat turn evicted beast's resident flagship to
# cold-load it. Same shape as the check above (#252/#287).
- run: python3 script/check-alias-targets.py
deploy-cortex:
needs: [config-check]
@@ -526,6 +532,161 @@ jobs:
# dependency on cortex or the neurons (the sweep loop is version-aware
# and picks up whatever each neuron reports whenever). So it runs
# alongside the cortex→neurons chain rather than after it.
# The vhost sync gets its own job, with NO `needs`, because it is not
# part of deploying the gateway binary and must not be able to stop
# one.
#
# It was briefly a step inside deploy-cortex, and that broke the fleet
# on its first run (2026-08-27, run 1284): the step hard-fails until an
# operator has run script/infra-setup.sh to install its sudoers grant,
# a failed step fails the job, and `deploy-neurons` has
# `needs: [deploy-cortex]` — so a missing nginx permission silently
# skipped the rollout of all three neuron flavours. The fleet stayed
# two commits behind while the run reported only "deploy-cortex:
# failure".
#
# #287 names this ordering trap. The lesson is narrower than "order the
# steps": a step whose prerequisite is installed by hand does not
# belong in a job that something else depends on, because its failure
# is not a statement about that job. Moving it later within
# deploy-cortex would not have helped — the job still ends red and the
# dependents still skip.
#
# Failing hard here is still right: drift between the repo and the live
# file is a real defect, and this job going red says that and nothing
# else.
deploy-nginx:
runs-on: fedora-43
if: >-
${{
github.event_name == 'workflow_dispatch'
|| github.event.workflow_run.conclusion == 'success'
}}
steps:
- uses: actions/checkout@v4
- name: SSH init
run: |
mkdir -p ~/.ssh
echo "${DEPLOY_KEY}" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new \
gitea_ci@hanzalova.internal 'hostname -f'
# nginx vhost config for the internal site. Version-controlled in
# `asset/nginx/` since the beginning, but until now only ever
# installed by `script/infra-setup.sh` — a one-time, run-by-hand
# script. So the repo copy and the live file could diverge with
# nothing to detect it, which is the #252 shape: two sources of
# truth for one thing and nothing forcing agreement.
#
# Writes to sites-available and (re)points sites-enabled at it as
# a symlink. On this host sites-enabled was a *regular file*, not a
# symlink — two independent copies, of which nginx reads one. An
# operator editing "the config" would reasonably edit
# sites-available, watch `nginx -t` pass, reload, and see nothing
# change. Making the link canonical is part of the sync rather
# than a separate chore, so the layout cannot drift back.
#
# `nginx -t` gates the reload: a config that fails validation is
# left un-reloaded with the previous one still serving.
- name: Sync internal nginx vhost
run: |
src="asset/nginx/helexa.internal.conf"
# The sudoers drop-in that permits this is installed by
# script/infra-setup.sh, which is run by hand, so the live copy
# can lag the repo by however long it has been since anyone ran
# it. Check EVERY grant the step needs, not a representative
# one: the first version of this preflight tested `nginx -t`
# alone, which had existed for ages, so it passed and the job
# then died on the two helexa-ratelimit.conf grants added in the
# same commit as the check. A preflight that names a remedy
# while probing the wrong permission is worse than none — it
# tells you the sudoers are fine seconds before they are not.
#
# Each probe runs the command in a harmless form that still
# matches its sudoers pattern (sha256sum reads; nginx -t and
# `systemctl show` do not mutate). rsync and ln have no such
# form, so they are checked with `sudo -nl`, which reports
# whether a command is permitted without running it.
missing=""
probe() { # probe <description> <command...>
local what="$1"; shift
ssh gitea_ci@hanzalova.internal "$@" >/dev/null 2>&1 \
|| missing="${missing}
- ${what}"
}
probe "nginx -t" \
'sudo -n /usr/sbin/nginx -t'
probe "sha256sum /etc/nginx/sites-available/helexa.internal.conf" \
'sudo -n /usr/bin/sha256sum /etc/nginx/sites-available/helexa.internal.conf'
probe "sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf" \
'sudo -n /usr/bin/sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf'
probe "rsync -> /etc/nginx/sites-available/helexa.internal.conf" \
'sudo -nl /usr/bin/rsync -a x /etc/nginx/sites-available/helexa.internal.conf'
probe "rsync -> /etc/nginx/conf.d/helexa-ratelimit.conf" \
'sudo -nl /usr/bin/rsync -a x /etc/nginx/conf.d/helexa-ratelimit.conf'
probe "ln -sfn sites-available -> sites-enabled" \
'sudo -nl /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf /etc/nginx/sites-enabled/helexa.internal.conf'
probe "systemctl reload nginx" \
'sudo -nl /usr/bin/systemctl reload nginx'
if [ -n "${missing}" ]; then
echo "FAIL: gitea_ci is missing nginx sudoers grants on hanzalova:${missing}" >&2
echo "" >&2
echo " Re-run 'script/infra-setup.sh' once from an operator shell to" >&2
echo " install the current asset/sudoers.d/cortex-host.conf. Having run" >&2
echo " it before is not enough — grants are added to that file as deploy" >&2
echo " steps grow, and the host keeps whatever revision it last saw." >&2
exit 1
fi
rl="asset/nginx/helexa-ratelimit.conf"
local_sum=$(sha256sum "${src}" | cut -d' ' -f1)
local_rl=$(sha256sum "${rl}" | cut -d' ' -f1)
remote_sum=$(ssh gitea_ci@hanzalova.internal \
'sudo sha256sum /etc/nginx/sites-available/helexa.internal.conf 2>/dev/null | cut -d" " -f1' || true)
remote_rl=$(ssh gitea_ci@hanzalova.internal \
'sudo sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf 2>/dev/null | cut -d" " -f1' || true)
link_ok=$(ssh gitea_ci@hanzalova.internal \
'test -L /etc/nginx/sites-enabled/helexa.internal.conf && echo yes || echo no')
if [ "${local_sum}" = "${remote_sum}" ] \
&& [ "${local_rl}" = "${remote_rl}" ] \
&& [ "${link_ok}" = "yes" ]; then
echo "nginx config unchanged, sites-enabled is a symlink — nothing to do"
exit 0
fi
# Order matters: the vhost references names defined at http
# context by helexa-ratelimit.conf (`limit_req_zone`s and the
# `helexa_noquery` log format). Ship that first, or a vhost
# referencing a not-yet-installed name fails `nginx -t` and the
# deploy stops with the two files out of step.
echo "syncing nginx rate-limit/log-format: ${remote_rl:-<absent>} -> ${local_rl}"
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
"${rl}" gitea_ci@hanzalova.internal:/etc/nginx/conf.d/helexa-ratelimit.conf
echo "syncing nginx vhost: ${remote_sum:-<absent>} -> ${local_sum} (symlink=${link_ok})"
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
"${src}" gitea_ci@hanzalova.internal:/etc/nginx/sites-available/helexa.internal.conf
ssh gitea_ci@hanzalova.internal 'bash -s' <<'NGINX'
set -eu
sudo /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf \
/etc/nginx/sites-enabled/helexa.internal.conf
if ! sudo /usr/sbin/nginx -t; then
echo "FAIL: nginx -t rejected the synced config; not reloading" >&2
exit 1
fi
sudo /usr/bin/systemctl reload nginx
echo "nginx reloaded"
NGINX
# Roll helexa-router + helexa-upstream onto the federation service host
# (gallumbits). Same manifest-gated dnf pattern as deploy-cortex; the
# two services are independent daemons on one host, so one job handles
@@ -539,6 +700,10 @@ jobs:
|| github.event.workflow_run.conclusion == 'success'
}}
steps:
# Needed since "Sync router config" reads helexa-router.toml out of
# the tree. This job managed without a checkout while it only ran
# package upgrades over ssh.
- uses: actions/checkout@v4
- name: SSH init
run: |
mkdir -p ~/.ssh
@@ -606,6 +771,96 @@ jobs:
done
DEPLOY
# Router config (#287). Tracked because it holds no secrets, and
# synced here because being untracked is what let its
# `helexa/balanced` alias keep pointing at a retired model until
# beast fell over on 2026-08-27.
#
# After the package step, deliberately: this needs a sudoers grant
# installed by the hand-run script/infra-setup.sh, so on a host that
# has not seen the current grants it fails — and a step that fails
# before the package work would hold the service binaries hostage to
# an unrelated permission. deploy-gallumbits has no dependent jobs,
# so the blast radius stops here either way.
#
# Restarts only on a digest change: the router re-reads its aliases
# at startup, and a needless restart drops in-flight federation
# requests.
- name: Sync router config
run: |
missing=""
probe() {
local what="$1"; shift
ssh gitea_ci@gallumbits.kosherinata.internal "$@" >/dev/null 2>&1 \
|| missing="${missing}
- ${what}"
}
probe "sha256sum /etc/helexa-router/helexa-router.toml" \
'sudo -n /usr/bin/sha256sum /etc/helexa-router/helexa-router.toml'
probe "rsync -> /etc/helexa-router/helexa-router.toml" \
'sudo -nl /usr/bin/rsync -a x /etc/helexa-router/helexa-router.toml'
probe "systemctl restart helexa-router.service" \
'sudo -nl /usr/bin/systemctl restart helexa-router.service'
if [ -n "${missing}" ]; then
echo "FAIL: gitea_ci is missing sudoers grants on gallumbits:${missing}" >&2
echo "" >&2
echo " Re-run 'script/infra-setup.sh' once from an operator shell to" >&2
echo " install the current asset/sudoers.d/gallumbits-host.conf." >&2
exit 1
fi
local_sum=$(sha256sum helexa-router.toml | cut -d' ' -f1)
remote_sum=$(ssh gitea_ci@gallumbits.kosherinata.internal \
'sudo -n /usr/bin/sha256sum /etc/helexa-router/helexa-router.toml 2>/dev/null | cut -d" " -f1' || true)
if [ "${local_sum}" = "${remote_sum}" ]; then
echo "router config unchanged (${local_sum})"
exit 0
fi
# Two questions, not one. The bytes decide whether to SYNC; the
# parsed settings decide whether to RESTART. The first run of
# this step proved why they differ: the only change was a
# comment header, and it bounced the router anyway, dropping
# whatever federation requests were in flight. The router reads
# its aliases at startup, so a comment is not a reason to
# interrupt anyone.
#
# Reading the remote copy into CI is safe here precisely because
# this file holds no secrets — the same property that lets it be
# tracked at all.
#
# The `cat` grant is deliberately NOT in the preflight above: if
# it is missing the read yields nothing, the comparison reports
# "changed", and the step restarts on any byte difference. That
# is the old behaviour, not a broken one, so a host mid-way
# through a sudoers rollout degrades instead of failing.
remote_copy="$(mktemp)"
trap 'rm -f "${remote_copy}"' EXIT
ssh gitea_ci@gallumbits.kosherinata.internal \
'sudo -n /usr/bin/cat /etc/helexa-router/helexa-router.toml' \
> "${remote_copy}" 2>/dev/null || true
settings_changed=yes
if python3 script/toml-settings-equal.py "${remote_copy}" helexa-router.toml; then
settings_changed=no
fi
echo "syncing helexa-router.toml: ${remote_sum:-<absent>} -> ${local_sum}"
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
helexa-router.toml \
gitea_ci@gallumbits.kosherinata.internal:/etc/helexa-router/helexa-router.toml
if [ "${settings_changed}" = yes ]; then
ssh gitea_ci@gallumbits.kosherinata.internal \
'sudo -n /usr/bin/systemctl restart helexa-router.service'
echo "helexa-router restarted: settings changed"
else
echo "helexa-router left running: only comments or formatting changed"
fi
- name: Ensure firewalld allows helexa-router + helexa-upstream + helexa-tools + helexa-angels
run: |
ssh gitea_ci@gallumbits.kosherinata.internal '
@@ -870,6 +1125,12 @@ jobs:
cd helexa.ai
npm ci
npm run i18n:check
# Also run here, not just in ci.yml, for the same reason
# i18n:check is: pushes to main skip ci.yml, so this is the only
# gate a merge itself passes through. Cheap, and it is the exact
# case the web CI comment describes — a defect reaching main and
# surfacing as a broken deploy afterwards.
npm test
npm run build
- name: SSH init
+14 -1
View File
@@ -8,10 +8,23 @@ helexa.ai/.env.local
*.swo
.idea/
.vscode/
# Secret-bearing service configs. These carry API keys, a JWT secret, DB
# passwords and SMTP credentials, so they are hand-synced by
# script/infra-setup.sh rather than shipped by CI (#287 Gap 1).
#
# helexa-router.toml is deliberately NOT here: it holds no secrets, only
# a listen address, the product-tier aliases and a mesh endpoint. It was
# ignored by association, and that is precisely how its `helexa/balanced`
# alias came to point at a retired model for long enough to take beast
# down (2026-08-27). A file nothing can diff is a file nothing can catch.
# Operator-owned credential layer, merged after the CI-deployed config
# by cortex / helexa-upstream / helexa-angels. Must never be committed.
secrets.toml
**/secrets.toml
cortex.toml
doc/plan/*
/target-cuda/
.claude/
helexa-router.toml
helexa-upstream.toml
helexa-angels.toml
+38 -25
View File
@@ -7,10 +7,10 @@
# ─────────────────────────────────────────────────────────────────────
# neuron.toml for beast.hanzalova.internal
#
# 2x RTX 5090 (32 GB each) — TP-2 capable. Pre-warms Qwen3.6-27B with
# q5k ISQ across both GPUs at activation, matching the validate-neuron
# 2x RTX 5090 (32 GB each) — TP-2 capable. Pre-warms Qwen3.8-27B with
# q6k ISQ across both GPUs at activation, matching the validate-neuron
# invocation: `validate-neuron.sh beast.hanzalova.internal
# Qwen/Qwen3.6-27B q5k 2`.
# Qwen/Qwen3.8-27B q6k 2`.
#
# Synced to /etc/neuron/neuron.toml by script/infra-setup.sh. Edits
# take effect after the next deploy workflow run restarts the service
@@ -29,24 +29,33 @@ name = "candle"
# count. NEURON_BATCHING=0 (systemd drop-in) is the kill switch;
# removing this section reverts to batch-1 serialization.
[harness.candle.admission]
# 2, not 8 (#291). max_in_flight is both the admission bound and the
# batch engine's slot count, and the KV budget is divided by it at load.
# At 8 that gave each slot 18,852 tokens out of 150,816 — against a
# 131,072-token advertised context, which needs 4 GB of KV per card on
# its own and 32 GB across eight slots, on a 31 GB card already holding
# ~21 GB of weights. The advertised context and the slot count were
# incompatible by roughly 7x, and an agentic session ran in a ~19k
# window while seven idle slots held reservations.
# 8. max_in_flight bounds how many requests run concurrently. It does
# NOT partition the KV budget: since #257 (`eca42510`, 2026-08-15) KV is
# a shared pool and each request reserves what its own prompt needs via
# `enter_with_kv(principal, kv_reservation_mb(prompt_len, ...))`. A
# request too big for what is free waits for bytes to come back rather
# than being handed a fixed 1/N slice.
#
# Seven days of paired request spans on this host: 89.6% of busy time
# was ONE request in flight, 93.6% was two or fewer, and utilisation was
# 4.9%. Most of what little concurrency exists is helexa-bench's own
# synthetic bursts. Two slots cover 93.6% of observed load and give each
# ~75k tokens; the batch engine is still worth 1.79x there (5.24x at
# eight is throughput nobody is asking this host for). Anything above
# two queues on admission (#53) with Retry-After rather than being
# refused.
max_in_flight = 2
# This value was briefly set to 2 (#291) on the stated grounds that "the
# KV budget is divided by it at load, so 8 gives each slot 18,852
# tokens". That was true before #257 and false after it, and #291 was
# filed and actioned after that landed. Nothing in the tree divides the
# budget by the slot count — `kv_budget` is a semaphore of MiB permits
# and the engine's `max_slots` is only the batch width.
#
# The concurrency measurement in #291 stands as data (89.6% of busy time
# at one request in flight, 4.9% utilisation over seven days) but the
# conclusion drawn from it does not, and the traffic it described has
# changed: while it was taken, the router's `helexa/balanced` alias
# pointed web chat at a second 27B, so per-model concurrency was split
# across two models that were evicting each other. With both tiers now
# resolving to this model, chat concurrency lands here.
#
# What actually bounds a busy moment is the pool: 3,689 MiB at
# 32 KiB/token/card is ~118k tokens of KV to share. Eight ordinary chat
# turns fit easily; a few very long agentic sessions will queue on bytes,
# which is the correct behaviour and is what #257 built.
max_in_flight = 8
# Snapshot budget for the cross-request prefix cache (#292).
#
@@ -83,11 +92,15 @@ max_in_flight = 2
budget_mb = 2048
[[default_models]]
# Qwen3.8-27B is the resident flagship (#251). Architecturally identical
# to the 3.6-27B it replaced — same model_type, layer counts and vision
# tower — so the load spec is unchanged. The 3.6-27B stays catalogued at
# the same residency class, so it can still be cold-loaded on demand and
# the two displace each other in either direction.
# Qwen3.8-27B is the resident flagship (#251). It replaced the
# 3.6-27B — architecturally identical, same model_type, layer counts and
# vision tower — so the load spec is unchanged.
#
# The 3.6-27B is no longer catalogued (retired 2026-08-27). While both
# were listed at the same residency class they could displace each other
# in either direction, which is what let a stale `helexa/balanced` alias
# evict this model on every authenticated web-chat turn. See models.toml
# for the incident that ended the arrangement.
model_id = "Qwen/Qwen3.8-27B"
harness = "candle"
quant = "q6k"
+21
View File
@@ -24,3 +24,24 @@ limit_req_zone $binary_remote_addr zone=helexa_tools:10m rate=30r/m;
limit_req_zone $binary_remote_addr zone=angels_invite:1m rate=20r/m;
limit_req_zone $binary_remote_addr zone=angels_auth:1m rate=12r/m;
limit_req_status 429;
# Access-log format for the search locations, which must not record what
# people searched for.
#
# The default `main` format logs `$request` — method, full URI *and
# query string*. For a search proxy that means every query a user types
# is written to /var/log/nginx/access.log next to their IP, for
# authenticated and anonymous visitors alike. SearXNG's own web UI
# defaults to POST partly to avoid exactly this.
#
# `$uri` is the path with the query stripped, so the operational signal
# survives — who called, when, which endpoint, what status, how big, how
# long — and the query itself does not. `$request_method` is kept
# explicit since `$uri` drops it.
#
# Rate-limit debugging and abuse detection both work from this: they
# need counts, statuses and client addresses, never the search terms.
log_format helexa_noquery '$remote_addr - $remote_user [$time_local] '
'"$request_method $uri $server_protocol" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $request_time';
+37 -1
View File
@@ -73,6 +73,9 @@ server {
location = /tools/fetch {
limit_req zone=helexa_tools burst=10 nodelay;
limit_except GET { deny all; }
# The `url=` parameter is as revealing as a search query: it is
# the page a user is having read to them. Same treatment.
access_log /var/log/nginx/access.log helexa_noquery;
proxy_pass http://gallumbits.kosherinata.internal:8889/fetch;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
@@ -83,7 +86,40 @@ server {
# federation service host. GET only; the SPA requests format=json.
location = /tools/web_search {
limit_req zone=helexa_tools burst=10 nodelay;
limit_except GET { deny all; }
# GET and POST: SearXNG accepts both, and a client that POSTs
# keeps its query out of any intermediary's request line. Other
# methods have no meaning here.
limit_except GET POST { deny all; }
# A search query is small. Bounding the body stops this becoming
# a way to push arbitrary bytes at the upstream.
client_max_body_size 8k;
# Never log the query — see helexa_noquery in
# conf.d/helexa-ratelimit.conf.
access_log /var/log/nginx/access.log helexa_noquery;
proxy_pass http://gallumbits.kosherinata.internal:8888/search;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_read_timeout 30s;
}
# The same upstream under SearXNG's own path, for clients that speak
# the SearXNG API rather than our app's.
#
# `/tools/web_search` is our name for it, and a generic SearXNG
# client cannot guess it: they take a base URL and append `/search`,
# because that is what every SearXNG instance serves. Without this
# block such a request falls through to `location /` and is answered
# by the SPA — HTTP 200, `text/html`, and a JSON parse error at the
# client. A 404 would at least be honest; 200-with-HTML sends people
# looking for a bug in their own parser.
#
# Same upstream, same rate-limit zone, same GET-only restriction:
# this is an alias, not a second, laxer door.
location = /search {
limit_req zone=helexa_tools burst=10 nodelay;
limit_except GET POST { deny all; }
client_max_body_size 8k;
access_log /var/log/nginx/access.log helexa_noquery;
proxy_pass http://gallumbits.kosherinata.internal:8888/search;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+12
View File
@@ -13,6 +13,18 @@ gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/cortex/models.toml
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /var/www/bench.helexa.ai/
# deploy-website rsyncs the built helexa.ai SPA into the nginx webroot.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /var/www/helexa.ai/
# deploy.yml's "Sync internal nginx vhost" step: write the vhost, make
# sites-enabled a symlink to it, validate, reload. Scoped to the one
# file and the one service — `nginx -t` is included because a reload
# must never happen on a config nginx has rejected.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/nginx/sites-available/helexa.internal.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/nginx/conf.d/helexa-ratelimit.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/nginx/sites-available/helexa.internal.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf /etc/nginx/sites-enabled/helexa.internal.conf
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/nginx -t
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl reload nginx
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start cortex.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop cortex.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now cortex.service
+25 -3
View File
@@ -4,9 +4,17 @@
# rolls out helexa-router (data plane), helexa-upstream (account/budget
# authority), helexa-tools (grounding fetcher) and helexa-angels (the
# confidential investor portal) package upgrades.
# Configs are NOT synced by CI — they carry secrets (DB password, JWT
# secret, operator tokens) and ride in script/infra-setup.sh under the
# operator's own sudo instead.
# The SECRET-BEARING configs are not synced by CI — helexa-upstream.toml
# and helexa-angels.toml carry a DB password, a JWT secret and SMTP
# credentials, and ride in script/infra-setup.sh under the operator's own
# sudo instead (#287 Gap 1, still open).
#
# helexa-router.toml is different and IS synced here: it holds no secrets,
# only a listen address, the product-tier aliases and a mesh endpoint. It
# was excluded by association with its neighbours, which is how its
# `helexa/balanced` alias came to point at a model retired from the
# catalogue and took beast down on 2026-08-27. A config CI cannot write is
# a config no check can defend.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start helexa-router.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop helexa-router.service
@@ -21,6 +29,20 @@ gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start helexa-angels.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop helexa-angels.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now helexa-angels.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload
# deploy-gallumbits' "Sync router config" step. Scoped to the one file:
# the neighbouring service configs stay operator-only until #287 Gap 1
# gives them a render-from-secrets path.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/helexa-router/helexa-router.toml
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/helexa-router/helexa-router.toml
# Read-back, so the sync can tell a settings change from a comment
# change and restart only for the former. Safe to grant and safe to read
# into CI because this file holds no secrets — the same property that
# lets it be tracked. If this grant is absent the step degrades to
# restarting on any byte change, which is why it is not in the step's
# fail-hard preflight.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/cat /etc/helexa-router/helexa-router.toml
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl restart helexa-router.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/dnf install --refresh --allowerasing -y helexa-router
gitea_ci ALL=(root) NOPASSWD: /usr/bin/dnf upgrade --refresh --allowerasing -y helexa-router
gitea_ci ALL=(root) NOPASSWD: /usr/bin/dnf install --refresh --allowerasing -y helexa-upstream
+141
View File
@@ -143,15 +143,48 @@ pub struct NeuronEndpoint {
impl GatewayConfig {
/// Load configuration from a TOML file, with environment variable overrides.
/// Env vars are prefixed with `CORTEX_` and use `__` as a separator.
///
/// A sibling `secrets.toml`, if present, is merged **after** the main
/// file and therefore wins. It exists so the two have different
/// owners: CI writes the main config, and only the operator writes
/// the secrets — see [`secrets_path`].
pub fn load(path: impl AsRef<Path>) -> Result<Self, Box<figment::Error>> {
let path = path.as_ref();
Figment::new()
.merge(Toml::file(path))
.merge(Toml::file(secrets_path(path)))
.merge(Env::prefixed("CORTEX_").split("__"))
.extract()
.map_err(Box::new)
}
}
/// `secrets.toml` beside the main config.
///
/// The split exists to make a class of outage impossible rather than
/// merely unlikely. Config that CI cannot write is config no check can
/// defend: `helexa-router.toml` was excluded from git because it sat
/// beside secret-bearing files, and its `helexa/balanced` alias then
/// pointed at a retired model for long enough to take a node down
/// (2026-08-27). The fix is not to hand CI the credentials — it is to
/// stop mixing the two in one file.
///
/// So `cortex.toml` holds structure and is deployed by CI, while
/// `secrets.toml` holds API keys and the upstream bearer, is written
/// only by the operator, and is never read, written or diffed by the
/// pipeline. Because it merges last it also wins, so a value present in
/// both is the operator's.
///
/// A missing file is not an error — figment treats an absent
/// `Toml::file` as an empty source — so a host that keeps everything in
/// one file, and every existing deployment, keeps working unchanged.
fn secrets_path(config: &Path) -> std::path::PathBuf {
config
.parent()
.unwrap_or_else(|| Path::new("."))
.join("secrets.toml")
}
impl Default for GatewayConfig {
fn default() -> Self {
Self {
@@ -170,3 +203,111 @@ impl Default for GatewayConfig {
}
}
}
#[cfg(test)]
mod secrets_layer_tests {
use super::*;
use std::io::Write;
/// Write `main` (and optionally `secrets.toml`) into a fresh directory
/// under the target dir, and load through the real code path.
fn load_with(dir: &str, main: &str, secrets: Option<&str>) -> GatewayConfig {
// `CARGO_TARGET_TMPDIR` is only set for integration tests, not for
// unit tests inside src/, so derive a unique directory instead.
let base = std::env::temp_dir().join(format!("helexa-cortex-config-{dir}"));
let _ = std::fs::remove_dir_all(&base);
std::fs::create_dir_all(&base).expect("create test dir");
let cfg = base.join("cortex.toml");
write!(
std::fs::File::create(&cfg).expect("create config"),
"{main}"
)
.expect("write");
if let Some(s) = secrets {
let p = base.join("secrets.toml");
write!(std::fs::File::create(&p).expect("create secrets"), "{s}").expect("write");
}
GatewayConfig::load(&cfg).expect("config loads")
}
const MAIN: &str = r#"
[gateway]
listen = "0.0.0.0:31313"
metrics_listen = "0.0.0.0:31314"
[eviction]
strategy = "lru"
defrag_after_cycles = 50
[entitlements]
require_auth = true
[[neurons]]
name = "beast"
endpoint = "http://beast.invalid:13131"
"#;
/// The whole point: a host with no secrets.toml behaves exactly as
/// before. Every current deployment is in this state, so if this
/// regressed the change would take the fleet down on rollout rather
/// than on adoption.
#[test]
fn an_absent_secrets_file_is_not_an_error() {
let cfg = load_with("secrets_absent", MAIN, None);
assert!(cfg.entitlements.require_auth);
assert!(
cfg.entitlements.keys.is_empty(),
"no keys configured anywhere means no keys"
);
}
/// The array-of-tables case that ruled out env-var overrides:
/// `[[entitlements.keys]]` cannot be expressed as `CORTEX_*` env vars,
/// so the second TOML layer is what carries it.
#[test]
fn secrets_file_supplies_the_entitlement_keys() {
let cfg = load_with(
"secrets_keys",
MAIN,
Some(
r#"
[[entitlements.keys]]
key = "sk-test-aaa"
account_id = "acct-a"
key_id = "a"
[[entitlements.keys]]
key = "sk-test-bbb"
account_id = "acct-b"
key_id = "b"
"#,
),
);
assert_eq!(cfg.entitlements.keys.len(), 2);
assert_eq!(cfg.entitlements.keys[0].key, "sk-test-aaa");
assert_eq!(cfg.entitlements.keys[1].account_id, "acct-b");
assert!(
cfg.entitlements.require_auth,
"structure from the CI-owned file must survive the merge"
);
}
/// Ordering is the safety property. If CI ever ships a placeholder in
/// the main file, the operator's real value must still win — otherwise
/// a deploy silently swaps a live credential for a dummy.
#[test]
fn the_operators_value_wins_over_the_deployed_one() {
let cfg = load_with(
"secrets_precedence",
&format!(
"{MAIN}\n[upstream]\nenabled = true\nurl = \"https://example.invalid\"\nbearer = \"PLACEHOLDER\"\n"
),
Some("[upstream]\nbearer = \"real-token\"\n"),
);
assert_eq!(
cfg.upstream.bearer, "real-token",
"secrets.toml merges last, so it wins"
);
assert_eq!(
cfg.upstream.url, "https://example.invalid",
"fields the operator does not override keep the deployed value"
);
}
}
+5 -5
View File
@@ -80,7 +80,7 @@ The fastest path: env-var single-endpoint config.
```sh
export HELEXA_ACP_BASE_URL=http://hanzalova.internal:31313/v1
export HELEXA_ACP_MODEL=Qwen/Qwen3.6-27B
export HELEXA_ACP_MODEL=Qwen/Qwen3.8-27B
helexa-acp # speaks ACP over stdin/stdout; not interactive
```
@@ -117,7 +117,7 @@ default_endpoint = "helexa"
name = "helexa"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.6-27B"
default_model = "Qwen/Qwen3.8-27B"
max_tokens = 8192
context_window = 32768
@@ -138,7 +138,7 @@ default_model = "claude-opus-4"
Restart Zed. The model dropdown lists every model from every
configured endpoint with the `endpoint:model` selector
(`helexa:Qwen/Qwen3.6-27B`, `openrouter:anthropic/claude-opus-4`,
(`helexa:Qwen/Qwen3.8-27B`, `openrouter:anthropic/claude-opus-4`,
…). Switch mid-session; the next prompt routes to the new endpoint.
When only one endpoint is configured the prefix is dropped (model
@@ -166,7 +166,7 @@ Copy-pasteable blocks. Mix and match.
name = "helexa"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.6-27B"
default_model = "Qwen/Qwen3.8-27B"
max_tokens = 8192
context_window = 32768
```
@@ -241,7 +241,7 @@ vLLM.
name = "lan"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.6-27B"
default_model = "Qwen/Qwen3.8-27B"
[[endpoints]]
name = "cloud"
+29 -1
View File
@@ -214,11 +214,39 @@ fn default_notify_to() -> String {
}
impl AngelsConfig {
/// A sibling `secrets.toml`, if present, merges after the main file
/// and wins — see [`secrets_path`].
pub fn load(path: impl AsRef<Path>) -> anyhow::Result<Self> {
let path = path.as_ref();
let cfg: Self = Figment::new()
.merge(Toml::file(path.as_ref()))
.merge(Toml::file(path))
.merge(Toml::file(secrets_path(path)))
.merge(Env::prefixed("ANGELS_").split("__"))
.extract()?;
Ok(cfg)
}
}
/// `secrets.toml` beside the main config.
///
/// The main file holds structure and is deployed by CI; this one holds
/// credentials, is written only by the operator, and is never read,
/// written or diffed by the pipeline. Merging it last means the
/// operator's value wins, so a deploy cannot swap a live credential for
/// a placeholder.
///
/// The split exists because config CI cannot write is config no check
/// can defend — `helexa-router.toml` was kept out of git for sitting
/// beside secret-bearing files, and its stale alias then took a node
/// down (2026-08-27). The answer is to separate secrets from structure,
/// not to hand the pipeline the credentials.
///
/// A missing file is not an error: figment treats an absent
/// `Toml::file` as an empty source, so existing single-file deployments
/// keep working untouched.
fn secrets_path(config: &Path) -> std::path::PathBuf {
config
.parent()
.unwrap_or_else(|| Path::new("."))
.join("secrets.toml")
}
+29
View File
@@ -287,15 +287,44 @@ fn default_from_addr() -> String {
impl UpstreamConfig {
/// Load from a TOML file with `UPSTREAM_`-prefixed env overrides
/// (`__` nesting separator).
///
/// A sibling `secrets.toml`, if present, merges after the main file
/// and wins — see [`secrets_path`].
pub fn load(path: impl AsRef<Path>) -> Result<Self, Box<figment::Error>> {
let path = path.as_ref();
Figment::new()
.merge(Toml::file(path))
.merge(Toml::file(secrets_path(path)))
.merge(Env::prefixed("UPSTREAM_").split("__"))
.extract()
.map_err(Box::new)
}
}
/// `secrets.toml` beside the main config.
///
/// The main file holds structure and is deployed by CI; this one holds
/// credentials, is written only by the operator, and is never read,
/// written or diffed by the pipeline. Merging it last means the
/// operator's value wins, so a deploy cannot swap a live credential for
/// a placeholder.
///
/// The split exists because config CI cannot write is config no check
/// can defend — `helexa-router.toml` was kept out of git for sitting
/// beside secret-bearing files, and its stale alias then took a node
/// down (2026-08-27). The answer is to separate secrets from structure,
/// not to hand the pipeline the credentials.
///
/// A missing file is not an error: figment treats an absent
/// `Toml::file` as an empty source, so existing single-file deployments
/// keep working untouched.
fn secrets_path(config: &Path) -> std::path::PathBuf {
config
.parent()
.unwrap_or_else(|| Path::new("."))
.join("secrets.toml")
}
#[cfg(test)]
mod tests {
use super::*;
+17 -4
View File
@@ -2,6 +2,7 @@
use crate::activation::ActivationTracker;
use crate::harness::HarnessRegistry;
use crate::harness::candle::WireSurface;
use crate::harness::candle::{CandleHarness, InferenceError};
use crate::harness::preflight::PreflightError;
use crate::health::HealthCache;
@@ -362,7 +363,7 @@ async fn chat_completions(
if req.stream.unwrap_or(false) {
match candle
.chat_completion_stream_with(req, chat_config, principal)
.chat_completion_stream_with(req, chat_config, principal, WireSurface::Chat)
.await
{
Ok(rx) => {
@@ -381,7 +382,10 @@ async fn chat_completions(
Err(e) => inference_error_response(e),
}
} else {
match candle.chat_completion(req, principal).await {
match candle
.chat_completion(req, principal, WireSurface::Chat)
.await
{
Ok(resp) => Json(resp).into_response(),
Err(e) => inference_error_response(e),
}
@@ -434,7 +438,13 @@ async fn responses(
if stream_requested {
match candle
.responses_stream(chat_req, response_id, message_item_id, principal)
.responses_stream(
chat_req,
response_id,
message_item_id,
principal,
WireSurface::Responses,
)
.await
{
Ok(rx) => {
@@ -458,7 +468,10 @@ async fn responses(
// and translate the result. We don't currently re-tokenise
// to compute usage; the harness returns it via the chat
// response and we pass it through.
match candle.chat_completion(chat_req, principal).await {
match candle
.chat_completion(chat_req, principal, WireSurface::Responses)
.await
{
Ok(chat_resp) => {
// Extract the assistant text (chat completions
// always emits one choice on the candle path).
+155 -12
View File
@@ -669,6 +669,34 @@ fn requested_sampling(request: &ChatCompletionRequest) -> super::sampling::Reque
}
}
/// Which API surface a request arrived on.
///
/// Carried into the tracing span because neuron translates
/// `/v1/responses` into a chat request at the door
/// (`openai_responses::request_to_chat`) and every line after that
/// point — spans, finish lines, rendered prompts — is emitted by the
/// shared chat path. Without this the journal reads as though every
/// client used `/v1/chat/completions`, which is not a missing detail
/// but an actively wrong one: it says something false about the caller.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum WireSurface {
/// `POST /v1/chat/completions`
Chat,
/// `POST /v1/responses`
Responses,
}
impl WireSurface {
/// The value recorded as `wire=` on every span, matching the names
/// cortex already logs via `log_inbound` so the two journals join.
pub fn as_str(self) -> &'static str {
match self {
Self::Chat => "openai-chat",
Self::Responses => "openai-responses",
}
}
}
pub struct LoadedModel {
pub model_id: String,
/// What the model published in `generation_config.json` (#272).
@@ -2879,8 +2907,9 @@ impl CandleHarness {
&self,
request: ChatCompletionRequest,
principal: Option<String>,
wire: WireSurface,
) -> Result<ChatCompletionResponse, InferenceError> {
let stream = self.inference_stream(request, principal).await?;
let stream = self.inference_stream(request, principal, wire).await?;
wire_chat::collect_chat_completion(
stream.events,
stream.id,
@@ -2907,11 +2936,13 @@ impl CandleHarness {
&self,
request: ChatCompletionRequest,
principal: Option<String>,
wire: WireSurface,
) -> Result<mpsc::Receiver<ChatCompletionChunk>, InferenceError> {
self.chat_completion_stream_with(
request,
wire_chat::ChatProjectionConfig::default(),
principal,
wire,
)
.await
}
@@ -2925,8 +2956,9 @@ impl CandleHarness {
request: ChatCompletionRequest,
mut config: wire_chat::ChatProjectionConfig,
principal: Option<String>,
wire: WireSurface,
) -> Result<mpsc::Receiver<ChatCompletionChunk>, InferenceError> {
let stream = self.inference_stream(request, principal).await?;
let stream = self.inference_stream(request, principal, wire).await?;
// Fill in the model's reasoning markers if the caller
// didn't pre-populate them — they're a property of the
// loaded model (which the HTTP handler doesn't reach into
@@ -2954,9 +2986,10 @@ impl CandleHarness {
response_id: String,
message_item_id: String,
principal: Option<String>,
wire: WireSurface,
) -> Result<mpsc::Receiver<crate::wire::openai_responses::ResponseStreamFrame>, InferenceError>
{
let stream = self.inference_stream(request, principal).await?;
let stream = self.inference_stream(request, principal, wire).await?;
let meta = crate::wire::openai_responses::ResponseMeta {
response_id,
created_at: stream.created,
@@ -2978,6 +3011,7 @@ impl CandleHarness {
&self,
request: ChatCompletionRequest,
principal: Option<String>,
wire: WireSurface,
) -> Result<InferenceStream, InferenceError> {
let handle = {
let models = self.models.read().await;
@@ -3002,7 +3036,7 @@ impl CandleHarness {
LoadedHandle::Single(m) => m,
#[cfg(feature = "cuda")]
LoadedHandle::Tp(m) => {
return self.inference_tp_stream(m, request, principal).await;
return self.inference_tp_stream(m, request, principal, wire).await;
}
LoadedHandle::Image(_) => {
return Err(InferenceError::WrongModality {
@@ -3140,7 +3174,12 @@ impl CandleHarness {
// executor so we capture the span explicitly and re-enter it
// inside the closure to keep the req_id on every emitted line.
let req_id = new_req_id();
let span = tracing::info_span!("chat_stream", req_id = %req_id, model = %model_id);
let span = tracing::info_span!(
"chat_stream",
req_id = %req_id,
model = %model_id,
wire = wire.as_str()
);
let prompt_len = prompt_tokens.len();
let req_start = std::time::Instant::now();
// Cloned `Arc<LoadedModel>` so the spawned task can mark the
@@ -4714,6 +4753,7 @@ impl CandleHarness {
tp: Arc<TpLoadedModel>,
request: ChatCompletionRequest,
principal: Option<String>,
wire: WireSurface,
) -> Result<InferenceStream, InferenceError> {
if tp.poisoned.load(Ordering::Acquire) {
return Err(self.trigger_recovery(&request.model).await);
@@ -4865,14 +4905,19 @@ impl CandleHarness {
// of this inference; concurrent requests against the same TP
// model serialise behind it.
//
// Tagged with the same req_id span as the non-streaming path
// so the journal can be reconstructed regardless of which API
// surface the client hit.
// Tagged with the same req_id span as the non-streaming path,
// and with the surface the request actually arrived on. Both
// matter: neuron translates `/v1/responses` into a chat request
// at the door, so without `wire` every line here reads as
// though the caller used `/v1/chat/completions`. The value
// matches what cortex logs via `log_inbound`, so the two
// journals join on it.
let req_id = new_req_id();
let span = tracing::info_span!(
"tp_chat_stream",
req_id = %req_id,
model = %model_id
model = %model_id,
wire = wire.as_str()
);
let req_start = std::time::Instant::now();
let (vram_free_mb, vram_total_mb) = tp.query_vram().await;
@@ -5708,8 +5753,33 @@ fn parse_qwen_xml_tool_call(
schemas: &ToolSchemas,
) -> Option<(String, String, String)> {
// `<function=NAME>` — name runs to the closing `>` or end of line.
let after_fn = body.split("<function=").nth(1)?;
let name = after_fn.split(['>', '\n']).next()?.trim().to_string();
//
// Falls back to a recovery for a malformation seen in production
// (2026-08-29): the model tagged the function itself as a parameter,
// emitting `<tool_call><parameter=web_search><parameter=query>…`
// with no `<function=` at all. Strict parsing returned None, so the
// whole block fell through into `content` and the user's answer was
// a screenful of raw markup.
//
// The recovery is deliberately narrow: a leading `<parameter=X>` is
// only promoted when X is a tool this request actually offered. That
// is what keeps it from inventing function names out of ordinary
// parameters — an unknown name still returns None, and the caller
// still owes the user something better than markup.
let (name, after_fn, skip_param) = match body.split_once("<function=") {
Some((_, after)) => {
let n = after.split(['>', '\n']).next()?.trim().to_string();
(n, after, None)
}
None => {
let after = body.split_once("<parameter=").map(|(_, a)| a)?;
let n = after.split(['>', '\n']).next()?.trim().to_string();
if !schemas.contains_key(&n) {
return None;
}
(n.clone(), after, Some(n))
}
};
if name.is_empty() {
return None;
}
@@ -5732,7 +5802,8 @@ fn parse_qwen_xml_tool_call(
.to_string();
let after_gt = seg.split_once('>').map(|(_, after)| after).unwrap_or("");
let value_raw = after_gt.split("</parameter>").next().unwrap_or("").trim();
if !key.is_empty() {
// Skip the pseudo-parameter that was actually the function name.
if !key.is_empty() && Some(&key) != skip_param.as_ref() {
let declared = param_types.and_then(|m| m.get(&key)).map(String::as_str);
args.insert(key, coerce_param_value(value_raw, declared));
}
@@ -7573,6 +7644,26 @@ mod tests {
assert_eq!(effort_in(&out), None);
}
/// The wire surface must serialise as the string cortex logs, so a
/// journal search joins the two.
///
/// neuron translates `/v1/responses` into a chat request at the
/// door, and every line after that is emitted by the shared chat
/// path. Before `wire` was recorded the journal showed
/// `chat_completion (stream): done` for every request whatever the
/// surface — worse than omitting the detail, because it asserts
/// something false about the caller. It cost a real investigation
/// into why a Responses-configured client "was using chat
/// completions". It was not.
#[test]
fn the_wire_surface_matches_what_cortex_logs() {
// These strings are load-bearing: `cortex-gateway`'s
// `log_inbound` emits the same two, so `journalctl | grep
// openai-responses` spans both services.
assert_eq!(WireSurface::Chat.as_str(), "openai-chat");
assert_eq!(WireSurface::Responses.as_str(), "openai-responses");
}
/// Effort must not select a *rung's* budget — that was the
/// guillotine (#290): the model was told to reason at `xhigh` and
/// cut off at `medium`'s 12,288. The reserve below is not that; it
@@ -8737,6 +8828,58 @@ mod tests {
assert_eq!(v["s"], "hello"); // un-JSON-parseable → string
}
/// The shape Qwen3.8-27B emitted on helexa.ai on 2026-08-29: it
/// tagged the function itself as a parameter, so there is no
/// `<function=` anywhere. Strict parsing returned None and the whole
/// block was re-emitted as text — the user's answer to "what's the
/// best hardware to run X on?" was a screenful of raw markup.
#[test]
fn recovers_a_tool_call_that_tagged_the_function_as_a_parameter() {
let body = concat!(
"<parameter=web_search>",
" <parameter=query> \"Qwen3.8 Flash Next\" hardware GPU </parameter>",
" </function>"
);
let mut schemas = ToolSchemas::new();
let mut params = std::collections::HashMap::new();
params.insert("query".to_string(), "string".to_string());
schemas.insert("web_search".to_string(), params);
let (_, name, args) = parse_tool_call_body(body, 0, &schemas).expect("recovered");
assert_eq!(name, "web_search");
let v: serde_json::Value = serde_json::from_str(&args).unwrap();
assert_eq!(v["query"], "\"Qwen3.8 Flash Next\" hardware GPU");
// The promoted name must not also appear as an argument.
assert!(
v.get("web_search").is_none(),
"the pseudo-parameter carrying the function name is not a parameter"
);
}
/// The recovery only promotes a name the request actually offered.
/// Without that guard any `<parameter=x>` would mint a function call,
/// and a model rambling in angle brackets would start invoking tools.
#[test]
fn does_not_invent_a_function_from_an_unknown_parameter() {
let body = "<parameter=not_a_tool> <parameter=query> hi </parameter>";
let mut schemas = ToolSchemas::new();
schemas.insert("web_search".to_string(), std::collections::HashMap::new());
assert!(parse_tool_call_body(body, 0, &schemas).is_none());
}
/// A well-formed call must not be affected by the recovery path.
#[test]
fn well_formed_call_still_takes_the_function_branch() {
let body = "<function=web_search>\n<parameter=query>\nrust\n</parameter>\n</function>";
let mut schemas = ToolSchemas::new();
let mut params = std::collections::HashMap::new();
params.insert("query".to_string(), "string".to_string());
schemas.insert("web_search".to_string(), params);
let (_, name, args) = parse_tool_call_body(body, 0, &schemas).expect("parsed");
assert_eq!(name, "web_search");
assert_eq!(args, r#"{"query":"rust"}"#);
}
#[test]
fn unparseable_tool_call_body_returns_none() {
// Neither JSON nor a `<function=…>` block — caller re-emits as text.
@@ -36,6 +36,28 @@ use crate::config::ContextLimitConfig;
/// few turns.
const RATE_EMA_ALPHA: f64 = 0.3;
/// Smallest prompt whose prefill time says anything about throughput.
///
/// A prefill has a fixed per-request cost — tokenize, launch, sync —
/// that dominates a short prompt completely. The deploy's own smoke
/// probe prefills 19 tokens in ~130 ms and reads as **145 tok/s** on a
/// host that measures ~1,400. Folding that in does not make the
/// estimate slightly worse; it replaces it.
///
/// That mattered because the fallback it displaces is the whole
/// cold-start guard: with no sample at all, `derive_limit` uses
/// `bootstrap_prefill_tok_per_sec` (800, ≈96k context at the default
/// 120 s target). One 19-token probe replaced it with 145 tok/s, and
/// the host advertised **17,408** — a sixth of what it can serve —
/// until real traffic happened to warm it (#295).
///
/// 512 excludes the smoke probe and the short auxiliary calls agent
/// harnesses make (observed at 19, 143 and 469 tokens) while admitting
/// ordinary prompts. At ~1,400 tok/s a 512-token prefill runs ~370 ms,
/// so fixed overhead is roughly a tenth of it — informative rather than
/// dominant.
const MIN_PREFILL_SAMPLE_TOKENS: usize = 512;
/// Fold one throughput sample (`tokens` in `elapsed`) into the EMA held in
/// `bits`. No-op for degenerate inputs so a probe request or a clock blip
/// can't poison the average.
@@ -99,7 +121,15 @@ impl ThroughputEma {
/// Fold one prefill measurement (`prompt_tokens` processed in
/// `elapsed`) into the prefill EMA.
///
/// Samples below [`MIN_PREFILL_SAMPLE_TOKENS`] are ignored: they
/// measure fixed per-request cost rather than throughput, and the
/// value they displace — the bootstrap estimate — is better than
/// they are.
pub fn record(&self, prompt_tokens: usize, elapsed: Duration) {
if prompt_tokens < MIN_PREFILL_SAMPLE_TOKENS {
return;
}
fold_rate(&self.prefill_bits, prompt_tokens, elapsed);
}
@@ -356,6 +386,60 @@ pub fn derive_limit(
#[cfg(test)]
mod tests {
/// The deploy's smoke probe must not set the advertised context.
///
/// 19 tokens in 130 ms reads as ~145 tok/s on a host that measures
/// ~1,400. Before this guard that one sample displaced the
/// bootstrap estimate and the host advertised 17,408 instead of
/// ~96,000 — until real traffic happened along (#295).
#[test]
fn a_smoke_probe_does_not_poison_the_prefill_rate() {
let ema = ThroughputEma::new();
ema.record(19, Duration::from_millis(130));
assert_eq!(
ema.get(),
None,
"a 19-token prefill must leave the EMA unset so the bootstrap value stands"
);
}
/// The short auxiliary calls agent harnesses make are the same
/// shape as the probe and equally uninformative — observed at 143
/// and 469 tokens against prompts three orders of magnitude larger.
#[test]
fn short_auxiliary_prompts_are_ignored_too() {
let ema = ThroughputEma::new();
for tokens in [143usize, 469] {
ema.record(tokens, Duration::from_millis(200));
}
assert_eq!(ema.get(), None);
}
/// A real prompt still records, and still tracks a shift.
#[test]
fn representative_prefills_still_measure() {
let ema = ThroughputEma::new();
ema.record(1400, Duration::from_secs(1));
let first = ema.get().expect("a 1400-token sample must record");
assert!((first - 1400.0).abs() < 1.0, "got {first}");
// A slower host is tracked, not ignored.
for _ in 0..20 {
ema.record(700, Duration::from_secs(1));
}
let after = ema.get().unwrap();
assert!(after < first && after > 600.0, "got {after}");
}
/// Exactly at the threshold counts — the boundary is inclusive so
/// the constant means what it says.
#[test]
fn the_threshold_itself_is_a_valid_sample() {
let ema = ThroughputEma::new();
ema.record(MIN_PREFILL_SAMPLE_TOKENS, Duration::from_secs(1));
assert!(ema.get().is_some());
}
use super::*;
/// The ceiling a client is told to plan against must not be the
+1 -1
View File
@@ -35,7 +35,7 @@ default_endpoint = "helexa"
name = "helexa"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.6-27B"
default_model = "Qwen/Qwen3.8-27B"
max_tokens = 8192
# Compaction kicks in when the rolling history grows past this token
# budget. Set to your model's context window. Disable by removing
+1 -1
View File
@@ -23,7 +23,7 @@ listen = "0.0.0.0:8088"
#
# [aliases]
# "helexa/small" = "Qwen/Qwen3-8B"
# "helexa/balanced" = "Qwen/Qwen3.6-27B"
# "helexa/balanced" = "Qwen/Qwen3.8-27B"
# -- Downstream cortexes -------------------------------------------------
# Each [[cortexes]] entry is an operator-run cortex the router may dispatch
+37
View File
@@ -0,0 +1,37 @@
# ─────────────────────────────────────────────────────────────────────
# MANAGED BY CI — .gitea/workflows/deploy.yml is the only writer.
#
# Edits made directly to the deployed copy on gallumbits are reverted by
# the next deploy, and until then the fleet runs something no commit
# describes. Change it here, commit, let the pipeline ship it.
# ─────────────────────────────────────────────────────────────────────
# helexa-router — federation data plane on gallumbits.
#
# Tracked despite living beside the secret-bearing service configs,
# because it holds no secrets: a listen address, the product-tier
# aliases, and a mesh endpoint. Being untracked cost an outage on
# 2026-08-27 — `helexa/balanced` still resolved to Qwen/Qwen3.6-27B
# after that model had been retired from the catalogue, so every
# authenticated web-chat turn evicted beast's resident flagship and
# cold-loaded a model nobody had asked for. Nothing could diff the two
# files, so nothing did.
#
# `script/check-alias-targets.py` now fails CI if any alias below names
# a model that models.toml does not define.
[router]
listen = "0.0.0.0:8088"
poll_interval_secs = 10
region = "eu-central"
# Product tiers → real model ids (#166). The SPA sends the tier names.
[aliases]
"helexa/small" = "Qwen/Qwen3-8B"
"helexa/balanced" = "Qwen/Qwen3.8-27B"
# hanzalova operator cortex, plaintext over the WireGuard mesh (the mesh
# is the trust boundary; no TLS pin needed for http://).
[[cortexes]]
name = "hanzalova"
endpoint = "http://hanzalova.internal:31313"
region = "eu-central"
@@ -44,6 +44,70 @@ change.
VRAM fragmentation has had time to accumulate. It is a signal to restart
that neuron, not an automatic action.
## secrets.toml
Credentials do not belong in the file CI deploys. Put them in a
`secrets.toml` beside the main config and the service will merge it
last, so its values win:
```
/etc/cortex/cortex.toml structure — deployed by CI, safe to diff
/etc/cortex/secrets.toml credentials — yours, root-only, never in git
```
```toml
# /etc/cortex/secrets.toml (root:cortex 0640)
[[entitlements.keys]]
key = "sk-…"
account_id = "acct-a"
key_id = "a"
[upstream]
bearer = "…"
```
`helexa-upstream` and `helexa-angels` read a `secrets.toml` beside their
own configs the same way — the DB URL, JWT secret and SMTP URL belong
there.
**An absent `secrets.toml` is not an error.** A single-file deployment
keeps working exactly as before, so this is opt-in per host and can be
adopted one service at a time.
### Why bother
Because a config CI cannot write is a config no check can defend, and
the cost of that is not hypothetical. `helexa-router.toml` was kept out
of git for sitting beside secret-bearing files. Nothing could diff it
against the catalogue, so its `helexa/balanced` alias went on pointing
at a model that had been retired — and since both models were
deliberately given equal residency priority so they could displace each
other, every authenticated chat turn evicted the resident flagship to
cold-load the retired one. Four of those overlapped and the node ended
up holding 32 GB of GPU memory with no model loaded and nothing
serving.
Splitting secrets from structure is what lets the structure be
version-controlled, reviewed and checked. The ordering matters too: the
operator's file merges last, so a deploy can never replace a live
credential with a placeholder.
### Moving existing secrets
One-time, per host, and reversible — the service reads both files, so
you can move a value and restart before moving the next:
1. Create `secrets.toml` beside the config, owned `root:<service>` mode
`0640` (`0600` where the service runs as root).
2. Move the credential blocks across. Leave everything else in place.
3. Restart the service and confirm it still authenticates — for cortex,
a request with a real key should be billed rather than treated as
anonymous.
4. Delete the moved lines from the main file only once step 3 passes.
Until step 4 the value exists in both files, which is safe: the
operator's copy wins.
## models.toml
The catalogue says what each model needs and where it may run:
@@ -78,7 +142,7 @@ Clients should not hardcode model ids. Aliases let them ask for a role:
[aliases]
"helexa/small" = "Qwen/Qwen3-1.7B"
"helexa/balanced" = "Qwen/Qwen3-8B"
"helexa/large" = "Qwen/Qwen3.6-27B"
"helexa/large" = "Qwen/Qwen3.8-27B"
"helexa/image" = "Tongyi-MAI/Z-Image-Turbo"
```
@@ -58,6 +58,32 @@ Two things that bite:
- **Cold loads are disk-bound.** A 50 GB model off spinning rust is a
very different experience from the same model on NVMe.
### Where weights come from
By default a bare `org/name` id is fetched from HuggingFace. Operators
running a mirror, or serving models from more than one place, can
declare sources explicitly:
```toml
[harness.candle]
default_source = "huggingface"
[harness.candle.sources.huggingface]
endpoint = "https://huggingface.co"
auth_env = "HF_TOKEN" # env var name, never the token itself
cache_dir = "/archive/llm" # optional; per-source cache
```
`auth_env` names an environment variable rather than holding a token, so
credentials stay out of the config file — worth keeping to even when the
file is not obviously shared, because config is exactly what gets copied
between hosts.
Give a source its own `cache_dir` when two sources can serve the same
`org/name`. The cache tree is keyed on the model id alone, so sharing a
directory between sources that disagree about what `org/name` means will
collide on disk.
## Default models
Models listed as defaults load when the daemon activates, so the host
@@ -66,7 +92,7 @@ load cost:
```toml
[[default_models]]
model_id = "Qwen/Qwen3.6-27B"
model_id = "Qwen/Qwen3.8-27B"
harness = "candle"
quant = "q6k"
tensor_parallel = 2
@@ -91,10 +117,17 @@ Each loaded model has bounded admission control:
[harness.candle.admission]
max_in_flight = 8 # concurrent requests actually running
max_queue_depth = 8 # waiting before rejection
max_wait_secs = 30 # how long a queued request waits
max_wait_secs = 30 # how long a queued request waits for a slot
kv_max_wait_secs = 300 # how long it waits for enough KV budget
max_per_principal = 2 # fair share, so one caller cannot monopolise
```
`kv_max_wait_secs` is deliberately much longer than `max_wait_secs`.
The first waits for a slot to turn over; the second waits for a whole
long-context sequence to finish, which on a 27B can be minutes. A
request that needs KV budget nobody has yet is not a request to reject
in thirty seconds.
Over-capacity requests are rejected promptly with `429`/`503` and a
`Retry-After` rather than being allowed to queue invisibly. A fast
rejection a client can retry is far better than a request that hangs
@@ -102,8 +135,34 @@ until something times out.
`max_in_flight` above 1 enables batched decode on architectures that
support it, so concurrent requests share a decode step instead of
serialising. Raise it on big cards, leave it at 1 on small ones — the
KV cache for several concurrent long contexts is itself substantial.
serialising.
**`max_in_flight` does not partition the KV budget.** KV is a shared
pool, and each request reserves what its own prompt actually needs when
it is admitted — which is why `kv_max_wait_secs` above exists at all. A
request larger than what is currently free waits for bytes to come back
rather than being handed a fixed 1/N slice, and a single long session
can legitimately hold most of the pool.
So the slot count and the context window are separate questions. What
bounds a busy moment is the pool: divide `kv_budget_mb` by the model's
KV cost per token (`GET /health` reports both) to see how many tokens
are available to share. Many ordinary chat turns fit at once; a few very
long agentic sessions will queue on bytes.
Size `max_in_flight` against the concurrency you actually want to serve
concurrently, not against context. Anything above it queues on admission
with a `Retry-After` rather than being refused, so setting it too low
converts capacity you have into latency your callers feel. Setting it
very high mainly costs batch-step efficiency, since the engine pads to
the widest active sequence.
Measure before assuming you need a high number — but measure the traffic
you will have, not the traffic you had. On one production host, seven
days of spans showed 89.6% of busy time at a single request in flight;
that host's chat tier was at the time being routed to a *different*
model, and the figure stopped describing it the moment that was
corrected.
`max_per_principal` is what stops one busy client consuming the whole
model.
@@ -150,7 +209,8 @@ reservation is doing its job, not that the model is overloaded.
neuron refuses work it cannot finish rather than crashing partway:
- a **static floor** of free VRAM below which prefill is refused
(`NEURON_MIN_FREE_VRAM_MB`, default 1500)
(`NEURON_MIN_FREE_VRAM_MB`, or `min_free_floor_mb` under
`[harness.candle.context_limit]`; default 1500)
- a **length-aware check** that estimates the KV cache the request will
actually need and refuses if it will not fit
@@ -159,15 +219,207 @@ can pass the static floor and still be impossible, because KV grows with
prompt *and* generation length. See [context
limits](/docs/operating/context-limits) for how the numbers relate.
`activation_headroom_mb` reserves VRAM per card for prefill activations
on top of the weights and the KV cache, before the context ceiling is
computed. Raise it if prefill OOMs on a host whose steady-state
occupancy looks fine — activations are transient and do not show up in
the resident figure.
A host that cannot currently serve a model reports that on `/health`,
and cortex stops routing to it — an unservable replica is not a routing
target.
## Image models
```toml
[harness.candle.image]
te_device = "cpu" # where the text encoder runs
te_resident = true # keep it loaded between generations
max_dim = 2048 # resolution ceiling per side
```
The text encoder is the placement decision. On CPU the ~8 GB encoder
never touches the GPU — prompt encoding is one short forward and the
features are tiny — which is what lets a 24 GB card serve a BF16 DiT at
all. Move it to `cuda` only on a card with room to spare.
`te_resident` trades memory for latency: resident costs ~16 GB of system
RAM on CPU (or ~8 GB of VRAM on CUDA) and saves a few seconds per
request. CPU defaults to resident, CUDA to rebuilding per request, which
are the right defaults for where each is scarce.
`max_dim` rejects oversized requests **before admission**, so an
impossible resolution costs nothing. That matters more than it sounds:
an OOM on the device poisons the worker context, so refusing early is
the difference between one rejected request and a model that needs
restarting.
## Prompt caps
`NEURON_MAX_PROMPT_TOKENS` sets a hard ceiling; larger prompts are
rejected before any GPU work, so a runaway client costs nothing.
## Sampling
Every model publishes its own sampling defaults in
`generation_config.json`, and a caller can override them per request.
Between those two sits an operator override, for the cases where the
model's published defaults are wrong for how you serve it:
```toml
[[default_models]]
model_id = "Qwen/Qwen3.8-27B"
[default_models.sampling]
temperature = 0.6
presence_penalty = 1.0
```
Precedence is **request > operator > model > built-in**. Anything you
leave unset changes nothing, so an empty table behaves exactly as no
table at all.
Set the same values in the gateway's `models.toml` under
`[models.sampling]`. A model that is cold-loaded on demand gets its
sampling from the catalogue, not from this file, so the two must agree
or the same model samples differently depending on how it arrived —
`script/check-config-consistency.py` fails the deploy if they diverge.
### Which knob for which symptom
| symptom | knob |
|---|---|
| Structurally malformed output — truncated JSON, unbalanced braces | `temperature` |
| The think block restates the same passage over and over | `presence_penalty` |
| Common tokens over-represented across a long answer | `frequency_penalty` |
| Short immediate loops — "no, no, no…" | `repeat_penalty` |
| You need two runs to be comparable | `seed` |
**`temperature`** is the blunt one. Qwen3.8-27B publishes `1.0`, which
measured a 20% structural-defect rate on long agentic turns against 0/60
at `0.6`; the fleet therefore overrides it. Note that `0.6` is also what
Qwen recommend for their thinking variants, so this is not fighting the
model.
**`presence_penalty`** and **`frequency_penalty`** are scored over the
*whole* generated sequence. They are the ones that matter for reasoning
models, and Qwen name `presence_penalty` (0–2, 1.5 when severe) as the
remedy for the endless repetition their models fall into during long
thinking. Both default to `0.0`, which is off.
**`repeat_penalty`** (default `1.1`) and **`repeat_last_n`** (default
`64`) are the older, *windowed* form: the penalty only sees the last
`repeat_last_n` tokens. That is fine for chat and close to useless for a
think block — a 20,000-token deliberation restating a passage 5,000
tokens later is entirely outside a 64-token window. Measured across
recorded sessions, **86% of think blocks over 60k characters contained
verbatim repeated sentences**, one of them repeating 76% of its own
content.
Widening the window is the obvious move and the wrong one: the penalty
is token-level, and source code legitimately repeats tokens constantly
(`const`, `function`, brace runs), so a wide window at a meaningful
penalty degrades exactly the output you care about. Reach for
`presence_penalty` first — what recurs is passages, not tokens.
**`seed`** pins the sampler's RNG for every request to this model. It
exists because parameter search needs a control and the clients that
would otherwise pin it often cannot. Leaving it set is a product
decision, not just a testing one: every identical prompt then returns an
identical answer to every caller, which is predictable and cacheable but
removes the variation a user may expect from retrying.
## Reasoning
```toml
[default_models.sampling]
# nothing here — reasoning settings live in their own places, below
```
**`preserve_thinking`** controls whether prior turns keep their think
blocks when the conversation is re-rendered. It is the model's own chat
template control, not ours:
```toml
[[default_models]]
preserve_thinking = false # keep reasoning only for the turn in progress
```
Unset is the default and means the template decides. On Qwen3.8-27B that
is `true` — every prior turn's reasoning is replayed. Full replay is
what makes prompt growth track *total* completion rather than visible
output: one measured session produced 28,672 reasoning tokens on turn 1
and carried all of them into turn 2's 31,269-token prompt, at ~22 s of
prefill on every turn thereafter.
Setting it to `false` is a legitimate experiment, not a recommendation.
Whether narrowing replay *helps* the model is unmeasured. Set it in both
`models.toml` and this file, like sampling, and the value is reported on
`/v1/models` and logged at load so a session can be attributed to the
arm it ran under.
**`reasoning_budget.answer_reserve_tokens`** (default `4096`) holds back
part of the caller's output budget for the answer:
```toml
[harness.candle.reasoning_budget]
answer_reserve_tokens = 4096
```
A reasoning model with no stopping criterion fills whatever budget it is
given. Measured on Qwen3.8-27B: given 16,384 tokens it used 16,384;
given 32,768 it used 32,768 — and both times returned no answer at all.
The reserve bounds the think block at `max_tokens - answer_reserve` so
the rest survives for the reply.
Deliberately *not* derived from reasoning effort. Effort is a prompt
instruction the model is trained to follow; making it also set a
guillotine is how a model came to be told to reason at `xhigh` and then
cut off at `medium`'s budget. Set `0` to disable, restoring unbounded
reasoning.
The trade to know: when the reserve fires, the model is interrupted
mid-deliberation and its answer phase tends to state an *intention*
rather than a conclusion. That is better than returning nothing, but it
is not free.
## The prefix cache
Snapshots of previously-served prefixes, so a follow-up turn re-uses the
KV cache instead of re-prefilling the whole conversation:
```toml
[harness.candle.prefix_cache]
enabled = true
budget_mb = 2048 # snapshot bytes per loaded model
max_entries = 8 # live snapshots per model, regardless of budget
```
**Reach for `budget_mb` when `cached_tokens` stops growing mid-session.**
That is the symptom: the prompt keeps growing, the cached prefix does
not, and every turn re-prefills the difference. Observed on a 1 GiB
default — `cached_tokens` pinned at 2,016 for ten consecutive turns
while the prompt reached 52,869, so ~50k tokens were re-prefilled every turn
at roughly 37 s each, against ~20 s of actual decode.
The number is not free, and the direction of the trade is the opposite
of what it looks like. **The snapshot budget is reserved before the KV
budget, not from what is left over.** On a 2×32 GB host serving a 27B,
measured 1:1 — the token column is the pool all concurrent requests
share, not a per-slot allowance:
| `budget_mb` | KV budget | tokens in the pool | holds a 53k snapshot |
|---|---|---|---|
| 1024 | 4713 MB | 150,800 | no |
| 2048 | 3689 MB | 118,000 | yes |
| 3072 | 2665 MB | 85,300 | yes, but a session cannot reach 53k |
So raising it too far starves the KV cache and shortens how long a
single session can run — the model can no longer reach the context whose
snapshot the larger cache was meant to hold. Size it against the deepest
session you actually serve, and check `kv_budget_mb` in the journal
after a reload.
## Checking on it
```sh
@@ -24,7 +24,7 @@ confined anywhere.
```toml
[[models]]
id = "Qwen/Qwen3.6-27B"
id = "Qwen/Qwen3.8-27B"
harness = "candle"
min_devices = 2
min_device_vram_mb = 24000
@@ -32,6 +32,43 @@ daemon now does automatically.
| `limit.input` | same | compaction trigger — opencode compacts to keep the prompt at/under this |
| `limit.output` | same | generation reserve opencode leaves below the wall |
### Tuning the derived limit
Since #67 the neuron computes `limit` itself (see below), and three
`[harness.candle.context_limit]` settings shape that computation:
```toml
[harness.candle.context_limit]
target_prefill_latency_secs = 120 # longest acceptable prefill per turn
bootstrap_prefill_tok_per_sec = 800 # cold-start estimate until measured
output_reserve_tokens = 8192 # what a request gets when it names no cap
max_output_tokens = 32768 # the largest output a request may name
```
**`target_prefill_latency_secs`** is the one that usually binds. The
throughput ceiling is `target_prefill_latency_secs × measured
prefill tok/s`, so on a host measuring ~1,400 tok/s a 120-second target
advertises about 168k — above most hard ceilings, in which case it stops
mattering. On a slower host, or one whose recent traffic has been small
prompts, it is what caps the window.
**`bootstrap_prefill_tok_per_sec`** applies until the model has measured
its own rate, and stops the advertised window collapsing on a
freshly-restarted host. Prefill samples below 512 tokens are ignored
when computing that rate: a short prompt is dominated by fixed
per-request cost and says nothing about throughput — the deploy's own
19-token smoke probe reads as ~145 tok/s on a host that measures ~1,400.
So a host that has served only small requests keeps the bootstrap
estimate rather than adopting a misleading one, and no warm-up is
needed after a restart.
**`output_reserve_tokens`** is what a request that names no cap
receives. **`max_output_tokens`** is the ceiling advertised to clients
as `max_output_tokens`, and it is the *ceiling*, not the reserve —
publishing the reserve tells every client that reads the advertisement
to cap itself there, which on a reasoning model can be less than a
single think block.
## How they must relate
For a single model on a single neuron, all of these must hold:
@@ -60,9 +97,9 @@ Notes:
lands, treat the VRAM-safe ceiling in rule 5 as a hard limit you set
`NEURON_MAX_PROMPT_TOKENS` below, not something the daemon enforces.
## VRAM cost of context (Qwen3.6-27B on beast)
## VRAM cost of context (Qwen3.8-27B on beast)
Qwen3.6-27B is a hybrid linear-attention model: of its 64 layers only
Qwen3.8-27B is a hybrid linear-attention model: of its 64 layers only
every 4th is full-attention (`full_attention_interval = 4` → **16**
full-attn layers); the rest are `linear_attention` with constant-size
recurrent state. KV cache grows **only** on the 16 full-attn layers
@@ -131,7 +168,7 @@ value wins over a hand override of the same variable. Use `local.conf`
only for genuinely host-local, transient experiments — and remember a
later deploy will re-assert `model.conf`.
**opencode** — `opencode.json`, `provider.models."Qwen/Qwen3.6-27B".limit`:
**opencode** — `opencode.json`, `provider.models."Qwen/Qwen3.8-27B".limit`:
```json
{ "context": 131072, "input": 122880, "output": 8192 }
+451 -1
View File
@@ -36,7 +36,8 @@
"globals": "^16.5.0",
"typescript": "~5.9.3",
"typescript-eslint": "^8.46.4",
"vite": "^7.2.0"
"vite": "^7.2.0",
"vitest": "^3.2.7"
}
},
"node_modules/@babel/code-frame": {
@@ -1814,6 +1815,17 @@
"@swc/counter": "^0.1.3"
}
},
"node_modules/@types/chai": {
"version": "5.2.3",
"resolved": "https://registry.npmjs.org/@types/chai/-/chai-5.2.3.tgz",
"integrity": "sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/deep-eql": "*",
"assertion-error": "^2.0.1"
}
},
"node_modules/@types/debug": {
"version": "4.1.13",
"resolved": "https://registry.npmjs.org/@types/debug/-/debug-4.1.13.tgz",
@@ -1823,6 +1835,13 @@
"@types/ms": "*"
}
},
"node_modules/@types/deep-eql": {
"version": "4.0.2",
"resolved": "https://registry.npmjs.org/@types/deep-eql/-/deep-eql-4.0.2.tgz",
"integrity": "sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==",
"dev": true,
"license": "MIT"
},
"node_modules/@types/estree": {
"version": "1.0.9",
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz",
@@ -2243,6 +2262,121 @@
"vite": "^4 || ^5 || ^6 || ^7 || ^8"
}
},
"node_modules/@vitest/expect": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-3.2.7.tgz",
"integrity": "sha512-E8eBXaKibuvH2pSZErOjdVb5vF4PbKYcrnluBTYxEk1l/VhhwZg1kZQsdtjq+CsF5CFydf2Rdkz7jDHKSisi3w==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/chai": "^5.2.2",
"@vitest/spy": "3.2.7",
"@vitest/utils": "3.2.7",
"chai": "^5.2.0",
"tinyrainbow": "^2.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/mocker": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-3.2.7.tgz",
"integrity": "sha512-Trr0hYO9CM3Wj6ksWHRhK9IZpIY6wTMO5u/MqXurMxT57sWBaOPEtP3Oq60ihZuh5JsiagKfz95OcxdEP6dBrA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/spy": "3.2.7",
"estree-walker": "^3.0.3",
"magic-string": "^0.30.17"
},
"funding": {
"url": "https://opencollective.com/vitest"
},
"peerDependencies": {
"msw": "^2.4.9",
"vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0"
},
"peerDependenciesMeta": {
"msw": {
"optional": true
},
"vite": {
"optional": true
}
}
},
"node_modules/@vitest/pretty-format": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-3.2.7.tgz",
"integrity": "sha512-KUHlwqVu0sRlhCdyPdQ/wBoTfRahjUky1MubOmYw9fWfIZy1gNoHpuaaQBPAaMaVYdQYHJLurzj8ECCj5OwTqA==",
"dev": true,
"license": "MIT",
"dependencies": {
"tinyrainbow": "^2.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/runner": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-3.2.7.tgz",
"integrity": "sha512-sB9y4ovltoQP+WaUPwmSxO9WIg9Ig694Di5PalVPsYHklAdE027mehpWF2SQSVq+k6sFgaivbTjTJwZLSHbedA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/utils": "3.2.7",
"pathe": "^2.0.3",
"strip-literal": "^3.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/snapshot": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-3.2.7.tgz",
"integrity": "sha512-7C+MwShwtBSI5Buwoyg3s/iY1eHL9PKAf+O1wVh/TdnjXUtkoL/9YQtre90i4MtNXM6edP1wJ2zOBpfCyhIS7g==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "3.2.7",
"magic-string": "^0.30.17",
"pathe": "^2.0.3"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/spy": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-3.2.7.tgz",
"integrity": "sha512-Q2eQGI6d2L/hBtZ0qNuKcAGid68XK6cv1xsoaIma6PaJhHPoqcEJhYpXZ/5myCMqkNgtP6UKuBhbc0nHKnrkuQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"tinyspy": "^4.0.3"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/@vitest/utils": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-3.2.7.tgz",
"integrity": "sha512-x6BDOd7dyo3PFLY3I9/HJ25X/6OurhGXk2/B9gOZNPF7XDVjeBK4k01lQE5uvDpbuheErh91qYuE1E2OEjK3Rw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "3.2.7",
"loupe": "^3.1.4",
"tinyrainbow": "^2.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/acorn": {
"version": "8.17.0",
"resolved": "https://registry.npmjs.org/acorn/-/acorn-8.17.0.tgz",
@@ -2318,6 +2452,16 @@
"node": ">=10"
}
},
"node_modules/assertion-error": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-2.0.1.tgz",
"integrity": "sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=12"
}
},
"node_modules/bail": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/bail/-/bail-2.0.2.tgz",
@@ -2412,6 +2556,16 @@
"node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7"
}
},
"node_modules/cac": {
"version": "6.7.14",
"resolved": "https://registry.npmjs.org/cac/-/cac-6.7.14.tgz",
"integrity": "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=8"
}
},
"node_modules/callsites": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/callsites/-/callsites-3.1.0.tgz",
@@ -2453,6 +2607,23 @@
"url": "https://github.com/sponsors/wooorm"
}
},
"node_modules/chai": {
"version": "5.3.3",
"resolved": "https://registry.npmjs.org/chai/-/chai-5.3.3.tgz",
"integrity": "sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==",
"dev": true,
"license": "MIT",
"dependencies": {
"assertion-error": "^2.0.1",
"check-error": "^2.1.1",
"deep-eql": "^5.0.1",
"loupe": "^3.1.0",
"pathval": "^2.0.0"
},
"engines": {
"node": ">=18"
}
},
"node_modules/chalk": {
"version": "4.1.2",
"resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz",
@@ -2510,6 +2681,16 @@
"url": "https://github.com/sponsors/wooorm"
}
},
"node_modules/check-error": {
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/check-error/-/check-error-2.1.3.tgz",
"integrity": "sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 16"
}
},
"node_modules/classnames": {
"version": "2.5.1",
"resolved": "https://registry.npmjs.org/classnames/-/classnames-2.5.1.tgz",
@@ -2633,6 +2814,16 @@
"url": "https://github.com/sponsors/wooorm"
}
},
"node_modules/deep-eql": {
"version": "5.0.2",
"resolved": "https://registry.npmjs.org/deep-eql/-/deep-eql-5.0.2.tgz",
"integrity": "sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=6"
}
},
"node_modules/deep-is": {
"version": "0.1.4",
"resolved": "https://registry.npmjs.org/deep-is/-/deep-is-0.1.4.tgz",
@@ -2695,6 +2886,13 @@
"dev": true,
"license": "ISC"
},
"node_modules/es-module-lexer": {
"version": "1.7.0",
"resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-1.7.0.tgz",
"integrity": "sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==",
"dev": true,
"license": "MIT"
},
"node_modules/esbuild": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.27.7.tgz",
@@ -2944,6 +3142,16 @@
"url": "https://opencollective.com/unified"
}
},
"node_modules/estree-walker": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-3.0.3.tgz",
"integrity": "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/estree": "^1.0.0"
}
},
"node_modules/esutils": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/esutils/-/esutils-2.0.3.tgz",
@@ -2954,6 +3162,16 @@
"node": ">=0.10.0"
}
},
"node_modules/expect-type": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.4.0.tgz",
"integrity": "sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==",
"dev": true,
"license": "Apache-2.0",
"engines": {
"node": ">=12.0.0"
}
},
"node_modules/extend": {
"version": "3.0.2",
"resolved": "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz",
@@ -3539,6 +3757,13 @@
"loose-envify": "cli.js"
}
},
"node_modules/loupe": {
"version": "3.2.1",
"resolved": "https://registry.npmjs.org/loupe/-/loupe-3.2.1.tgz",
"integrity": "sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==",
"dev": true,
"license": "MIT"
},
"node_modules/lowlight": {
"version": "3.3.0",
"resolved": "https://registry.npmjs.org/lowlight/-/lowlight-3.3.0.tgz",
@@ -3564,6 +3789,16 @@
"yallist": "^3.0.2"
}
},
"node_modules/magic-string": {
"version": "0.30.21",
"resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz",
"integrity": "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@jridgewell/sourcemap-codec": "^1.5.5"
}
},
"node_modules/markdown-table": {
"version": "3.0.4",
"resolved": "https://registry.npmjs.org/markdown-table/-/markdown-table-3.0.4.tgz",
@@ -4591,6 +4826,23 @@
"node": ">=8"
}
},
"node_modules/pathe": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz",
"integrity": "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==",
"dev": true,
"license": "MIT"
},
"node_modules/pathval": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/pathval/-/pathval-2.0.1.tgz",
"integrity": "sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 14.16"
}
},
"node_modules/picocolors": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
@@ -5096,6 +5348,13 @@
"node": ">=8"
}
},
"node_modules/siginfo": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/siginfo/-/siginfo-2.0.0.tgz",
"integrity": "sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==",
"dev": true,
"license": "ISC"
},
"node_modules/source-map-js": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
@@ -5116,6 +5375,20 @@
"url": "https://github.com/sponsors/wooorm"
}
},
"node_modules/stackback": {
"version": "0.0.2",
"resolved": "https://registry.npmjs.org/stackback/-/stackback-0.0.2.tgz",
"integrity": "sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==",
"dev": true,
"license": "MIT"
},
"node_modules/std-env": {
"version": "3.10.0",
"resolved": "https://registry.npmjs.org/std-env/-/std-env-3.10.0.tgz",
"integrity": "sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==",
"dev": true,
"license": "MIT"
},
"node_modules/stringify-entities": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/stringify-entities/-/stringify-entities-4.0.4.tgz",
@@ -5143,6 +5416,26 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/strip-literal": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/strip-literal/-/strip-literal-3.1.0.tgz",
"integrity": "sha512-8r3mkIM/2+PpjHoOtiAW8Rg3jJLHaV7xPwG+YRGrv6FP0wwk/toTpATxWYOW0BKdWwl82VT2tFYi5DlROa0Mxg==",
"dev": true,
"license": "MIT",
"dependencies": {
"js-tokens": "^9.0.1"
},
"funding": {
"url": "https://github.com/sponsors/antfu"
}
},
"node_modules/strip-literal/node_modules/js-tokens": {
"version": "9.0.1",
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-9.0.1.tgz",
"integrity": "sha512-mxa9E9ITFOt0ban3j6L5MpjwegGz6lBQmM1IJkWeBZGcMxto50+eWdjC/52xDbS2vy0k7vIMK0Fe2wfL9OQSpQ==",
"dev": true,
"license": "MIT"
},
"node_modules/style-to-js": {
"version": "1.1.21",
"resolved": "https://registry.npmjs.org/style-to-js/-/style-to-js-1.1.21.tgz",
@@ -5174,6 +5467,20 @@
"node": ">=8"
}
},
"node_modules/tinybench": {
"version": "2.9.0",
"resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz",
"integrity": "sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==",
"dev": true,
"license": "MIT"
},
"node_modules/tinyexec": {
"version": "0.3.2",
"resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-0.3.2.tgz",
"integrity": "sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==",
"dev": true,
"license": "MIT"
},
"node_modules/tinyglobby": {
"version": "0.2.17",
"resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.17.tgz",
@@ -5191,6 +5498,36 @@
"url": "https://github.com/sponsors/SuperchupuDev"
}
},
"node_modules/tinypool": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/tinypool/-/tinypool-1.1.1.tgz",
"integrity": "sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==",
"dev": true,
"license": "MIT",
"engines": {
"node": "^18.0.0 || >=20.0.0"
}
},
"node_modules/tinyrainbow": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-2.0.0.tgz",
"integrity": "sha512-op4nsTR47R6p0vMUUoYl/a+ljLFVtlfaXkLQmqfLR1qHma1h/ysYk4hEXZ880bf2CYgTskvTa/e196Vd5dDQXw==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/tinyspy": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/tinyspy/-/tinyspy-4.0.4.tgz",
"integrity": "sha512-azl+t0z7pw/z958Gy9svOTuzqIk6xq+NSheJzn5MMWtWTFywIacg2wUlzKFGtt3cthx0r2SxMK0yzJOR0IES7Q==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/trim-lines": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/trim-lines/-/trim-lines-3.0.1.tgz",
@@ -5557,6 +5894,102 @@
}
}
},
"node_modules/vite-node": {
"version": "3.2.4",
"resolved": "https://registry.npmjs.org/vite-node/-/vite-node-3.2.4.tgz",
"integrity": "sha512-EbKSKh+bh1E1IFxeO0pg1n4dvoOTt0UDiXMd/qn++r98+jPO1xtJilvXldeuQ8giIB5IkpjCgMleHMNEsGH6pg==",
"dev": true,
"license": "MIT",
"dependencies": {
"cac": "^6.7.14",
"debug": "^4.4.1",
"es-module-lexer": "^1.7.0",
"pathe": "^2.0.3",
"vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0"
},
"bin": {
"vite-node": "vite-node.mjs"
},
"engines": {
"node": "^18.0.0 || ^20.0.0 || >=22.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
}
},
"node_modules/vitest": {
"version": "3.2.7",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-3.2.7.tgz",
"integrity": "sha512-KrxIJ62Fd89gfysR4WotlgZABiz2dqFPgqGzX7s+CwsqLFomRH7777ZcrOD6+WVAh7khPQP41A+BKbpcJFrdEg==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/chai": "^5.2.2",
"@vitest/expect": "3.2.7",
"@vitest/mocker": "3.2.7",
"@vitest/pretty-format": "^3.2.7",
"@vitest/runner": "3.2.7",
"@vitest/snapshot": "3.2.7",
"@vitest/spy": "3.2.7",
"@vitest/utils": "3.2.7",
"chai": "^5.2.0",
"debug": "^4.4.1",
"expect-type": "^1.2.1",
"magic-string": "^0.30.17",
"pathe": "^2.0.3",
"picomatch": "^4.0.2",
"std-env": "^3.9.0",
"tinybench": "^2.9.0",
"tinyexec": "^0.3.2",
"tinyglobby": "^0.2.14",
"tinypool": "^1.1.1",
"tinyrainbow": "^2.0.0",
"vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0",
"vite-node": "3.2.4",
"why-is-node-running": "^2.3.0"
},
"bin": {
"vitest": "vitest.mjs"
},
"engines": {
"node": "^18.0.0 || ^20.0.0 || >=22.0.0"
},
"funding": {
"url": "https://opencollective.com/vitest"
},
"peerDependencies": {
"@edge-runtime/vm": "*",
"@types/debug": "^4.1.12",
"@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0",
"@vitest/browser": "3.2.7",
"@vitest/ui": "3.2.7",
"happy-dom": "*",
"jsdom": "*"
},
"peerDependenciesMeta": {
"@edge-runtime/vm": {
"optional": true
},
"@types/debug": {
"optional": true
},
"@types/node": {
"optional": true
},
"@vitest/browser": {
"optional": true
},
"@vitest/ui": {
"optional": true
},
"happy-dom": {
"optional": true
},
"jsdom": {
"optional": true
}
}
},
"node_modules/void-elements": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/void-elements/-/void-elements-3.1.0.tgz",
@@ -5591,6 +6024,23 @@
"node": ">= 8"
}
},
"node_modules/why-is-node-running": {
"version": "2.3.0",
"resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-2.3.0.tgz",
"integrity": "sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==",
"dev": true,
"license": "MIT",
"dependencies": {
"siginfo": "^2.0.0",
"stackback": "0.0.2"
},
"bin": {
"why-is-node-running": "cli.js"
},
"engines": {
"node": ">=8"
}
},
"node_modules/word-wrap": {
"version": "1.2.5",
"resolved": "https://registry.npmjs.org/word-wrap/-/word-wrap-1.2.5.tgz",
+4 -1
View File
@@ -9,6 +9,8 @@
"preview": "vite preview",
"lint": "eslint .",
"typecheck": "tsc -b",
"test": "vitest run",
"test:watch": "vitest",
"i18n:check": "node ./scripts/check-i18n-keys.mjs",
"i18n:meta": "node ./scripts/check-i18n-metadata.mjs",
"i18n:lang-labels": "node ./scripts/check-i18n-lang-labels.mjs",
@@ -43,6 +45,7 @@
"globals": "^16.5.0",
"typescript": "~5.9.3",
"typescript-eslint": "^8.46.4",
"vite": "^7.2.0"
"vite": "^7.2.0",
"vitest": "^3.2.7"
}
}
+105
View File
@@ -2063,3 +2063,108 @@ html[dir="rtl"] .hx-landing-composer textarea {
display: none;
}
}
/* Assistant working — reasoning and tool calls (#TurnBlocks).
*
* Deliberately quieter than the answer: muted text, no bubble of its
* own, hairline separators. The whole point is that it can be present
* without competing, so it reads as an annotation on the reply rather
* than as more reply. */
.hx-blocks {
display: flex;
flex-direction: column;
gap: 0.25rem;
margin-bottom: 0.5rem;
}
.hx-block {
border-left: 2px solid var(--color-border-subtle);
padding-left: 0.5rem;
}
.hx-block-open {
border-left-color: var(--color-accent-soft);
}
.hx-block-head {
display: flex;
align-items: center;
gap: 0.35rem;
width: 100%;
padding: 0.15rem 0;
background: none;
border: 0;
color: var(--color-text-muted);
font-size: 0.78rem;
text-align: start;
cursor: pointer;
}
.hx-block-head:hover {
color: var(--color-text);
}
.hx-block-caret {
flex: none;
transition: transform 0.12s ease;
}
.hx-block-caret-open {
transform: rotate(90deg);
}
.hx-block-label {
flex: none;
font-weight: 600;
}
.hx-block-running {
flex: none;
opacity: 0.7;
font-style: italic;
}
/* The peek is the first thing to give up space — it is a hint, and a
* wrapped one would cost the header its single-line shape. */
.hx-block-peek {
flex: 1 1 auto;
min-width: 0;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
opacity: 0.6;
}
.hx-block-body {
/* Capped and scrollable: a long think block would otherwise push the
* answer off-screen, which defeats the point of showing it. */
max-height: 16rem;
overflow-y: auto;
margin: 0.15rem 0 0.35rem;
padding-inline-end: 0.4rem;
color: var(--color-text-muted);
font-size: 0.82rem;
line-height: 1.45;
white-space: pre-wrap;
overflow-wrap: anywhere;
}
/* Reasoning toggle in the composer. Sits with the send button rather
* than in a settings panel: it changes what the next message does, so it
* belongs where the message is written. */
.hx-thinking-toggle {
display: inline-flex;
align-items: center;
gap: 0.35rem;
margin: 0;
padding: 0 0.5rem;
color: var(--color-text-muted);
font-size: 0.78rem;
white-space: nowrap;
cursor: pointer;
user-select: none;
}
.hx-thinking-toggle:hover {
color: var(--color-text);
}
.hx-thinking-toggle input {
margin: 0;
cursor: pointer;
}
/* Narrow screens: the composer is already tight, and the label is the
* expendable half — the checkbox still reads as a control. */
@media (max-width: 30rem) {
.hx-thinking-toggle span {
display: none;
}
}
+129
View File
@@ -0,0 +1,129 @@
import { useState } from "react";
import { useTranslation } from "react-i18next";
import { FaChevronRight, FaBrain, FaWrench } from "react-icons/fa6";
import type { TurnBlock } from "../data/db";
import { isOpen } from "../lib/turnBlockState";
/**
* The assistant's working — reasoning and tool calls — shown above its
* answer.
*
* Two things are in tension. All of this should be *available*, because a
* model that reasons for two minutes and then answers has done work worth
* inspecting, and a tool call the user cannot see is a claim they cannot
* check. But none of it is what they asked for, so it must not bury the
* answer.
*
* The resolution is that visibility follows activity:
*
* * While a block is the live one — still streaming, no answer text yet
* — it is open. That span is otherwise dead air; a spinner would say
* "working", this says what it is working *on*.
* * The moment answer text starts, every block collapses to a one-line
* header. The thing the user asked for takes the space.
* * Completed turns load collapsed, so scrollback stays readable.
*
* A block the user opens or closes by hand stays that way. Explicit
* intent outranks the automatic rule, including for the live block —
* someone who closes a noisy reasoning stream should not have it reopen
* itself on the next token.
*/
export function TurnBlocks({
blocks,
streaming,
answerStarted,
}: {
blocks: TurnBlock[];
/** This turn is still being generated. */
streaming: boolean;
/** Answer text has begun arriving, so the working is now history. */
answerStarted: boolean;
}) {
const { t } = useTranslation();
// Only blocks the user has touched appear here; everything else follows
// the automatic rule. Keyed by index, which is stable because blocks
// are only ever appended within a turn.
const [overrides, setOverrides] = useState<Record<number, boolean>>({});
if (blocks.length === 0) return null;
return (
<div className="hx-blocks">
{blocks.map((b, i) => {
const open = isOpen(i, blocks.length, streaming, answerStarted, overrides[i]);
const body = b.kind === "reasoning" ? b.text : blockToolBody(b);
return (
<div key={i} className={`hx-block${open ? " hx-block-open" : ""}`}>
<button
type="button"
className="hx-block-head"
aria-expanded={open}
onClick={() => setOverrides((o) => ({ ...o, [i]: !open }))}
>
<FaChevronRight
className={`hx-block-caret${open ? " hx-block-caret-open" : ""}`}
size={10}
aria-hidden="true"
/>
{b.kind === "reasoning" ? (
<FaBrain size={11} aria-hidden="true" />
) : (
<FaWrench size={11} aria-hidden="true" />
)}
<span className="hx-block-label">
{b.kind === "reasoning"
? t("chat:blockReasoning")
: t("chat:blockTool", { name: b.name })}
</span>
{/* A pending tool call is the one case where the header
* alone should say something is happening: its body is
* empty until the result lands, so an open block would
* otherwise look like a stall. */}
{b.kind === "tool" && b.result === undefined && streaming && (
<span className="hx-block-running">{t("chat:blockRunning")}</span>
)}
{!open && (
<span className="hx-block-peek">{peek(body)}</span>
)}
</button>
{open && <div className="hx-block-body">{body}</div>}
</div>
);
})}
</div>
);
}
/** Arguments, then the result once it arrives. */
function blockToolBody(b: Extract<TurnBlock, { kind: "tool" }>): string {
const args = prettyArgs(b.args);
return b.result === undefined ? args : `${args}\n\n${b.result}`;
}
/**
* Tool arguments as sent, pretty-printed when they are JSON.
*
* Models emit malformed arguments often enough that this must never
* throw — a UI that blanks because the model produced bad JSON hides the
* very thing worth seeing.
*/
function prettyArgs(raw: string): string {
try {
return JSON.stringify(JSON.parse(raw), null, 2);
} catch {
return raw;
}
}
/**
* The tail of a collapsed block, so a closed row still carries a hint of
* its content.
*
* The tail rather than the head: while a block streams, the end is where
* the model currently is. A head-anchored preview freezes on the first
* few words and stops telling you anything.
*/
function peek(text: string): string {
const flat = text.replace(/\s+/g, " ").trim();
return flat.length <= 80 ? flat : `…${flat.slice(-79)}`;
}
+27
View File
@@ -39,6 +39,23 @@ export interface MessageSource {
url: string;
}
/**
* Work the assistant did on the way to an answer: a stretch of reasoning,
* or one tool call and its result.
*
* Kept as an ordered list rather than two separate fields because the
* order is the story — a turn can reason, call a tool, reason about what
* came back, and call another. Flattening that into "the reasoning" and
* "the tool calls" would lose which thought preceded which call.
*
* `text` and `result` grow while streaming, so a block is also its own
* progress indicator: a reasoning block with text and no following block
* is what the model is doing *right now*.
*/
export type TurnBlock =
| { kind: "reasoning"; text: string }
| { kind: "tool"; name: string; args: string; result?: string };
export interface Message {
id: string;
conversationId: string;
@@ -50,6 +67,16 @@ export interface Message {
promptTokens?: number;
completionTokens?: number;
sources?: MessageSource[];
/**
* Reasoning and tool calls, in the order they happened. Absent on user
* turns and on assistant turns that answered directly.
*
* Not indexed, so no schema version bump: Dexie stores the whole
* object and only the declared indexes constrain it. Messages written
* before this field existed simply have none, which renders as an
* answer with no working shown — correct, since none was captured.
*/
blocks?: TurnBlock[];
}
/**
+14 -1
View File
@@ -10,6 +10,7 @@ import {
type Message,
type MessageRole,
type Project,
type TurnBlock,
} from "./db";
function uuid(): string {
@@ -138,10 +139,22 @@ export async function setMessageContent(id: string, content: string): Promise<vo
await db.messages.update(id, { content });
}
/**
* Write the whole block list as an absolute snapshot.
*
* Same discipline as `setMessageContent`: the caller owns the array and
* writes it entire, because a read-modify-write append races itself when
* deltas arrive faster than the IndexedDB round-trip — which is how the
* content path dropped tokens before it was written this way.
*/
export async function setMessageBlocks(id: string, blocks: TurnBlock[]): Promise<void> {
await db.messages.update(id, { blocks });
}
export async function finalizeMessage(
id: string,
patch: Partial<
Pick<Message, "status" | "errorCode" | "promptTokens" | "completionTokens" | "sources">
Pick<Message, "status" | "errorCode" | "promptTokens" | "completionTokens" | "sources" | "blocks">
>,
): Promise<void> {
await db.messages.update(id, patch);
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "{{host}} እየተነበበ ነው…",
"copyCode": "ኮድ ቅዳ",
"copiedCode": "ተቀድቷል",
"anonNearLimit": "ወደ የስም-አልባ ተጠቃሚ ገደብ እየተቃረቡ ነው። ነፃ ድልድል ለማግኘት ይመዝገቡ።"
"anonNearLimit": "ወደ የስም-አልባ ተጠቃሚ ገደብ እየተቃረቡ ነው። ነፃ ድልድል ለማግኘት ይመዝገቡ።",
"blockReasoning": "በማሰብ ላይ",
"blockTool": "{{name}}",
"blockRunning": "በማስኬድ ላይ…",
"showThinking": "ማመዛዘን",
"showThinkingHint": "ሞዴሉ ከመመለሱ በፊት እንዲያስብ ጠይቅ፣ እና ማመዛዘኑን አሳይ"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "تجري قراءة {{host}}…",
"copyCode": "نسخ الكود",
"copiedCode": "تم النسخ",
"anonNearLimit": "أوشكت على بلوغ حد الاستخدام المجهول. أنشئ حسابًا للحصول على مخصّص مجاني."
"anonNearLimit": "أوشكت على بلوغ حد الاستخدام المجهول. أنشئ حسابًا للحصول على مخصّص مجاني.",
"blockReasoning": "جارٍ التفكير",
"blockTool": "{{name}}",
"blockRunning": "قيد التشغيل…",
"showThinking": "التفكير",
"showThinkingHint": "اطلب من النموذج التفكير قبل الإجابة، واعرض تفكيره"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "четене на {{host}}…",
"copyCode": "Копиране на кода",
"copiedCode": "Копирано",
"anonNearLimit": "Наближавате анонимния лимит. Регистрирайте се за безплатна квота."
"anonNearLimit": "Наближавате анонимния лимит. Регистрирайте се за безплатна квота.",
"blockReasoning": "Разсъждаване",
"blockTool": "{{name}}",
"blockRunning": "изпълнява се…",
"showThinking": "Разсъждение",
"showThinkingHint": "Моделът да мисли преди да отговори и да показва разсъжденията си"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "s'està llegint {{host}}…",
"copyCode": "Copia el codi",
"copiedCode": "Copiat",
"anonNearLimit": "Us acosteu al límit anònim. Registreu-vos per obtenir una assignació gratuïta."
"anonNearLimit": "Us acosteu al límit anònim. Registreu-vos per obtenir una assignació gratuïta.",
"blockReasoning": "Pensant",
"blockTool": "{{name}}",
"blockRunning": "en execució…",
"showThinking": "Raonament",
"showThinkingHint": "Demana al model que pensi abans de respondre i mostra'n el raonament"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "čtení {{host}}…",
"copyCode": "Kopírovat kód",
"copiedCode": "Zkopírováno",
"anonNearLimit": "Blížíte se k anonymnímu limitu. Zaregistrujte se a získejte bezplatný příděl."
"anonNearLimit": "Blížíte se k anonymnímu limitu. Zaregistrujte se a získejte bezplatný příděl.",
"blockReasoning": "Přemýšlení",
"blockTool": "{{name}}",
"blockRunning": "probíhá…",
"showThinking": "Uvažování",
"showThinkingHint": "Ať model před odpovědí přemýšlí a ukáže své uvažování"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "læser {{host}}…",
"copyCode": "Kopiér kode",
"copiedCode": "Kopieret",
"anonNearLimit": "Du nærmer dig den anonyme grænse. Opret en konto og få en gratis tildeling."
"anonNearLimit": "Du nærmer dig den anonyme grænse. Opret en konto og få en gratis tildeling.",
"blockReasoning": "Tænker",
"blockTool": "{{name}}",
"blockRunning": "kører…",
"showThinking": "Ræsonnement",
"showThinkingHint": "Bed modellen tænke før den svarer, og vis dens ræsonnement"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "{{host}} wird gelesen…",
"copyCode": "Code kopieren",
"copiedCode": "Kopiert",
"anonNearLimit": "Sie nähern sich dem anonymen Limit. Registrieren Sie sich für ein kostenloses Kontingent."
"anonNearLimit": "Sie nähern sich dem anonymen Limit. Registrieren Sie sich für ein kostenloses Kontingent.",
"blockReasoning": "Denkt nach",
"blockTool": "{{name}}",
"blockRunning": "läuft…",
"showThinking": "Denkprozess",
"showThinkingHint": "Das Modell vor der Antwort nachdenken lassen und den Denkprozess anzeigen"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "ανάγνωση του {{host}}…",
"copyCode": "Αντιγραφή κώδικα",
"copiedCode": "Αντιγράφηκε",
"anonNearLimit": "Πλησιάζετε το όριο ανώνυμης χρήσης. Εγγραφείτε για μια δωρεάν κατανομή."
"anonNearLimit": "Πλησιάζετε το όριο ανώνυμης χρήσης. Εγγραφείτε για μια δωρεάν κατανομή.",
"blockReasoning": "Σκέφτεται",
"blockTool": "{{name}}",
"blockRunning": "εκτελείται…",
"showThinking": "Συλλογισμός",
"showThinkingHint": "Ζήτα από το μοντέλο να σκεφτεί πριν απαντήσει και δείξε τον συλλογισμό του"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "reading {{host}}…",
"copyCode": "Copy code",
"copiedCode": "Copied",
"anonNearLimit": "You're close to the anonymous limit. Sign up for a free allocation."
"anonNearLimit": "You're close to the anonymous limit. Sign up for a free allocation.",
"blockReasoning": "Thinking",
"blockTool": "{{name}}",
"blockRunning": "running…",
"showThinking": "Reasoning",
"showThinkingHint": "Ask the model to think before answering, and show its reasoning"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "leyendo {{host}}…",
"copyCode": "Copiar código",
"copiedCode": "Copiado",
"anonNearLimit": "Estás cerca del límite anónimo. Regístrate para obtener una asignación gratuita."
"anonNearLimit": "Estás cerca del límite anónimo. Regístrate para obtener una asignación gratuita.",
"blockReasoning": "Pensando",
"blockTool": "{{name}}",
"blockRunning": "ejecutando…",
"showThinking": "Razonamiento",
"showThinkingHint": "Pide al modelo que piense antes de responder y muestra su razonamiento"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "loetakse {{host}}…",
"copyCode": "Kopeeri kood",
"copiedCode": "Kopeeritud",
"anonNearLimit": "Oled lähenemas anonüümse kasutuse piirile. Registreeru tasuta eraldise saamiseks."
"anonNearLimit": "Oled lähenemas anonüümse kasutuse piirile. Registreeru tasuta eraldise saamiseks.",
"blockReasoning": "Mõtleb",
"blockTool": "{{name}}",
"blockRunning": "töötab…",
"showThinking": "Arutlus",
"showThinkingHint": "Palu mudelil enne vastamist mõelda ja näita tema arutlust"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "در حال خواندن {{host}}…",
"copyCode": "کپی کد",
"copiedCode": "کپی شد",
"anonNearLimit": "به محدودیت کاربران ناشناس نزدیک می‌شوید. برای دریافت سهمیه رایگان ثبت‌نام کنید."
"anonNearLimit": "به محدودیت کاربران ناشناس نزدیک می‌شوید. برای دریافت سهمیه رایگان ثبت‌نام کنید.",
"blockReasoning": "در حال تفکر",
"blockTool": "{{name}}",
"blockRunning": "در حال اجرا…",
"showThinking": "استدلال",
"showThinkingHint": "از مدل بخواهید پیش از پاسخ فکر کند و استدلالش را نشان دهد"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "luetaan {{host}}…",
"copyCode": "Kopioi koodi",
"copiedCode": "Kopioitu",
"anonNearLimit": "Lähestyt anonyymin käytön rajaa. Rekisteröidy saadaksesi ilmaisen kiintiön."
"anonNearLimit": "Lähestyt anonyymin käytön rajaa. Rekisteröidy saadaksesi ilmaisen kiintiön.",
"blockReasoning": "Ajattelee",
"blockTool": "{{name}}",
"blockRunning": "suoritetaan…",
"showThinking": "Päättely",
"showThinkingHint": "Pyydä mallia ajattelemaan ennen vastaamista ja näytä sen päättely"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "lecture de {{host}}…",
"copyCode": "Copier le code",
"copiedCode": "Copié",
"anonNearLimit": "Vous approchez de la limite anonyme. Inscrivez-vous pour obtenir une allocation gratuite."
"anonNearLimit": "Vous approchez de la limite anonyme. Inscrivez-vous pour obtenir une allocation gratuite.",
"blockReasoning": "Réflexion",
"blockTool": "{{name}}",
"blockRunning": "en cours…",
"showThinking": "Raisonnement",
"showThinkingHint": "Demander au modèle de réfléchir avant de répondre et afficher son raisonnement"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "ag léamh {{host}}…",
"copyCode": "Cóipeáil an cód",
"copiedCode": "Cóipeáilte",
"anonNearLimit": "Tá tú ag druidim leis an teorainn gan ainm. Cláraigh le haghaidh leithdháileadh saor in aisce."
"anonNearLimit": "Tá tú ag druidim leis an teorainn gan ainm. Cláraigh le haghaidh leithdháileadh saor in aisce.",
"blockReasoning": "Ag machnamh",
"blockTool": "{{name}}",
"blockRunning": "ar siúl…",
"showThinking": "Réasúnaíocht",
"showThinkingHint": "Iarr ar an tsamhail machnamh a dhéanamh sula bhfreagraíonn sí, agus taispeáin é"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "ana karanta {{host}}…",
"copyCode": "Kwafi lambar",
"copiedCode": "An kwafa",
"anonNearLimit": "Ka kusa kai iyakar amfani ba tare da shiga ba. Yi rajista don samun rabo kyauta."
"anonNearLimit": "Ka kusa kai iyakar amfani ba tare da shiga ba. Yi rajista don samun rabo kyauta.",
"blockReasoning": "Yana tunani",
"blockTool": "{{name}}",
"blockRunning": "yana gudana…",
"showThinking": "Tunani",
"showThinkingHint": "Ka nemi samfurin ya yi tunani kafin amsa, kuma a nuna tunaninsa"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "מתבצעת קריאה של {{host}}…",
"copyCode": "העתקת הקוד",
"copiedCode": "הועתק",
"anonNearLimit": "אתם מתקרבים למגבלת השימוש האנונימי. הירשמו לקבלת הקצאה חינם."
"anonNearLimit": "אתם מתקרבים למגבלת השימוש האנונימי. הירשמו לקבלת הקצאה חינם.",
"blockReasoning": "חושב",
"blockTool": "{{name}}",
"blockRunning": "פועל…",
"showThinking": "חשיבה",
"showThinkingHint": "בקש מהמודל לחשוב לפני שהוא עונה, והצג את החשיבה שלו"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "čitanje {{host}}…",
"copyCode": "Kopiraj kôd",
"copiedCode": "Kopirano",
"anonNearLimit": "Približavate se anonimnom limitu. Registrirajte se za besplatnu kvotu."
"anonNearLimit": "Približavate se anonimnom limitu. Registrirajte se za besplatnu kvotu.",
"blockReasoning": "Razmišljanje",
"blockTool": "{{name}}",
"blockRunning": "izvodi se…",
"showThinking": "Zaključivanje",
"showThinkingHint": "Zatraži da model razmisli prije odgovora i prikaži njegovo zaključivanje"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "{{host}} olvasása…",
"copyCode": "Kód másolása",
"copiedCode": "Másolva",
"anonNearLimit": "Közeledsz az anonim limithez. Regisztrálj ingyenes keretért."
"anonNearLimit": "Közeledsz az anonim limithez. Regisztrálj ingyenes keretért.",
"blockReasoning": "Gondolkodik",
"blockTool": "{{name}}",
"blockRunning": "fut…",
"showThinking": "Gondolatmenet",
"showThinkingHint": "Kérd meg a modellt, hogy gondolkodjon válasz előtt, és mutasd a gondolatmenetét"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "lettura di {{host}} in corso…",
"copyCode": "Copia il codice",
"copiedCode": "Copiato",
"anonNearLimit": "Ti stai avvicinando al limite anonimo. Registrati per un’allocazione gratuita."
"anonNearLimit": "Ti stai avvicinando al limite anonimo. Registrati per un’allocazione gratuita.",
"blockReasoning": "Sto pensando",
"blockTool": "{{name}}",
"blockRunning": "in esecuzione…",
"showThinking": "Ragionamento",
"showThinkingHint": "Chiedi al modello di ragionare prima di rispondere e mostra il ragionamento"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "მიმდინარეობს {{host}}-ის კითხვა…",
"copyCode": "კოდის კოპირება",
"copiedCode": "დაკოპირდა",
"anonNearLimit": "უახლოვდები ანონიმურ ლიმიტს. დარეგისტრირდი უფასო კვოტისთვის."
"anonNearLimit": "უახლოვდები ანონიმურ ლიმიტს. დარეგისტრირდი უფასო კვოტისთვის.",
"blockReasoning": "ფიქრობს",
"blockTool": "{{name}}",
"blockRunning": "მიმდინარეობს…",
"showThinking": "მსჯელობა",
"showThinkingHint": "სთხოვე მოდელს იფიქროს პასუხამდე და აჩვენე მისი მსჯელობა"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "{{host}} оқылуда…",
"copyCode": "Кодты көшіру",
"copiedCode": "Көшірілді",
"anonNearLimit": "Анонимді шекке жақындап қалдыңыз. Тегін квота алу үшін тіркеліңіз."
"anonNearLimit": "Анонимді шекке жақындап қалдыңыз. Тегін квота алу үшін тіркеліңіз.",
"blockReasoning": "Ойлануда",
"blockTool": "{{name}}",
"blockRunning": "орындалуда…",
"showThinking": "Пайымдау",
"showThinkingHint": "Жауап бермес бұрын модель ойлансын және пайымдауын көрсетсін"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "skaitoma {{host}}…",
"copyCode": "Kopijuoti kodą",
"copiedCode": "Nukopijuota",
"anonNearLimit": "Artėjate prie anoniminio naudojimo ribos. Užsiregistruokite ir gaukite nemokamą kvotą."
"anonNearLimit": "Artėjate prie anoniminio naudojimo ribos. Užsiregistruokite ir gaukite nemokamą kvotą.",
"blockReasoning": "Mąsto",
"blockTool": "{{name}}",
"blockRunning": "vykdoma…",
"showThinking": "Samprotavimas",
"showThinkingHint": "Paprašyk modelio pamąstyti prieš atsakant ir rodyk jo samprotavimą"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "lasa {{host}}…",
"copyCode": "Kopēt kodu",
"copiedCode": "Nokopēts",
"anonNearLimit": "Tuvojaties anonīmās lietošanas limitam. Reģistrējieties bezmaksas piešķīrumam."
"anonNearLimit": "Tuvojaties anonīmās lietošanas limitam. Reģistrējieties bezmaksas piešķīrumam.",
"blockReasoning": "Domā",
"blockTool": "{{name}}",
"blockRunning": "notiek…",
"showThinking": "Spriešana",
"showThinkingHint": "Lūdz modelim padomāt pirms atbildes un rādi tā spriešanu"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "كنقراو {{host}}…",
"copyCode": "نسخ الكود",
"copiedCode": "تنسخ",
"anonNearLimit": "قربتي توصل للحد ديال الاستعمال بلا حساب. سجّل باش تاخد حصة بلاش."
"anonNearLimit": "قربتي توصل للحد ديال الاستعمال بلا حساب. سجّل باش تاخد حصة بلاش.",
"blockReasoning": "كيفكر",
"blockTool": "{{name}}",
"blockRunning": "كيخدم…",
"showThinking": "التفكير",
"showThinkingHint": "طلب من الموديل يفكر قبل ما يجاوب، وبيّن التفكير ديالو"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "qed naqra {{host}}…",
"copyCode": "Ikkopja l-kodiċi",
"copiedCode": "Ikkupjat",
"anonNearLimit": "Qed toqrob lejn il-limitu anonimu. Irreġistra għal allokazzjoni b'xejn."
"anonNearLimit": "Qed toqrob lejn il-limitu anonimu. Irreġistra għal allokazzjoni b'xejn.",
"blockReasoning": "Qed jaħseb",
"blockTool": "{{name}}",
"blockRunning": "għaddej…",
"showThinking": "Raġunament",
"showThinkingHint": "Itlob lill-mudell jaħseb qabel iwieġeb, u uri r-raġunament tiegħu"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "lezen van {{host}}…",
"copyCode": "Code kopiëren",
"copiedCode": "Gekopieerd",
"anonNearLimit": "Je nadert de anonieme limiet. Registreer je voor een gratis tegoed."
"anonNearLimit": "Je nadert de anonieme limiet. Registreer je voor een gratis tegoed.",
"blockReasoning": "Denkt na",
"blockTool": "{{name}}",
"blockRunning": "bezig…",
"showThinking": "Redenering",
"showThinkingHint": "Laat het model nadenken vóór het antwoordt en toon de redenering"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "leser {{host}}…",
"copyCode": "Kopier kode",
"copiedCode": "Kopiert",
"anonNearLimit": "Du nærmer deg den anonyme grensen. Registrer deg og få en gratis tildeling."
"anonNearLimit": "Du nærmer deg den anonyme grensen. Registrer deg og få en gratis tildeling.",
"blockReasoning": "Tenker",
"blockTool": "{{name}}",
"blockRunning": "kjører…",
"showThinking": "Resonnement",
"showThinkingHint": "Be modellen tenke før den svarer, og vis resonnementet"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "czytanie {{host}}…",
"copyCode": "Kopiuj kod",
"copiedCode": "Skopiowano",
"anonNearLimit": "Zbliżasz się do limitu anonimowego użycia. Zarejestruj się, aby otrzymać bezpłatny przydział."
"anonNearLimit": "Zbliżasz się do limitu anonimowego użycia. Zarejestruj się, aby otrzymać bezpłatny przydział.",
"blockReasoning": "Myślenie",
"blockTool": "{{name}}",
"blockRunning": "w toku…",
"showThinking": "Rozumowanie",
"showThinkingHint": "Poproś model, by pomyślał przed odpowiedzią, i pokaż jego rozumowanie"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "a ler {{host}}…",
"copyCode": "Copiar código",
"copiedCode": "Copiado",
"anonNearLimit": "Está a aproximar-se do limite anónimo. Registe-se para obter uma alocação gratuita."
"anonNearLimit": "Está a aproximar-se do limite anónimo. Registe-se para obter uma alocação gratuita.",
"blockReasoning": "A pensar",
"blockTool": "{{name}}",
"blockRunning": "em execução…",
"showThinking": "Raciocínio",
"showThinkingHint": "Pedir ao modelo que pense antes de responder e mostrar o raciocínio"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "se citește {{host}}…",
"copyCode": "Copiază codul",
"copiedCode": "Copiat",
"anonNearLimit": "Te apropii de limita pentru utilizatori anonimi. Creează un cont pentru o alocare gratuită."
"anonNearLimit": "Te apropii de limita pentru utilizatori anonimi. Creează un cont pentru o alocare gratuită.",
"blockReasoning": "Gândește",
"blockTool": "{{name}}",
"blockRunning": "în curs…",
"showThinking": "Raționament",
"showThinkingHint": "Cere modelului să gândească înainte de a răspunde și arată raționamentul"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "читаем {{host}}…",
"copyCode": "Копировать код",
"copiedCode": "Скопировано",
"anonNearLimit": "Вы приближаетесь к лимиту анонимного доступа. Зарегистрируйтесь и получите бесплатную квоту."
"anonNearLimit": "Вы приближаетесь к лимиту анонимного доступа. Зарегистрируйтесь и получите бесплатную квоту.",
"blockReasoning": "Размышление",
"blockTool": "{{name}}",
"blockRunning": "выполняется…",
"showThinking": "Рассуждение",
"showThinkingHint": "Просить модель подумать перед ответом и показывать её рассуждение"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "čítam {{host}}…",
"copyCode": "Kopírovať kód",
"copiedCode": "Skopírované",
"anonNearLimit": "Blížite sa k anonymnému limitu. Zaregistrujte sa a získajte bezplatný prídel."
"anonNearLimit": "Blížite sa k anonymnému limitu. Zaregistrujte sa a získajte bezplatný prídel.",
"blockReasoning": "Premýšľa",
"blockTool": "{{name}}",
"blockRunning": "prebieha…",
"showThinking": "Uvažovanie",
"showThinkingHint": "Nech model pred odpoveďou premýšľa a ukáže svoje uvažovanie"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "branje {{host}} …",
"copyCode": "Kopiraj kodo",
"copiedCode": "Kopirano",
"anonNearLimit": "Približujete se omejitvi za anonimne uporabnike. Registrirajte se za brezplačno dodelitev."
"anonNearLimit": "Približujete se omejitvi za anonimne uporabnike. Registrirajte se za brezplačno dodelitev.",
"blockReasoning": "Razmišlja",
"blockTool": "{{name}}",
"blockRunning": "poteka…",
"showThinking": "Sklepanje",
"showThinkingHint": "Naj model pred odgovorom premisli in pokaže svoje sklepanje"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "читам {{host}}…",
"copyCode": "Копирај кôд",
"copiedCode": "Копирано",
"anonNearLimit": "Приближавате се анонимном ограничењу. Региструјте се за бесплатну квоту."
"anonNearLimit": "Приближавате се анонимном ограничењу. Региструјте се за бесплатну квоту.",
"blockReasoning": "Размишљање",
"blockTool": "{{name}}",
"blockRunning": "у току…",
"showThinking": "Резоновање",
"showThinkingHint": "Затражи да модел размисли пре одговора и прикажи његово резоновање"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "läser {{host}}…",
"copyCode": "Kopiera kod",
"copiedCode": "Kopierad",
"anonNearLimit": "Du närmar dig den anonyma gränsen. Registrera dig för en gratis tilldelning."
"anonNearLimit": "Du närmar dig den anonyma gränsen. Registrera dig för en gratis tilldelning.",
"blockReasoning": "Tänker",
"blockTool": "{{name}}",
"blockRunning": "körs…",
"showThinking": "Resonemang",
"showThinkingHint": "Be modellen tänka innan den svarar och visa resonemanget"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "inasoma {{host}}…",
"copyCode": "Nakili msimbo",
"copiedCode": "Imenakiliwa",
"anonNearLimit": "Unakaribia kikomo cha matumizi bila akaunti. Jisajili upate mgao wa bure."
"anonNearLimit": "Unakaribia kikomo cha matumizi bila akaunti. Jisajili upate mgao wa bure.",
"blockReasoning": "Inafikiri",
"blockTool": "{{name}}",
"blockRunning": "inaendelea…",
"showThinking": "Ufikirio",
"showThinkingHint": "Mwombe modeli afikirie kabla ya kujibu, na uonyeshe ufikirio wake"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "{{host}} okunuyor…",
"copyCode": "Kodu kopyala",
"copiedCode": "Kopyalandı",
"anonNearLimit": "Anonim sınıra yaklaşıyorsunuz. Ücretsiz tahsisat için kaydolun."
"anonNearLimit": "Anonim sınıra yaklaşıyorsunuz. Ücretsiz tahsisat için kaydolun.",
"blockReasoning": "Düşünüyor",
"blockTool": "{{name}}",
"blockRunning": "çalışıyor…",
"showThinking": "Akıl yürütme",
"showThinkingHint": "Modelden yanıtlamadan önce düşünmesini iste ve akıl yürütmesini göster"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "читаємо {{host}}…",
"copyCode": "Копіювати код",
"copiedCode": "Скопійовано",
"anonNearLimit": "Ви наближаєтеся до анонімного ліміту. Зареєструйтеся, щоб отримати безкоштовну квоту."
"anonNearLimit": "Ви наближаєтеся до анонімного ліміту. Зареєструйтеся, щоб отримати безкоштовну квоту.",
"blockReasoning": "Роздуми",
"blockTool": "{{name}}",
"blockRunning": "виконується…",
"showThinking": "Міркування",
"showThinkingHint": "Просити модель подумати перед відповіддю та показувати її міркування"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "{{host}} oʻqilmoqda…",
"copyCode": "Kodni nusxalash",
"copiedCode": "Nusxalandi",
"anonNearLimit": "Anonim cheklovga yaqinlashyapsiz. Bepul kvota olish uchun roʻyxatdan oʻting."
"anonNearLimit": "Anonim cheklovga yaqinlashyapsiz. Bepul kvota olish uchun roʻyxatdan oʻting.",
"blockReasoning": "O‘ylanmoqda",
"blockTool": "{{name}}",
"blockRunning": "bajarilmoqda…",
"showThinking": "Mulohaza",
"showThinkingHint": "Modeldan javobdan oldin o‘ylashini so‘rang va mulohazasini ko‘rsating"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "ń ka {{host}}…",
"copyCode": "Daakọ kóòdù",
"copiedCode": "Ti daakọ",
"anonNearLimit": "O ti fẹ́rẹ̀ẹ́ dé ààlà aláìlórúkọ. Forúkọsílẹ̀ fún ìpín ọ̀fẹ́."
"anonNearLimit": "O ti fẹ́rẹ̀ẹ́ dé ààlà aláìlórúkọ. Forúkọsílẹ̀ fún ìpín ọ̀fẹ́.",
"blockReasoning": "Ó ń ronú",
"blockTool": "{{name}}",
"blockRunning": "ń ṣiṣẹ́…",
"showThinking": "Ìrònú",
"showThinkingHint": "Béèrè pé kí àwòṣe náà ronú kí ó tó dáhùn, kí a sì fi ìrònú rẹ̀ hàn"
}
+6 -1
View File
@@ -29,5 +29,10 @@
"reading": "kufundwa {{host}}…",
"copyCode": "Kopisha ikhodi",
"copiedCode": "Kukopishiwe",
"anonNearLimit": "Usondela emkhawulweni wokungaziwa. Bhalisa ukuze uthole isabelo samahhala."
"anonNearLimit": "Usondela emkhawulweni wokungaziwa. Bhalisa ukuze uthole isabelo samahhala.",
"blockReasoning": "Iyacabanga",
"blockTool": "{{name}}",
"blockRunning": "iyasebenza…",
"showThinking": "Ukucabanga",
"showThinkingHint": "Cela imodeli icabange ngaphambi kokuphendula, futhi ubonise ukucabanga kwayo"
}
+154
View File
@@ -0,0 +1,154 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { streamChatCompletion, type ToolCall } from "./chatClient";
/**
* SSE framing.
*
* The parser buffers bytes and splits on a blank line, so the interesting
* cases are all about where the network happens to cut the stream — which
* is precisely what clicking around never reproduces, because a fast
* local origin tends to deliver each frame whole (#304).
*/
/** A response whose body yields exactly these chunks, in order. */
function respondWith(chunks: string[]): Response {
const encoder = new TextEncoder();
const body = new ReadableStream<Uint8Array>({
start(controller) {
for (const c of chunks) controller.enqueue(encoder.encode(c));
controller.close();
},
});
return new Response(body, {
status: 200,
headers: { "content-type": "text/event-stream" },
});
}
function frame(delta: Record<string, unknown>): string {
return `data: ${JSON.stringify({ choices: [{ delta }] })}\n\n`;
}
/** Run a stream and collect everything the handlers saw. */
async function collect(chunks: string[]) {
vi.stubGlobal("fetch", vi.fn().mockResolvedValue(respondWith(chunks)));
const text: string[] = [];
const reasoning: string[] = [];
const tools: ToolCall[] = [];
let done = false;
let error: { code: string; message: string } | null = null;
await streamChatCompletion(
{
model: "m",
messages: [{ role: "user", content: "hi" }],
signal: new AbortController().signal,
},
{
onDelta: (t) => text.push(t),
onReasoning: (t) => reasoning.push(t),
onToolCall: (c) => tools.push(c),
onDone: () => {
done = true;
},
onError: (code, message) => {
error = { code, message };
},
},
);
return { text: text.join(""), reasoning: reasoning.join(""), tools, done, error };
}
afterEach(() => vi.unstubAllGlobals());
describe("SSE framing", () => {
it("delivers content deltas in order", async () => {
const r = await collect([frame({ content: "he" }), frame({ content: "llo" }), "data: [DONE]\n\n"]);
expect(r.text).toBe("hello");
expect(r.done).toBe(true);
expect(r.error).toBeNull();
});
it("reassembles a frame split across two reads", async () => {
// The case unit tests exist for. A chunk boundary can fall anywhere,
// including mid-JSON; the buffer must hold the partial frame rather
// than trying to parse it.
const whole = frame({ content: "split" });
const cut = Math.floor(whole.length / 2);
const r = await collect([whole.slice(0, cut), whole.slice(cut), "data: [DONE]\n\n"]);
expect(r.text).toBe("split");
});
it("handles several frames arriving in one read", async () => {
const r = await collect([
frame({ content: "a" }) + frame({ content: "b" }) + frame({ content: "c" }),
"data: [DONE]\n\n",
]);
expect(r.text).toBe("abc");
});
it("separates reasoning from content", async () => {
// `reasoning_content` must never leak into the answer: the whole
// reason neuron emits it as its own field is that an unaware client
// still renders clean output.
const r = await collect([
frame({ reasoning_content: "weighing " }),
frame({ reasoning_content: "options" }),
frame({ content: "the answer" }),
"data: [DONE]\n\n",
]);
expect(r.reasoning).toBe("weighing options");
expect(r.text).toBe("the answer");
});
it("stops at [DONE] and ignores anything after it", async () => {
const r = await collect([
frame({ content: "kept" }),
"data: [DONE]\n\n",
frame({ content: "after the end" }),
]);
expect(r.text).toBe("kept");
expect(r.done).toBe(true);
});
it("skips a malformed frame without losing the stream", async () => {
// One bad frame must not abort the turn — the tokens around it are
// still the model's answer.
const r = await collect([
frame({ content: "before " }),
"data: {not json\n\n",
frame({ content: "after" }),
"data: [DONE]\n\n",
]);
expect(r.text).toBe("before after");
expect(r.error).toBeNull();
});
it("ignores non-data lines", async () => {
// Comments (keep-alives) and event: lines are legal SSE and carry
// nothing this client needs.
const r = await collect([
": keep-alive\n\n",
"event: ping\ndata: {}\n\n",
frame({ content: "ok" }),
"data: [DONE]\n\n",
]);
expect(r.text).toBe("ok");
});
it("surfaces a tool call once its name and id are present", async () => {
const r = await collect([
frame({
tool_calls: [
{ id: "call_1", function: { name: "web_search", arguments: '{"query":"q"}' } },
],
}),
"data: [DONE]\n\n",
]);
expect(r.tools).toHaveLength(1);
expect(r.tools[0]).toMatchObject({
id: "call_1",
function: { name: "web_search", arguments: '{"query":"q"}' },
});
});
});
+33
View File
@@ -21,6 +21,19 @@ export interface ChatMessage {
export interface StreamHandlers {
onDelta: (text: string) => void;
/**
* A reasoning delta, from `choice.delta.reasoning_content`.
*
* Not in the OpenAI spec but the de-facto slot — DeepSeek, vLLM and
* SGLang all use it, and neuron emits it by default rather than
* folding reasoning into `content`, so a client that ignores the
* field still shows a clean answer.
*
* Worth handling rather than dropping: on a hard prompt the model can
* reason for minutes, and these are the only events on the wire for
* that whole span. A UI that ignores them looks hung.
*/
onReasoning?: (text: string) => void;
/** A complete tool call arrived (neuron buffers the whole
* `<tool_call>` block, so arguments are never fragmented). */
onToolCall?: (call: ToolCall) => void;
@@ -36,6 +49,17 @@ export interface StreamOptions {
messages: ChatMessage[];
/** OpenAI tools array; omitted = no tools offered. */
tools?: readonly unknown[];
/**
* Ask the model to reason, and to send that reasoning back.
*
* neuron couples generation to surfacing: unless a caller says it will
* display reasoning, `enable_thinking` defaults to false and the model
* produces none at all. That default is right — an undisplayed think
* block still eats the output budget — but it means the reasoning UI
* shows nothing until the client opts in. Verified against the live
* router: without this, `reasoning_tokens=0` on every request.
*/
includeThinking?: boolean;
signal: AbortSignal;
}
@@ -71,6 +95,11 @@ export async function streamChatCompletion(
accept: "text/event-stream",
};
if (opts.apiKey) headers.authorization = `Bearer ${opts.apiKey}`;
// Only sent when on. Absent means "naïve client", which is the
// server-side default and the behaviour every other OpenAI client
// gets — worth preserving exactly, since this toggle exists partly
// to demonstrate both paths.
if (opts.includeThinking) headers["x-include-thinking"] = "true";
resp = await fetch(`${base}/v1/chat/completions`, {
method: "POST",
headers,
@@ -134,6 +163,10 @@ export async function streamChatCompletion(
const json = JSON.parse(data);
const delta = json?.choices?.[0]?.delta?.content;
if (typeof delta === "string" && delta) h.onDelta(delta);
const reasoning = json?.choices?.[0]?.delta?.reasoning_content;
if (typeof reasoning === "string" && reasoning && h.onReasoning) {
h.onReasoning(reasoning);
}
const toolCalls = json?.choices?.[0]?.delta?.tool_calls;
if (Array.isArray(toolCalls) && h.onToolCall) {
for (const tc of toolCalls) {
+73
View File
@@ -0,0 +1,73 @@
import { describe, expect, it } from "vitest";
import { isLive, isOpen } from "./turnBlockState";
/**
* The visibility rule for the assistant's working, stated as behaviours
* rather than as a truth table.
*
* These began as a throwaway node harness written while building
* `TurnBlocks`, because the project had no runner (#304). Its first
* version re-declared the rule instead of importing it — which checks
* the description and not the code, and would have passed against any
* implementation at all. Importing the real module is the whole point.
*/
describe("which blocks are open", () => {
describe("while the turn is streaming and no answer has started", () => {
it("opens the block being written", () => {
expect(isOpen(0, 1, true, false, undefined)).toBe(true);
});
it("collapses an earlier block once a new one begins", () => {
// Reasoning, then a tool call: the reasoning is finished work.
expect(isOpen(0, 2, true, false, undefined)).toBe(false);
expect(isOpen(1, 2, true, false, undefined)).toBe(true);
});
});
it("collapses everything the moment answer text arrives", () => {
// The block that was live one token ago is no longer live. This is
// the behaviour the whole feature exists for: the answer takes the
// space as soon as there is an answer.
expect(isOpen(1, 2, true, true, undefined)).toBe(false);
expect(isOpen(0, 2, true, true, undefined)).toBe(false);
});
it("loads completed turns collapsed", () => {
expect(isOpen(0, 3, false, true, undefined)).toBe(false);
expect(isOpen(2, 3, false, true, undefined)).toBe(false);
});
it("collapses a turn stopped mid-reasoning", () => {
// Stop pressed while thinking: no answer ever arrives, but the turn
// is over. Keyed on `streaming`, not on the presence of an answer —
// otherwise an abandoned turn would sit permanently expanded.
expect(isOpen(0, 1, false, false, undefined)).toBe(false);
});
describe("when the user has clicked", () => {
it("keeps a finished block open", () => {
expect(isOpen(0, 3, false, true, true)).toBe(true);
});
it("keeps the live block closed", () => {
// Someone who closes a noisy reasoning stream must not have it
// spring open again on the next token.
expect(isOpen(1, 2, true, false, false)).toBe(false);
});
});
});
describe("isLive", () => {
it("is true only for the final block of a streaming, answerless turn", () => {
expect(isLive(2, 3, true, false)).toBe(true);
expect(isLive(1, 3, true, false)).toBe(false);
expect(isLive(2, 3, true, true)).toBe(false);
expect(isLive(2, 3, false, false)).toBe(false);
});
it("is false when there are no blocks", () => {
// count 0 makes `count - 1` negative; no index can match, and
// nothing should claim to be live in an empty list.
expect(isLive(0, 0, true, false)).toBe(false);
});
});
+43
View File
@@ -0,0 +1,43 @@
/**
* When the assistant's working is shown, and when it gets out of the way.
*
* Kept in its own module, free of React and icon imports, so the rule can
* be imported and exercised on its own. It is the entire behavioural
* contract of `TurnBlocks`, and as an inline expression inside the
* component the only way to check it was to click around.
*/
/**
* Is this the block the assistant is working in right now?
*
* Only the last block can be live, and only while the turn is streaming
* with no answer text yet. Once answer text starts, the working becomes
* history and nothing is live — including the block that was live a
* token earlier.
*/
export function isLive(
index: number,
count: number,
streaming: boolean,
answerStarted: boolean,
): boolean {
return streaming && !answerStarted && index === count - 1;
}
/**
* Should this block be rendered open?
*
* `override` is the user's own click, and it wins outright. Someone who
* closes a noisy reasoning stream should not have it spring back open on
* the next token, and someone who opens a finished block should keep it
* open while the rest of the turn streams past.
*/
export function isOpen(
index: number,
count: number,
streaming: boolean,
answerStarted: boolean,
override: boolean | undefined,
): boolean {
return override ?? isLive(index, count, streaming, answerStarted);
}
+111
View File
@@ -0,0 +1,111 @@
import { describe, expect, it } from "vitest";
import type { TurnBlock } from "../data/db";
import { appendReasoning, appendToolCall, attachToolResult } from "./turnBlocks";
describe("appendReasoning", () => {
it("starts a block when there is nothing to extend", () => {
const blocks: TurnBlock[] = [];
appendReasoning(blocks, "let me ");
expect(blocks).toEqual([{ kind: "reasoning", text: "let me " }]);
});
it("extends the open reasoning block across deltas", () => {
// Reasoning arrives token by token; one block, not one per token.
const blocks: TurnBlock[] = [];
appendReasoning(blocks, "let me ");
appendReasoning(blocks, "check ");
appendReasoning(blocks, "that");
expect(blocks).toEqual([{ kind: "reasoning", text: "let me check that" }]);
});
it("starts a new block for reasoning that follows a tool call", () => {
// think → act → think is three blocks, not one interrupted one.
// Merging them would claim the model reasoned continuously through a
// call it was actually waiting on.
const blocks: TurnBlock[] = [];
appendReasoning(blocks, "I should search");
appendToolCall(blocks, "web_search", '{"query":"a"}');
appendReasoning(blocks, "those results suggest");
expect(blocks.map((b) => b.kind)).toEqual(["reasoning", "tool", "reasoning"]);
expect(blocks[0]).toEqual({ kind: "reasoning", text: "I should search" });
expect(blocks[2]).toEqual({ kind: "reasoning", text: "those results suggest" });
});
});
describe("attachToolResult", () => {
it("fills in the call it answers", () => {
const blocks: TurnBlock[] = [];
appendToolCall(blocks, "web_search", '{"query":"helexa"}');
expect(attachToolResult(blocks, "web_search", "one hit")).toBe(true);
expect(blocks[0]).toEqual({
kind: "tool",
name: "web_search",
args: '{"query":"helexa"}',
result: "one hit",
});
});
it("pairs repeated calls to one tool in order", () => {
// The case that matters, and the one the first implementation got
// backwards. Two searches in a turn is ordinary; results come back
// in call order, so first result belongs to first call. Matching
// newest-first silently files each answer under the wrong query and
// the UI looks perfectly healthy.
const blocks: TurnBlock[] = [];
appendToolCall(blocks, "web_search", '{"query":"first"}');
appendToolCall(blocks, "web_search", '{"query":"second"}');
attachToolResult(blocks, "web_search", "results for first");
attachToolResult(blocks, "web_search", "results for second");
const tools = blocks.filter((b) => b.kind === "tool");
expect(tools[0]).toMatchObject({
args: '{"query":"first"}',
result: "results for first",
});
expect(tools[1]).toMatchObject({
args: '{"query":"second"}',
result: "results for second",
});
});
it("keeps interleaved tools apart", () => {
// Ordering is guaranteed per tool, not globally, so a read_page
// result must not be able to land on a pending web_search.
const blocks: TurnBlock[] = [];
appendToolCall(blocks, "web_search", '{"query":"q"}');
appendToolCall(blocks, "read_page", '{"url":"https://example.com"}');
attachToolResult(blocks, "read_page", "page text");
// `not.toHaveProperty` rather than `toMatchObject({result: undefined})`
// — the latter passes against an absent key *and* an explicit
// undefined, so it would not distinguish "still pending" from
// "answered with nothing".
expect(blocks[0]).toMatchObject({ name: "web_search" });
expect(blocks[0]).not.toHaveProperty("result");
expect(blocks[1]).toMatchObject({ name: "read_page", result: "page text" });
});
it("reports a result with no matching call instead of dropping it", () => {
const blocks: TurnBlock[] = [];
appendToolCall(blocks, "web_search", "{}");
attachToolResult(blocks, "web_search", "first");
// Second result, only one call recorded: nowhere to put it.
expect(attachToolResult(blocks, "web_search", "second")).toBe(false);
expect(blocks[0]).toMatchObject({ result: "first" });
});
it("does not overwrite a result already attached", () => {
const blocks: TurnBlock[] = [];
appendToolCall(blocks, "web_search", "{}");
appendToolCall(blocks, "web_search", "{}");
attachToolResult(blocks, "web_search", "one");
attachToolResult(blocks, "web_search", "two");
expect(blocks.map((b) => (b.kind === "tool" ? b.result : null))).toEqual([
"one",
"two",
]);
});
});
+70
View File
@@ -0,0 +1,70 @@
import type { TurnBlock } from "../data/db";
/**
* Building the ordered record of what the assistant did on the way to an
* answer.
*
* Kept separate from `useChat` so the rules can be tested (#304). The
* result-matching one in particular is subtle and fails silently: get it
* wrong and the UI shows one search's answer under a different search's
* query, with nothing to indicate anything went astray.
*
* Every function mutates the array in place. The caller owns it and
* writes absolute snapshots to storage, matching how streamed content is
* persisted — read-modify-write appends race each other when deltas
* arrive faster than an IndexedDB round-trip.
*/
/**
* Add a reasoning delta.
*
* Extends the open reasoning block, or starts one if the previous block
* was something else. That is what preserves think → act → think order:
* reasoning after a tool call is a *new* thought about the result, not a
* continuation of the thought that led to the call.
*/
export function appendReasoning(blocks: TurnBlock[], text: string): void {
const last = blocks[blocks.length - 1];
if (last?.kind === "reasoning") last.text += text;
else blocks.push({ kind: "reasoning", text });
}
/** Record a tool call. Its result arrives later, via `attachToolResult`. */
export function appendToolCall(blocks: TurnBlock[], name: string, args: string): void {
blocks.push({ kind: "tool", name, args });
}
/**
* Attach a result to the call it answers.
*
* Matches the **oldest** block with this tool name still awaiting a
* result — first-in, first-out. `useChat` records calls in arrival order
* and then executes them in that same order, so the first result belongs
* to the first call.
*
* Scanning from the newest end instead is wrong in a way nothing would
* report: with two `web_search` calls in one turn — the ordinary case,
* not an edge one — the first query's results land under the second
* query's heading and the UI looks entirely healthy. The first draft of
* this did exactly that, and only writing the rule down as a testable
* function made it visible.
*
* Matching by name as well as by pending-ness keeps interleaved tools
* apart: ordering is guaranteed per tool, not globally.
*
* Returns whether a home was found. `false` means a result arrived for a
* call that was never recorded — worth surfacing rather than dropping.
*/
export function attachToolResult(
blocks: TurnBlock[],
name: string,
result: string,
): boolean {
for (const b of blocks) {
if (b.kind === "tool" && b.name === name && b.result === undefined) {
b.result = result;
return true;
}
}
return false;
}
+39 -2
View File
@@ -9,12 +9,13 @@
// across rounds; consulted sources persist as citations.
import { useRef, useState } from "react";
import type { MessageSource } from "../data/db";
import type { MessageSource, TurnBlock } from "../data/db";
import {
addMessage,
finalizeMessage,
listMessages,
renameConversation,
setMessageBlocks,
setMessageContent,
} from "../data/repositories";
import {
@@ -29,6 +30,7 @@ import {
WEB_SEARCH_TOOL,
} from "./searchTool";
import { buildSystemPrompt } from "./systemPrompt";
import { appendReasoning, appendToolCall, attachToolResult } from "./turnBlocks";
/** The tool activity currently executing, for the UI status line. */
export interface ToolActivity {
@@ -57,6 +59,8 @@ export function useChat(opts: {
* `true` unconditionally. Also selects the tool-aware system prompt.
*/
toolsEnabled: boolean;
/** Ask the model to reason, and show that reasoning. */
includeThinking?: boolean;
}): UseChat {
const [streaming, setStreaming] = useState(false);
const [activity, setActivity] = useState<ToolActivity | null>(null);
@@ -123,6 +127,28 @@ export function useChat(opts: {
}
};
// Reasoning and tool calls, in the order they happened. Written as
// absolute snapshots on the same coalesced-write discipline as
// `acc` — Dexie updates race each other otherwise.
const blocks: TurnBlock[] = [];
let blocksWriting = false;
let blocksDirty = false;
const flushBlocks = async () => {
if (blocksWriting) {
blocksDirty = true;
return;
}
blocksWriting = true;
try {
do {
blocksDirty = false;
await setMessageBlocks(assistantId, blocks.map((b) => ({ ...b })));
} while (blocksDirty);
} finally {
blocksWriting = false;
}
};
const sources: MessageSource[] = [];
const seenUrls = new Set<string>();
let failed = false;
@@ -153,6 +179,7 @@ export function useChat(opts: {
apiKey: opts.apiKey,
model: opts.model,
messages: reqMessages,
includeThinking: opts.includeThinking,
tools: offerTools ? [WEB_SEARCH_TOOL, READ_PAGE_TOOL] : undefined,
signal: controller.signal,
},
@@ -161,7 +188,15 @@ export function useChat(opts: {
acc += t;
flushed = flush();
},
onToolCall: (call) => toolCalls.push(call),
onReasoning: (t) => {
appendReasoning(blocks, t);
void flushBlocks();
},
onToolCall: (call) => {
toolCalls.push(call);
appendToolCall(blocks, call.function.name, call.function.arguments);
void flushBlocks();
},
onUsage: (p, c) =>
void finalizeMessage(assistantId, { promptTokens: p, completionTokens: c }),
onDone: () => {},
@@ -221,6 +256,8 @@ export function useChat(opts: {
content = r.content;
}
reqMessages.push({ role: "tool", tool_call_id: call.id, content });
attachToolResult(blocks, call.function.name, content);
void flushBlocks();
}
setActivity(null);
// Visual seam between the pre-search text and the answer round.
+47
View File
@@ -14,6 +14,7 @@ import {
LuX,
} from "react-icons/lu";
import Markdown from "../components/Markdown";
import { TurnBlocks } from "../components/TurnBlocks";
import { CHAT_API_KEY, db } from "../data/db";
import {
archiveProject,
@@ -31,6 +32,9 @@ import { useAuth } from "../auth/context";
import { ensureChatKey } from "../lib/ensureChatKey";
import { accountApi } from "../api/account";
/** Persisted across reloads so a demo keeps whatever was last shown. */
const THINKING_PREF_KEY = "helexa.chat.showThinking";
const ANON_MODEL = import.meta.env.VITE_ANON_MODEL || "helexa/small";
const AUTH_MODEL = import.meta.env.VITE_DEFAULT_MODEL || "helexa/balanced";
const ANON_MESSAGE_CAP = 20;
@@ -157,11 +161,35 @@ export default function Chat() {
[],
);
// Reasoning is opt-in per #304's showcase argument: neuron only
// *generates* reasoning for callers that say they will display it, so
// this toggle switches a real server-side behaviour rather than just a
// rendering preference. Both paths are worth demonstrating — an
// integrator evaluating helexa wants to see the reasoning path and the
// clean-output path that every vanilla OpenAI client gets.
const [showThinking, setShowThinking] = useState<boolean>(() => {
try {
return window.localStorage.getItem(THINKING_PREF_KEY) === "1";
} catch {
// Private windows and blocked site data throw on access, not on
// read — default off rather than taking the page down with it.
return false;
}
});
useEffect(() => {
try {
window.localStorage.setItem(THINKING_PREF_KEY, showThinking ? "1" : "0");
} catch {
/* preference is a convenience; losing it is not worth an error */
}
}, [showThinking]);
const { streaming, activity, error, send, stop } = useChat({
model,
apiKey: authed ? (chatApiKey ?? undefined) : undefined,
locale: i18n.language,
toolsEnabled: authed || anonWebSearch,
includeThinking: showThinking,
});
const [draft, setDraft] = useState("");
const threadRef = useRef<HTMLDivElement>(null);
@@ -374,6 +402,17 @@ export default function Chat() {
>
{/* Assistant turns are markdown (#194); user turns stay
* literal — people type `*` and `_` as punctuation. */}
{/* The working comes before the answer, because that
* is the order it happened in — and because a reader
* scrolling back wants the conclusion nearest the
* next turn, not buried under its own reasoning. */}
{m.role !== "user" && m.blocks && m.blocks.length > 0 && (
<TurnBlocks
blocks={m.blocks}
streaming={m.status === "streaming"}
answerStarted={m.content.length > 0}
/>
)}
{m.role === "user" ? (
m.content
) : (
@@ -485,6 +524,14 @@ export default function Chat() {
}
}}
/>
<label className="hx-thinking-toggle" title={t("chat:showThinkingHint")}>
<input
type="checkbox"
checked={showThinking}
onChange={(e) => setShowThinking(e.target.checked)}
/>
<span>{t("chat:showThinking")}</span>
</label>
{streaming ? (
<button
type="button"
+5 -5
View File
@@ -7,7 +7,7 @@
# returns and what the router can cold-load on demand.
#
# Field reference:
# id - Repo id in the source registry (e.g. "Qwen/Qwen3.6-27B").
# id - Repo id in the source registry (e.g. "Qwen/Qwen3.8-27B").
# Exact match.
# harness - which engine handles inference (currently "candle").
# quant - quantisation to serve this model with. Either a
@@ -118,7 +118,7 @@
# The example pins to a specific neuron name; adjust or remove the
# pinned_on entry for your own fleet.
[[models]]
id = "Qwen/Qwen3.6-27B"
id = "Qwen/Qwen3.8-27B"
harness = "candle"
vram_mb = 54000
min_devices = 2
@@ -169,13 +169,13 @@ limit.context = 8192
limit.output = 2048
# Helexa registry model — `source` pins this entry to the helexa
# scheme so cortex forwards `helexa:Helexa/Qwen3.6-27B-Uncensored` to
# scheme so cortex forwards `helexa:Helexa/Qwen3.8-27B-Uncensored` to
# neuron's /models/load. Requires the neuron config to declare a
# matching [harness.candle.sources.helexa] entry pointing at the
# helexa registry endpoint (see neuron.example.toml).
#
# [[models]]
# id = "Helexa/Qwen3.6-27B-Uncensored"
# id = "Helexa/Qwen3.8-27B-Uncensored"
# harness = "candle"
# source = "helexa"
# vram_mb = 54000
@@ -194,7 +194,7 @@ limit.output = 2048
# [aliases]
# "helexa/small" = "Qwen/Qwen3-1.7B"
# "helexa/balanced" = "Qwen/Qwen3-8B"
# "helexa/large" = "Qwen/Qwen3.6-27B"
# "helexa/large" = "Qwen/Qwen3.8-27B"
# Text-to-image (#197): Z-Image-Turbo (Apache 2.0). 6B S3-DiT + Qwen3-4B
# text encoder (host CPU by default; see [harness.candle.image]) + VAE.
+26 -46
View File
@@ -18,11 +18,11 @@
# quadbrat = 1x RTX 3060 (12 GB) → small / quantised
# Qwen3-Next single-GPU validation target (Stage 8c). Same architecture
# as Qwen3.6-27B (`model_type = "qwen3_5"`, same layer_types pattern,
# same Gated DeltaNet hyperparams) but small enough to fit on any
# fleet GPU in bf16 (~1.6 GB). Stepping stone — if this answers the
# as the 27B flagship (`model_type = "qwen3_5"`, same layer_types
# pattern, same Gated DeltaNet hyperparams) but small enough to fit on
# any fleet GPU in bf16 (~1.6 GB). Stepping stone — if this answers the
# Tbilisi probe correctly, the qwen3_5 code path is right and TP-2 on
# Qwen3.6-27B is purely a sharding/collective problem.
# the 27B is purely a sharding/collective problem.
[[models]]
id = "Qwen/Qwen3.5-0.8B"
harness = "candle"
@@ -75,49 +75,29 @@ residency_priority = 300
cost.input = 0.40
cost.output = 1.60
# Tensor-parallel target. Beast is the only neuron with two big GPUs.
[[models]]
id = "Qwen/Qwen3.6-27B"
harness = "candle"
# REQUIRED, not cosmetic (#252). Without it a cortex cold-load builds an
# unquantised request: the model loads bf16, leaves ~2 GB free per card
# against a ~3.9 GB prefill requirement, reports `loaded`, and then fails
# every request. Activation from neuron's default_models has always said
# q6k; this is the same fact on the path cortex uses.
quant = "q6k"
# Measured q6k residency across the pair (17.9 + 22.9 GB), not the 51.7 GB
# bf16 checkpoint size. The old 54000 described a model we do not serve.
vram_mb = 42000
min_devices = 2
min_device_vram_mb = 24000
pinned_on = ["beast"]
# The big-node class. Above the everyday class, so image generation can
# never take beast however idle the flagship is; equal to the frontier
# tier, so those two swap freely. `pinned_on` here is affinity only --
# it no longer implies immunity, so this number is what protects it.
residency_priority = 300
# Illustrative pricing (USD per 1M tokens) — top tier, the 27B flagship.
cost.input = 0.30
cost.output = 1.20
# DELIBERATE, TEMPORARY operator override (issue #67): unblocks the opencode
# helexa-provider implementation by giving /v1/models a limit to consume on
# beast. Mirrors the enforced NEURON_MAX_PROMPT_TOKENS=131072 there.
# To be REMOVED once neuron self-derives these — its removal is the
# acceptance test that the computed values replace this hand-set one.
#limit.context = 131072
#limit.input = 122880
#limit.output = 8192
# Qwen3.8-27B (#251). Architecturally identical to the 3.6-27B — same
# model_type, same layer counts, same vision tower; the two configs
# differ only in `transformers_version` — so it serves through the same
# qwen3_5 TP path with the same KV arithmetic.
# Qwen3.8-27B (#251) — the resident flagship, and beast's `default_models`
# pre-warm. Serves through the qwen3_5 TP path.
#
# Same residency class as the 3.6-27B *deliberately*: equal rank is what
# lets the two displace each other in BOTH directions, so whichever is
# asked for takes beast and the other can take it back. A higher number
# here would let this one claim the node permanently and make the
# comparison between them impossible.
# It replaced Qwen3.6-27B, which was retired from this catalogue on
# 2026-08-27 after the A/B it existed for had run its course. The two were
# architecturally identical (same model_type, layer counts and vision
# tower; the configs differed only in `transformers_version`) and were
# deliberately given equal `residency_priority` so they could displace
# each other in both directions.
#
# That symmetry is exactly what made keeping both expensive. The router's
# `helexa/balanced` alias still pointed at the 3.6, so every
# authenticated web-chat turn evicted the resident 3.8 and cold-loaded the
# 3.6 — a ~70 s TP layer load each way — and the next request for the
# flagship swapped it straight back. On 2026-08-27 four of those swaps
# overlapped and left beast holding 32 GB on one card with no model
# registered and nothing serving.
#
# The lesson is not "don't make models swappable": mutual displacement is
# the right design and #99 wants it. It is that two near-identical models
# at equal rank need a *reason* to both be present, and once the
# comparison is over one of them is just a way for a stale alias to take
# the node down.
[[models]]
id = "Qwen/Qwen3.8-27B"
harness = "candle"
+145
View File
@@ -0,0 +1,145 @@
#!/usr/bin/env python3
"""Fail when a product-tier alias names a model the catalogue does not define.
Two files describe one routing decision:
* `models.toml` — the catalogue cortex serves from, plus its own
`[aliases]` table.
* `helexa-router.toml` — the public edge's `[aliases]` table, which is
what an unauthenticated or web-chat caller
actually resolves against.
Nothing forced them to agree, and on 2026-08-27 they did not: the router
still mapped `helexa/balanced` to `Qwen/Qwen3.6-27B` after that model had
been retired. Because both 27Bs sat at equal `residency_priority` — a
deliberate choice so they could displace each other for an A/B — every
authenticated web-chat turn evicted beast's resident flagship and
cold-loaded the retired one, ~70 s of TP layer loading each way. Four of
those overlapped and left the node holding 32 GB on one card with no
model registered and nothing serving.
The failure needed no bad code, only two files and no check. This is that
check. It is the same shape as `check-config-consistency.py` (#252), which
exists because `models.toml` and a neuron config could disagree about
`quant`; that pairing now has a guard and this one did not.
Deliberately narrow: it asks only whether an alias target exists in the
catalogue. It does not judge *which* model a tier should point at — that
is a product decision, and a check that guessed at it would be wrong more
often than the operator.
Exit status is 0 when every alias resolves, 1 when any does not, 2 when a
file is missing or unparseable.
"""
from __future__ import annotations
import argparse
import difflib
import os
import sys
import tomllib
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
def load(path: str) -> dict:
try:
with open(path, "rb") as fh:
return tomllib.load(fh)
except FileNotFoundError:
print(f"error: {path} not found", file=sys.stderr)
raise SystemExit(2)
except tomllib.TOMLDecodeError as exc:
print(f"error: {path} is not valid TOML: {exc}", file=sys.stderr)
raise SystemExit(2)
def catalogue_ids(path: str) -> set[str]:
"""Every model id `models.toml` defines."""
return {m["id"] for m in load(path).get("models", []) if "id" in m}
def alias_table(path: str) -> dict[str, str]:
"""The `[aliases]` table, or an empty one if the file defines none."""
table = load(path).get("aliases", {})
return {k: v for k, v in table.items() if isinstance(v, str)}
def label(path: str) -> str:
"""Repo-relative when inside the tree, basename otherwise.
Deploy runs pass paths from outside the checkout, and a `../../..`
prefix makes a CI failure harder to read than the bare name.
"""
rel = os.path.relpath(path, REPO)
return rel if not rel.startswith("..") else os.path.basename(path)
def check(alias_path: str, catalogue_path: str, ids: set[str]) -> list[str]:
"""One human-readable line per alias that does not resolve."""
problems = []
for alias, target in sorted(alias_table(alias_path).items()):
if target in ids:
continue
# Name the likely replacement. A retired model is normally
# superseded by one with a near-identical id (3.6-27B -> 3.8-27B),
# so a close match turns a puzzle into an edit. Substring matching
# was tried first and listed every Qwen in the catalogue, which is
# noise dressed as help — difflib ranks by actual similarity and a
# 0.6 cutoff keeps it to genuine neighbours.
near = difflib.get_close_matches(target, sorted(ids), n=2, cutoff=0.6)
hint = f"\n did you mean: {', '.join(near)}" if near else ""
problems.append(
f'{label(alias_path)}: "{alias}" -> "{target}"\n'
f" no such model in {label(catalogue_path)}{hint}"
)
return problems
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument(
"--models",
default=os.path.join(REPO, "models.toml"),
help="path to models.toml (default: repo root)",
)
ap.add_argument(
"--aliases",
nargs="*",
default=[os.path.join(REPO, "helexa-router.toml"), os.path.join(REPO, "models.toml")],
help="config(s) carrying an [aliases] table (default: router config + models.toml)",
)
args = ap.parse_args()
ids = catalogue_ids(args.models)
if not ids:
print(f"error: {label(args.models)} defines no models", file=sys.stderr)
return 2
problems = []
checked = 0
for path in args.aliases:
table = alias_table(path)
checked += len(table)
problems.extend(check(path, args.models, ids))
if problems:
print("alias targets missing from the catalogue:\n", file=sys.stderr)
for p in problems:
print(f" {p}\n", file=sys.stderr)
print(
"An alias pointing at a model the catalogue does not define does not\n"
"fail loudly — cortex cold-loads it, which on a busy node means\n"
"evicting whatever is resident. Point it at a served model, or\n"
"restore the model to models.toml.",
file=sys.stderr,
)
return 1
print(f"alias targets OK: {checked} alias(es) resolve to catalogued models")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+25 -1
View File
@@ -322,7 +322,27 @@ if ! ssh "${cortex_host}" "sudo test -f '${int_cert}'"; then
fi
# Install the vhost + renewal timer once the cert exists.
#
# The rate-limit drop-in goes FIRST and unconditionally. It defines
# http-context names the vhost references — the `limit_req_zone`s and the
# `helexa_noquery` log format — and this host was getting the vhost
# without it: helexa-ratelimit.conf was shipped to the web and angels
# hosts only, so `helexa.internal.conf` could arrive referencing a log
# format nginx had never seen.
#
# The ordering below makes that a live hazard rather than a caught
# mistake: the symlink is created before `nginx -t` runs, so a failing
# test leaves a broken config wired into sites-enabled. The running nginx
# is unaffected (it is not reloaded), but the NEXT reload or reboot —
# by anything, including certificate renewal — fails on it.
#
# deploy.yml's deploy-nginx job ships the pair in this same order for the
# same reason.
if ssh "${cortex_host}" "sudo test -f '${int_cert}'"; then
rsync --archive --compress --chown root:root --chmod 0644 --rsync-path 'sudo rsync' \
"${repo_path}/asset/nginx/helexa-ratelimit.conf" \
"${cortex_host}:/etc/nginx/conf.d/helexa-ratelimit.conf" \
|| echo " failed to install helexa-ratelimit.conf"
if rsync --archive --compress --chown root:root --chmod 0644 --rsync-path 'sudo rsync' \
"${repo_path}/asset/nginx/${int_domain}.conf" \
"${cortex_host}:/etc/nginx/sites-available/${int_domain}.conf"; then
@@ -625,7 +645,11 @@ fi
#
# See the tracking issue; do not close the gap by adding syncs here.
echo "==> ${svc_host}: syncing service configs"
sync_config "${svc_host}" "${repo_path}/helexa-router.toml" /etc/helexa-router/helexa-router.toml
# helexa-router.toml is NOT synced here any more — it is tracked and
# deploy.yml's deploy-gallumbits job owns it (#287). Two writers for one
# file is how the fleet ends up running something no commit describes,
# and this file in particular spent long enough diverging to cost an
# outage. If it needs changing, change it in the repo.
# helexa-upstream.toml holds the session JWT secret, the SMTP submission
# password (no-reply mail) and the database URL.
sync_config "${svc_host}" "${repo_path}/helexa-upstream.toml" \
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env python3
"""Exit 0 when two TOML files carry the same settings, ignoring comments.
A deploy needs to answer two questions about a config it is about to
ship, and they are not the same question:
* has anything changed at all? → decides whether to **sync**
* have the settings changed? → decides whether to **restart**
Byte comparison answers only the first. Conflating them costs requests:
the first run of deploy.yml's router sync bounced helexa-router because
a comment header had changed, dropping whatever federation requests were
in flight. The router reads its aliases at startup, so a comment is not a
reason to interrupt anyone.
Usage:
toml-settings-equal.py <a.toml> <b.toml>
Exit status:
0 the parsed settings are identical (a restart would achieve nothing)
1 they differ, or either file is missing or unparseable
Failing "differ" on an unparseable file is deliberate. Being wrong that
way costs one restart; being wrong the other way leaves a service running
settings nobody deployed, which is the failure this whole area exists to
prevent.
"""
from __future__ import annotations
import sys
import tomllib
def load(path: str) -> dict | None:
try:
with open(path, "rb") as fh:
return tomllib.load(fh)
except (OSError, tomllib.TOMLDecodeError) as exc:
print(f"{path}: {exc}", file=sys.stderr)
return None
def main(argv: list[str]) -> int:
if len(argv) != 3:
print(f"usage: {argv[0]} <a.toml> <b.toml>", file=sys.stderr)
return 1
a, b = load(argv[1]), load(argv[2])
if a is None or b is None:
print("treating as changed: a file could not be parsed", file=sys.stderr)
return 1
if a == b:
print("settings identical")
return 0
# Name the top-level sections that moved. Enough to make a deploy log
# say *why* it restarted something, without dumping values — these
# files are non-secret today, but a log is the wrong place to learn
# that assumption has changed.
moved = sorted(k for k in set(a) | set(b) if a.get(k) != b.get(k))
print(f"settings differ in: {', '.join(moved)}")
return 1
if __name__ == "__main__":
raise SystemExit(main(sys.argv))