Compare commits

..
Author SHA1 Message Date
grenade 4b3bf433bb fix(deploy): restart the router for a settings change, not a comment
CI / Classify changes (push) Successful in 21s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Config consistency (push) Successful in 9s
CI / Format (push) Successful in 14s
CI / Clippy (push) Successful in 2m30s
CI / Test (push) Successful in 11m16s
CI / CUDA type-check (push) Successful in 27m27s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The sync step's first real run restarted helexa-router, and the only
difference was the MANAGED BY CI header I had just added to the tracked
copy. Parsed settings were identical. So as written, editing a comment
drops whatever federation requests are in flight.

Byte equality and settings equality answer different questions. The bytes
decide whether to SYNC; the parsed settings decide whether to RESTART.
The router reads its aliases at startup, so a comment is not a reason to
interrupt anyone.

`script/toml-settings-equal.py` makes the second comparison, as a script
rather than an inline heredoc — a heredoc inside a YAML block scalar
needs its terminator at column 0, which breaks out of the block, and the
first attempt at this did exactly that. Verified on all three cases: the
comment-only diff that caused the restart returns "settings identical";
Wednesday's alias diff returns "settings differ in: aliases"; a missing
or unparseable file returns changed, so being wrong costs one restart
rather than leaving a service on settings nobody deployed.

Needs a scoped `cat` grant to read the deployed copy back. It is
deliberately excluded from the step's fail-hard preflight: without it the
read yields nothing, the comparison says "changed", and the step restarts
on any byte difference — the previous behaviour. A host part-way through
a sudoers rollout degrades rather than fails.
2026-08-29 10:05:28 +03:00
grenade 98df7de7e5 Merge branch 'feat/ci-owns-the-router-config'
build-prerelease / Resolve version stamps + change detection (push) Successful in 22s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m45s
build-prerelease / Build neuron-blackwell (push) Successful in 17m35s
build-prerelease / Build helexa-bench binary (push) Successful in 1m53s
build-prerelease / Build cortex binary (push) Successful in 2m23s
build-prerelease / Build neuron-ampere (push) Successful in 2m41s
build-prerelease / Build helexa-angels binary (push) Successful in 2m48s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m51s
build-prerelease / Test (push) Successful in 9m31s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package cortex RPM (push) Successful in 26s
build-prerelease / Package helexa-upstream RPM (push) Successful in 27s
build-prerelease / Package helexa-bench RPM (push) Successful in 31s
build-prerelease / Package helexa-angels RPM (push) Successful in 28s
build-prerelease / Build helexa-router binary (push) Successful in 1m34s
build-prerelease / Package helexa-router RPM (push) Successful in 20s
build-prerelease / Build neuron-ada (push) Successful in 2m34s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m10s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m10s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 3m56s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 29s
2026-08-29 09:20:31 +03:00
grenade b89ab1fafb feat(config): an operator-owned secrets layer, so CI can own the rest
CI / Classify changes (push) Successful in 18s
CI / Config consistency (push) Successful in 8s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m27s
CI / Format (push) Successful in 12s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 15m51s
CI / CUDA type-check (push) Successful in 21m37s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The remaining #287 Gap 1 files could not be version-controlled because
they mix structure with credentials, and the proposed fix — render them
in CI from Gitea secrets — makes the pipeline a credential store. It also
is not needed: the secrets already exist on the hosts and work. What CI
needs is to own the structure without touching them.

cortex, helexa-upstream and helexa-angels now merge a `secrets.toml`
sibling after the main config and before env overrides. The main file
holds structure and can be deployed by CI; `secrets.toml` holds
credentials, is written only by the operator, and is never read, written
or diffed by the pipeline.

Merging last is the safety property, not an implementation detail: the
operator's value wins, so a deploy can never replace a live credential
with a placeholder. Pinned by `the_operators_value_wins_over_the_deployed_one`.

An absent secrets.toml is not an error — figment treats a missing
`Toml::file` as an empty source — so every existing single-file
deployment keeps working untouched and adoption is per host, per service.
That is the property most worth having: if it regressed, the change would
break the fleet on rollout rather than on adoption, so
`an_absent_secrets_file_is_not_an_error` asserts it directly.

A second TOML layer rather than env vars because
`[[entitlements.keys]]` is an array of tables, which `CORTEX_*`
`__`-split env overrides cannot express. The layer then carries every
secret uniformly instead of splitting the mechanism by field shape.

Docs cover the migration as four steps that are safe to stop between,
since a value present in both files resolves to the operator's.

This is the general form of the lesson from 2026-08-27: config CI cannot
write is config no check can defend. `helexa-router.toml` was excluded
from git for sitting beside secret-bearing files, and its stale alias
took beast down. The fix is to stop mixing the two, not to hand the
pipeline the credentials.

Refs #287.
2026-08-29 08:55:06 +03:00
grenade 2956e3843c feat(deploy): CI owns the router config, and aliases must name real models
CI / Test (push) Blocked by required conditions
CI / CUDA type-check (push) Blocked by required conditions
CI / Classify changes (push) Successful in 17s
CI / Config consistency (push) Successful in 9s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m31s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
`helexa-router.toml` was gitignored beside the secret-bearing service
configs, and it holds no secrets: a listen address, poll interval, region,
the product-tier aliases, and a mesh endpoint. It was excluded by
association.

That exclusion cost an outage. On 2026-08-27 the router still aliased
`helexa/balanced` to Qwen/Qwen3.6-27B after that model had been retired
from the catalogue. Both 27Bs sat at equal `residency_priority` — the
right design, so an A/B could displace either way — so every
authenticated web-chat turn evicted beast's resident flagship and
cold-loaded the retired one, ~70 s of TP layer loading each way. Four
overlapped and left the node holding 32 GB on one card with no model
registered and nothing serving, while `activation.state` said "ready".

No bad code was involved. Two files described one routing decision and
nothing could diff them, because one of them was not in git.

So:

- `helexa-router.toml` is tracked and carries the MANAGED BY CI banner.
- `deploy-gallumbits` syncs it, digest-gated, restarting the router only
  on change (it reads aliases at startup; a needless restart drops
  in-flight federation requests). The step sits after the package work
  and preflights its own sudoers grants, per the two ways that went
  wrong this week.
- `script/infra-setup.sh` no longer writes it. Two writers for one file
  is how a fleet ends up running something no commit describes.
- `asset/sudoers.d/gallumbits-host.conf` gains three scoped grants. That
  host had no rsync grant at all, which is why #287 lists this as
  blocked; the grants must be installed by a hand-run infra-setup before
  the step can pass, and it says so when they are not.
- `script/check-alias-targets.py` fails CI when any alias names a model
  the catalogue does not define, in both ci.yml and deploy.yml. Verified
  against Wednesday's exact config: it fails, and names Qwen/Qwen3.8-27B
  as the intended target.

The check is deliberately narrow — it asks whether a target exists, never
which model a tier ought to point at. That is a product decision, and a
check that guessed would be wrong more often than the operator.

The genuinely secret-bearing configs (cortex.toml, helexa-upstream.toml,
helexa-angels.toml) are untouched and still hand-synced. They need
rendering from Gitea secrets, which is the rest of #287 Gap 1.

Refs #287.
2026-08-29 08:47:26 +03:00
grenade c43f4bc74f Merge branch 'fix/infra-setup-ships-ratelimit-to-gateway'
build-prerelease / Resolve version stamps + change detection (push) Successful in 19s
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-08-29 08:42:03 +03:00
grenade cd0211a864 Merge branch 'fix/nginx-preflight-checks-every-grant' 2026-08-29 08:42:03 +03:00
grenade 65011311f2 fix(infra-setup): ship helexa-ratelimit.conf to the gateway, before the vhost
CI / Classify changes (push) Successful in 15s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Config consistency (push) Successful in 10s
CI / Format (push) Successful in 9s
CI / CUDA type-check (push) Successful in 22m16s
CI / Clippy (push) Successful in 2m38s
CI / Test (push) Successful in 7m34s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
helexa-ratelimit.conf was installed on the web and angels hosts but never
on the gateway, where helexa.internal.conf lives and references the names
it defines — the limit_req_zones and, since the search-logging change,
the `helexa_noquery` log format. The gateway's copy has no such format:
`grep -c helexa_noquery` returns 0 there against 4 uses in the vhost.

The ordering in this block turns that into a live hazard rather than a
caught mistake. The sites-enabled symlink is created BEFORE `nginx -t`
runs, so a config that fails validation is already wired in. The running
nginx survives (it is never reloaded), but the next reload or reboot —
by anything, including cert renewal via step@bench.timer, enabled three
lines later — fails on it.

Found while about to run this script to install the two sudoers grants a
deploy step needs. It would have pushed a vhost the host could not parse.

deploy.yml's deploy-nginx job already ships the pair in this order for
exactly this reason; this brings the hand-run path in line with it.
2026-08-27 23:31:09 +03:00
grenade 9b0b32a43b Merge branch 'fix/restore-max-in-flight-8'
build-prerelease / Resolve version stamps + change detection (push) Successful in 19s
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build cortex binary (push) Successful in 2m16s
build-prerelease / Package cortex RPM (push) Successful in 25s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m52s
build-prerelease / Test (push) Successful in 9m42s
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Successful in 1m28s
build-prerelease / Package helexa-router RPM (push) Successful in 18s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 14s
2026-08-27 23:27:54 +03:00
grenade bc7f4a301d fix(deploy): the nginx preflight must check every grant, not a representative one
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 11s
CI / CUDA type-check (push) Successful in 22m19s
CI / Clippy (push) Successful in 2m42s
CI / Test (push) Successful in 7m59s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
It tested `sudo -n nginx -t` and nothing else. That grant had existed for
some time, so the check passed on a host whose sudoers predated the two
helexa-ratelimit.conf entries added in the very commit that introduced
the check — and the job then failed on an opaque rsync denial, seconds
after reporting the permissions were fine.

That is worse than having no preflight. A check that names a remedy while
probing the wrong permission actively misleads: the operator reads "the
sudoers are OK" and looks elsewhere.

Verified against the live host, where the split is real:

    GRANTED rsync sites-available:  OK      MISSING rsync ratelimit:      FAIL
    GRANTED ln -sfn:                OK      MISSING sha256sum ratelimit:  FAIL
    GRANTED systemctl reload:       OK

Now probes all seven. sha256sum, `nginx -t` and `systemctl` are safe to
invoke directly; rsync and ln have no harmless form that still matches
their sudoers pattern, so those use `sudo -nl`, which answers whether a
command is permitted without running it.

The failure message also now says that having run script/infra-setup.sh
before is not enough — grants accumulate in that file as deploy steps
grow, and a host keeps whatever revision it last saw. That was the actual
situation here.
2026-08-27 23:27:40 +03:00
grenade 318dc5f4e1 Merge branch 'fix/retire-qwen36-27b'
build-prerelease / Lint (fmt + clippy) (push) Blocked by required conditions
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Build cortex binary (push) Blocked by required conditions
build-prerelease / Build helexa-router binary (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 20s
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been cancelled
build-prerelease / Build neuron-ada (push) Has been cancelled
build-prerelease / Package cortex RPM (push) Has been cancelled
build-prerelease / Package helexa-router RPM (push) Has been cancelled
build-prerelease / Package helexa-tools RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-08-27 23:25:55 +03:00
grenade 21eefe7e45 fix(beast): restore max_in_flight = 8 — it never divided the KV budget
CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 16s
CI / Format (push) Has been skipped
CI / Clippy (push) Has been skipped
CI / Test (push) Has been skipped
CI / CUDA type-check (push) Has been skipped
CI / Web (lint + typecheck + i18n + build) (push) Successful in 1m47s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
I lowered this to 2 earlier today (#291) on the stated grounds that
"max_in_flight is both the admission bound and the batch engine's slot
count, and the KV budget is divided by it at load". The second half is
false, and has been since eca42510 (2026-08-15) made admission reserve
KV per request:

    admission.enter_with_kv(principal,
        kv_reservation_mb(prompt_len, requested_output, profile, cfg))

`kv_budget` is a semaphore of MiB permits; a request takes what its own
prompt needs and waits for bytes when the pool is short — which is what
`kv_max_wait_secs` is for. Nothing in the tree divides that budget by
the slot count, and the engine's `max_slots` is only the batch width.
The docs said both things twelve lines apart: `kv_max_wait_secs` was
already documented as "how long it waits for enough KV budget".

The measurement in #291 stands as data — seven days of spans, 89.6% of
busy time at one request in flight — but two things undercut the
conclusion. It cannot show that eight slots starve anything, because
they do not. And it described traffic that no longer exists: while it
was taken the router's `helexa/balanced` alias pointed web chat at a
second 27B (3b0b3a22), so per-model concurrency was split across two
models busy evicting each other. Both public tiers now resolve here, so
chat concurrency lands on this model.

Also corrects the operating docs, which carried the same claim, and the
prefix-cache trade table, whose "tokens per slot" column had silently
divided the pool by two. It is a shared pool, so the honest column is
what all concurrent requests draw from: 118,000 tokens at budget_mb =
2048, not 59,000 each.

Refs #291, #257.
2026-08-27 23:24:53 +03:00
grenade 3b0b3a2227 fix(catalogue): retire Qwen3.6-27B — a stale alias was evicting the flagship
CI / Config consistency (push) Successful in 8s
CI / Classify changes (push) Successful in 15s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m7s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 10m39s
CI / CUDA type-check (push) Successful in 19m28s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
beast served nothing for ~50 minutes on 2026-08-27 with 32 GB held on one
card, the other empty, `models: []`, and `activation.state: "ready"`.

The driver was the router's alias table. The web chat's authenticated
default is `helexa/balanced` (Chat.tsx:35), and on gallumbits that alias
still resolved to Qwen/Qwen3.6-27B while beast's resident model is the
3.8. The two sat at equal `residency_priority` deliberately, so they
displaced each other in both directions — meaning every authenticated
web-chat turn evicted the flagship and cold-loaded the 3.6 (~70 s of TP
layer loading), and the next flagship request swapped it straight back.
At 21:27–21:33 four of those overlapped and left the leader holding
accumulated shards with nothing registered.

Mutual displacement is not the bug; #99 wants it and the equal ranks were
right for the A/B they were set up for (#251). The bug is that the A/B
was over and the loser stayed catalogued, so a stale alias pointed real
user traffic at a model whose only remaining job was to evict the one
people were actually being served.

Retires the 3.6 from models.toml and repoints every operator-facing
mention at the 3.8: the docs under helexa.ai/content/docs/2-operating/,
the example catalogues and router/acp configs, and the helexa-acp README
(which told readers to set a default_model that would now 404).

Deliberately NOT renamed: doc/benchmarks.md and
crates/neuron/tests/fixtures/numerical/ record measurements and a parity
snapshot actually taken against the 3.6 — rewriting those would falsify
the record. Test fixtures elsewhere keep the old string too; it is an
arbitrary model id in those, and the churn would bury the real diff.

The live alias on gallumbits was corrected and the router restarted
ahead of this commit, since it was the active cause; helexa-router.toml
is gitignored, so that file is still hand-synced (#287).
2026-08-27 22:34:40 +03:00
grenade 2d1a4ebb2f Merge branch 'fix/nginx-sync-must-not-block-cortex-deploy'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
2026-08-27 13:24:03 +03:00
grenade d87327c3a2 fix(deploy): the nginx sync must be its own job, not a step that gates the fleet
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Config consistency (push) Successful in 6s
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 6m43s
CI / CUDA type-check (push) Successful in 20m55s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Run 1284 proved the first attempt wrong, and worse than predicted. The
sync step hard-fails until an operator runs script/infra-setup.sh to
install its sudoers grant; a failed step fails its job; and
`deploy-neurons` has `needs: [deploy-cortex]`. So a missing nginx
permission skipped the rollout of all three neuron flavours. The fleet
sat two commits behind on eb783c3 while the run reported nothing worse
than "deploy-cortex: failure" — the neuron jobs were *skipped*, which
reads like "nothing to do".

Moving the step later within deploy-cortex, as the previous commit did,
does not fix this. The job still ends red and the dependents still skip.
Order was never the problem.

`deploy-nginx` now stands alone with no `needs`, so its failure is a
statement about the vhost and nothing else. It still fails hard —
drift between the repo copy and the live file is a real defect and
should be red — but it can no longer take the fleet with it.

#287 names this trap: adding a sync step before its sudoers grant exists
breaks the deploy. Having written that down and then done it anyway, the
narrower rule is now recorded beside the job: a step whose prerequisite
is installed by hand does not belong in a job that other jobs depend on.

Refs #287.
2026-08-27 13:02:09 +03:00
grenade a17b3acb70 fix(deploy): the nginx sync must not hold the cortex binary hostage
CI / CUDA type-check (push) Blocked by required conditions
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Config consistency (push) Successful in 16s
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m31s
CI / Test (push) Successful in 6m40s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
The step I added in c6c6fc8a sat before the cortex package update, and
it hard-fails when `gitea_ci` lacks the nginx sudoers grant — which is
installed by hand, by `script/infra-setup.sh`, and has not been run yet.
So until an operator runs it, an unrelated permission blocks the gateway
binary from deploying at all.

That is the exact trap #287 names: adding the sync step before its
sudoers grant exists breaks every deploy. Writing it down and then
walking into it the same day is worth the comment now beside the step.

Moved after `Deploy cortex`, with the reasoning recorded so it does not
drift back. The hard failure stays — a job that could not sync the vhost
should be red, because it did not do everything it promised — but it now
fails after the package work rather than instead of it.
2026-08-27 12:36:47 +03:00
grenade c6c6fc8a50 Merge branch 'feat/nginx-search-alias-and-sync'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m12s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 3m6s
build-prerelease / Build neuron-ada (push) Successful in 3m16s
build-prerelease / Test (push) Successful in 9m39s
build-prerelease / Build neuron-blackwell (push) Successful in 17m3s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m2s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m8s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m6s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 26s
2026-08-27 12:33:49 +03:00
grenade 192030dff5 Merge branch 'fix/wire-surface-in-neuron-spans'
build-prerelease / Test (push) Blocked by required conditions
build-prerelease / Build neuron-blackwell (push) Blocked by required conditions
build-prerelease / Resolve version stamps + change detection (push) Successful in 14s
build-prerelease / Lint (fmt + clippy) (push) Successful in 3m1s
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Successful in 2m47s
build-prerelease / Build neuron-ada (push) Has been cancelled
build-prerelease / Package helexa-neuron-ada RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been cancelled
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been cancelled
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been cancelled
2026-08-27 12:26:01 +03:00
grenade 24b8de7f82 Merge branch 'fix/295-ignore-tiny-prefill-samples' 2026-08-27 12:26:01 +03:00
grenade 432f8009ce feat(nginx): accept POST on search, and keep queries out of the logs
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 14s
CI / Format (push) Successful in 8s
CI / Clippy (push) Successful in 2m33s
CI / CUDA type-check (push) Successful in 20m21s
CI / Test (push) Successful in 7m4s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
**Queries were being logged.** The `main` log format records
`$request` — method, path *and query string* — so every search a user
typed was written to /var/log/nginx/access.log beside their IP, for
authenticated and anonymous visitors alike:

    10.6.0.193 - - [27/Aug/2026] "GET /tools/web_search?q=bansko+weather&format=json HTTP/2.0" 200

A `helexa_noquery` format logs `$request_method $uri` instead, so the
operational signal survives — who, when, which endpoint, what status,
how big, how long — and the query does not. Rate-limit debugging and
abuse detection need counts, statuses and addresses, never search terms.

Applied to `/tools/fetch` as well: its `url=` parameter is the page a
user is having read to them, which is as revealing as a search query and
was equally in the clear.

**POST is now accepted** on both search locations. It was GET-only
because the SPA only GETs — least privilege by default rather than a
considered restriction. SearXNG accepts both, some clients POST, and a
POST keeps the query out of any intermediary's request line, which is
the same concern the log format addresses. `client_max_body_size 8k`
bounds it: a search query is small, and without a bound this becomes a
way to push arbitrary bytes at the upstream.

The log format lives in helexa-ratelimit.conf because `log_format` is
http-context only, and that file is already the http-context include for
this feature. That creates a dependency the deploy has to respect: the
vhost now references a name defined there, so the sync ships
helexa-ratelimit.conf *first*. Shipping the vhost alone would fail
`nginx -t` on an unknown log format — caught by the gate, but leaving
the two files out of step.

Both files validated together by installing them into the live paths on
hanzalova, running `nginx -t`, and restoring with a verified diff.
2026-08-27 12:10:48 +03:00
grenade 98ba45cbb4 feat(nginx): serve SearXNG's own path, and deploy the vhost from CI
CI / Test (push) Blocked by required conditions
CI / CUDA type-check (push) Blocked by required conditions
CI / Config consistency (push) Successful in 10s
CI / Classify changes (push) Successful in 20s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 11s
CI / Clippy (push) Successful in 2m38s
CI / Build cortex SRPM (push) Has been cancelled
CI / Build neuron SRPM (push) Has been cancelled
CI / Publish cortex to COPR (push) Has been cancelled
CI / Publish neuron to COPR (push) Has been cancelled
CI / Bump version in source (push) Has been cancelled
Three coupled changes; the first is useless without the second.

**`location = /search`.** `/tools/web_search` is our name for the
SearXNG upstream and a generic SearXNG client cannot guess it — they
take a base URL and append `/search`, because that is what every
instance serves. Measured on the live edge:

    /tools/web_search?q=…&format=json  -> application/json, SearXNG output
    /search?q=…&format=json            -> text/html (the SPA catch-all)

So such a client gets HTTP 200 with an HTML body and fails at JSON
parsing — which sends people looking for a bug in their own parser. A
404 would at least be honest. The new block is an alias, not a laxer
door: same upstream, same `helexa_tools` rate-limit zone, same
`limit_except GET`.

**The deploy now syncs the vhost.** `asset/nginx/*.conf` has been
version-controlled from the start but only ever installed by
`script/infra-setup.sh`, a one-time run-by-hand script — so the repo
copy and the live file could diverge with nothing to detect it. That is
the #252 shape: two sources of truth for one thing and nothing forcing
agreement. Digest-compared, `nginx -t` gates the reload, and a config
nginx rejects leaves the previous one serving.

**sites-enabled becomes a symlink.** On this host it was a *regular
file*, not a link to sites-available — two independent copies, of which
nginx reads one (`conf.d/sites-enabled.conf` includes
`sites-enabled/*.conf` and nothing includes sites-available). An
operator editing "the config" would reasonably edit sites-available,
watch `nginx -t` pass, reload, and see nothing change. The sync
(re)points the link as its mechanism rather than as a separate chore, so
the layout cannot drift back.

Validated by installing the candidate into sites-enabled on the real
host, running `nginx -t`, and restoring — `nginx -t` does not reload, so
the running config was never at risk. Worth noting the first attempt
tested sites-available and proved nothing, because nginx does not
include it; that is the same trap the symlink change exists to close.

**Requires one manual step before the next deploy.** The nginx sudoers
entries are added to `asset/sudoers.d/cortex-host.conf`, which is
installed by `script/infra-setup.sh`. Until that is re-run once, the new
deploy step fails — deliberately, with a preflight that names the remedy
rather than surfacing an opaque rsync permission error.
2026-08-27 12:05:35 +03:00
grenade 522682282c fix(observability): record which API surface a request arrived on
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 13s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 9s
CI / Clippy (push) Successful in 2m33s
CI / Test (push) Successful in 9m12s
CI / CUDA type-check (push) Successful in 28m29s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
neuron translates `/v1/responses` into a chat request at the door
(`openai_responses::request_to_chat`) and every line after that point is
emitted by the shared chat path. So the journal showed

    tp_chat_stream{req_id=… model=…}: chat_completion (stream): done

for every request, whatever the caller used. That is not a missing
detail; it asserts something false. It cost a real investigation into
why a Responses-configured client "was using chat completions" — it was
not, and nothing in neuron's journal could have said so.

The `tp_chat_stream` span even carried a comment claiming the journal
"can be reconstructed regardless of which API surface the client hit",
while recording nothing that would allow it.

Both spans now carry `wire=openai-chat` or `wire=openai-responses`,
threaded from the four handler entry points alongside the `principal`
that already made the same trip. The strings are deliberately the ones
`cortex-gateway`'s `log_inbound` already emits, so `journalctl | grep
openai-responses` spans both services instead of stopping at the
gateway.

A `WireSurface` enum rather than a `&'static str` parameter: with four
call sites feeding two shared inference paths, a bare string is one
copy-paste away from a span that lies in a new way.

The value is on the span, so every line already being emitted gains it —
finish lines with `reasoning_tokens` / `cached_tokens` / `finish_reason`,
rendered-prompt traces, admission rejections. Those are where the useful
per-turn data lives, and none of it could be attributed to a surface
before.

Note the TP call site is `#[cfg(feature = "cuda")]` and therefore
invisible to a local build: the CPU build was green with that one
argument missing. Caught by grepping the call sites rather than trusting
`cargo build`, which is the second time today that gate has mattered.
2026-08-27 11:35:41 +03:00
grenade 176ce47349 fix(context): ignore prefill samples too small to measure throughput
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 16s
CI / Format (push) Successful in 9s
CI / Web (lint + typecheck + i18n + build) (push) Successful in 2m22s
CI / Clippy (push) Successful in 2m37s
CI / Test (push) Successful in 6m50s
CI / CUDA type-check (push) Successful in 28m29s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
A restart collapsed the advertised context to a sixth of what the host
can serve, and stayed there until real traffic happened along. beast,
minutes after a deploy:

    prefill_tok_per_sec=145.16  throughput_ceiling=17419  -> context=17408
    limit: {"context": 17408, "input": 9216, ...}

against a `vram_ceiling` of 182,560 and a hard ceiling of 131,072.

The cause is not a missing guard — the guard exists and is displaced.
`derive_limit` falls back to `bootstrap_prefill_tok_per_sec` (800,
≈96k context) only when the EMA has *no* sample, and the deploy's own
smoke probe supplies one: 19 tokens in ~130 ms, which is almost entirely
fixed per-request cost and reads as 145 tok/s on a host that measures
~1,400. One probe replaced a conservative estimate with a wrong one.

`fold_rate` already intended to prevent this — "No-op for degenerate
inputs so a probe request or a clock blip can't poison the average" —
but only guarded zero tokens and zero elapsed, not the small-prompt case
that actually occurs.

Prefill samples below 512 tokens are now ignored. That excludes the
smoke probe and the short auxiliary calls agent harnesses make (observed
at 19, 143 and 469 tokens) while admitting ordinary prompts: at
~1,400 tok/s a 512-token prefill runs ~370 ms, so fixed overhead is
about a tenth of it rather than all of it.

Fixed here rather than by warming the host on deploy, which would have
made one caller's experience better once and left the next restart to
rediscover it.

Prefill only. `record_decode` has the same shape of flaw — a two-token
completion is equally overhead-dominated — but it feeds the #137 metrics
rather than the advertised limit, so it is left for a change that can be
attributed on its own.

Four tests: the 19-token probe leaves the EMA unset so the bootstrap
value stands, the 143/469-token auxiliary calls likewise, a
representative prefill still records and still tracks a shift, and the
threshold itself counts.

Docs updated: the operating page no longer tells operators to warm a
restarted host by hand.
2026-08-27 10:45:21 +03:00
grenade 00acd283b6 docs(operating): document every operator setting a neuron exposes
build-prerelease / Resolve version stamps + change detection (push) Successful in 16s
build-prerelease / Lint (fmt + clippy) (push) Has been skipped
build-prerelease / Build neuron-blackwell (push) Has been skipped
build-prerelease / Build neuron-ampere (push) Has been skipped
build-prerelease / Build neuron-ada (push) Has been skipped
build-prerelease / Package helexa-neuron-ada RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-ampere RPM (push) Has been skipped
build-prerelease / Package helexa-neuron-blackwell RPM (push) Has been skipped
build-prerelease / Test (push) Has been skipped
build-prerelease / Build cortex binary (push) Has been skipped
build-prerelease / Package cortex RPM (push) Has been skipped
build-prerelease / Build helexa-bench binary (push) Has been skipped
build-prerelease / Build helexa-upstream binary (push) Has been skipped
build-prerelease / Package helexa-bench RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Has been skipped
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Has been skipped
`config.rs` exposed 35 operator-settable keys and 17 appeared nowhere in
the operating docs — including every one this week's defects turned on.
An operator had no way to discover them, and #297 opened because two of
them hid real bugs: `prefix_cache.budget_mb` (the #292 cache plateau)
and `bootstrap_prefill_tok_per_sec` (which exists to prevent #295's
cold-start collapse and is defeated by the first tiny request).

Now 0 of 35 are undocumented.

Written as guidance rather than a key dump. Each lever says what it
controls, what the default is *and why*, and — the part that makes a
reference usable — **what symptom would make you reach for it**.
`budget_mb` is useless as "snapshot byte budget"; it is useful as "if
`cached_tokens` stops growing mid-session, this is why".

New sections in 03-neurons.md:

- **Sampling.** The full precedence chain (request > operator > model >
  built-in), a symptom-to-knob table, and why `presence_penalty` is the
  right lever for a repeating think block while widening `repeat_last_n`
  is the wrong one — the penalty is token-level and source code repeats
  tokens legitimately. Carries the measured finding that 86% of think
  blocks over 60k characters contained verbatim repeated sentences.
- **Reasoning.** `preserve_thinking` framed as the model's own template
  control and an unmeasured experiment rather than a recommendation, and
  `answer_reserve_tokens` with the measurement that motivated it (given
  16,384 tokens the model used 16,384 and returned no answer) plus the
  trade it introduces — a reserve that fires interrupts deliberation and
  the answer tends to state an intention rather than a conclusion.
- **The prefix cache.** Including the counter-intuitive direction: the
  snapshot budget is reserved *before* the KV budget, so raising it too
  far starves KV and shortens the sessions whose snapshots the larger
  cache was meant to hold. Table of the measured 1:1 trade.
- **Image models.** Why the text encoder defaults to CPU, what
  `te_resident` costs on each device, and why `max_dim` rejects before
  admission — a device OOM poisons the worker context, so refusing early
  is the difference between one rejected request and a restart.
- **Where weights come from.** `sources`, `default_source`, `auth_env`
  (an env var name, never a token) and per-source `cache_dir`, with the
  collision that shared caches cause.

05-context-limits.md gains the three settings that shape the derived
limit, and notes that `bootstrap_prefill_tok_per_sec` only applies when
the EMA has no sample at all — so a host that restarted recently should
be warmed before its `/v1/models` is trusted.

Two corrections while in here: the concurrency example advertised
`max_in_flight = 8`, which #291 showed divides the KV budget eight ways
and leaves a single long session unable to use the advertised context;
the example and its guidance now reflect the measured production
setting, with the request-span data behind it.
2026-08-27 10:35:34 +03:00
grenade eb783c345e Merge branch 'fix/300-anthropic-streaming-thinking'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Build helexa-bench binary (push) Successful in 2m8s
build-prerelease / Build neuron-ada (push) Successful in 2m27s
build-prerelease / Build cortex binary (push) Successful in 2m28s
build-prerelease / Build neuron-ampere (push) Successful in 2m33s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m32s
build-prerelease / Package helexa-bench RPM (push) Successful in 24s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m58s
build-prerelease / Test (push) Successful in 7m56s
build-prerelease / Build neuron-blackwell (push) Successful in 21m49s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package helexa-upstream RPM (push) Successful in 24s
build-prerelease / Package cortex RPM (push) Successful in 28s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m9s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m15s
build-prerelease / Build helexa-router binary (push) Successful in 1m29s
build-prerelease / Package helexa-router RPM (push) Successful in 17s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m13s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 28s
2026-08-27 10:04:24 +03:00
grenade 72d6a37986 fix(anthropic): carry thinking on the streaming path too
CI / Config consistency (push) Successful in 11s
CI / Classify changes (push) Successful in 14s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m37s
CI / Test (push) Successful in 13m46s
CI / CUDA type-check (push) Successful in 19m3s
CI / Build cortex SRPM (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
The #300 fix was incomplete and incomplete in the way that mattered:
it fixed `openai_to_anthropic`, the *non-streaming* translation, while
`AnthropicStreamTranslator` — the path Claude Code actually uses —
still dropped reasoning entirely.

`on_chunk` read `delta.content` and `delta.tool_calls` and nothing
else. `anthropic_sse.rs` had zero occurrences of "thinking". So an
Anthropic client asked for extended thinking via
`thinking.budget_tokens` (which `anthropic_to_openai` deliberately
accepts and normalises onto `reasoning.max_tokens`), was billed for the
reasoning tokens, and received a stream containing none of it — every
turn, silently. Merging the non-streaming fix alone would have left
that untouched and looked like a fix.

`reasoning_content` deltas now open a `thinking` content block and emit
`thinking_delta` events, ahead of the text block and closing before it,
which is Anthropic's ordering and the order the model produces them in.

Emitted without a `signature`, for the same reason as the non-streaming
block: Anthropic signs theirs, we cannot mint a valid one, and
inventing a value would be worse than omitting it. If a client rejects
an unsigned block the fix is a flag, not a return to deleting the
model's work.

Also moves two tests from the previous commit inside the `#[cfg(test)]`
module they were meant to be in — they had landed after its closing
brace, at file top level. They ran and passed there, which is exactly
why it went unnoticed.

Test asserts the full shape: a thinking block opens, `thinking_delta`
carries the text, and a following visible-text delta closes it and
opens a separate text block.
2026-08-27 09:43:36 +03:00
grenade 039a353adf Merge branch 'fix/300-return-the-models-reasoning'
build-prerelease / Resolve version stamps + change detection (push) Successful in 16s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m32s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Successful in 1m54s
build-prerelease / Build helexa-bench binary (push) Successful in 2m7s
build-prerelease / Build neuron-ada (push) Successful in 2m21s
build-prerelease / Build neuron-ampere (push) Successful in 2m24s
build-prerelease / Build cortex binary (push) Successful in 2m22s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m45s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Successful in 20s
build-prerelease / Package helexa-bench RPM (push) Successful in 23s
build-prerelease / Package cortex RPM (push) Successful in 1m14s
build-prerelease / Package helexa-upstream RPM (push) Successful in 1m19s
build-prerelease / Test (push) Successful in 16m23s
build-prerelease / Build neuron-blackwell (push) Successful in 21m42s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m4s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m39s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 3m50s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 31s
2026-08-27 09:29:47 +03:00
grenade 4e4ffd9b60 fix(reasoning): return the model's work to the client on every surface
CI / Config consistency (push) Successful in 7s
CI / Classify changes (push) Successful in 14s
CI / Web (lint + typecheck + i18n + build) (push) Has been skipped
CI / Format (push) Successful in 10s
CI / Clippy (push) Successful in 2m28s
CI / Test (push) Successful in 10m5s
CI / CUDA type-check (push) Successful in 27m51s
CI / Build cortex SRPM (push) Has been skipped
CI / Publish cortex to COPR (push) Has been skipped
CI / Build neuron SRPM (push) Has been skipped
CI / Publish neuron to COPR (push) Has been skipped
CI / Bump version in source (push) Has been skipped
Three of five response paths generated reasoning, metered it in `usage`,
and then discarded it. A client that never receives reasoning cannot
replay it, so on the next turn the model faced its own prior turn as a
bare assertion with the thinking that produced it deleted — and paid
for the deletion.

  /v1/chat/completions  streaming      already returned it
  /v1/chat/completions  non-streaming  DISCARDED -> now returned
  /v1/responses         streaming      already returned it
  /v1/responses         non-streaming  DISCARDED -> now returned
  /v1/messages          (Anthropic)    DISCARDED -> now returned

**chat, non-streaming.** `collect_chat_completion` had
`InferenceEvent::ReasoningDelta(_) => {}` — inherited from the path it
replaced, with a doc comment rationalising it. Now accumulated onto
`message.reasoning_content`, the field the streaming path already emits
as `choice.delta.reasoning_content`. Still kept out of `content`: it is
not the assistant's reply.

**responses, non-streaming.** `build_response` emitted a Message item
and nothing else, while the streaming projector beside it built a full
reasoning-item lifecycle. Observed live: a response reporting
`output_tokens_details.reasoning_tokens: 4000` with an empty `output` —
4,000 tokens generated, billed, invisible. Now emits a `reasoning` item
ahead of the message, same shape and ordering as streaming, via a new
`ResponsesOutputItem::Reasoning`.

**Anthropic.** `anthropic_to_openai` deliberately accepts
`thinking.budget_tokens` — the only quantitative reasoning control any
of our wire formats offers (#223) — and `openai_to_anthropic` returned
a bare text block. Taking a parameter and silently discarding what it
buys is the defect. Now emits a `thinking` block ahead of the text.

The Anthropic block carries no `signature`. Anthropic signs theirs and a
client round-tripping one may expect the field; we cannot mint a valid
signature and inventing one would be worse than omitting it. If a real
client rejects an unsigned block the fix is a flag, not a return to
deleting the model's work — recorded in the code so the next person
does not have to rediscover the trade.

Six tests, one pair per surface: reasoning survives, and absent
reasoning leaves the previous shape untouched (absent, not empty). The
lack of exactly these tests is why three paths drifted apart while a
fourth was correct.

Also carried: two byte-exactness regression tests written while chasing
a suspected content-corruption bug — a file read back through a tool
result, and one through the whole Responses->chat->prompt chain, both
asserting `<`, `&`, backticks, `${}`, quotes and blank lines survive
verbatim. The corruption turned out to be elsewhere; the tests are worth
keeping since that path had none.
2026-08-27 08:46:37 +03:00
grenade 7ad9f48cad Merge branch 'feat/operator-repetition-knobs'
build-prerelease / Resolve version stamps + change detection (push) Successful in 13s
build-prerelease / Lint (fmt + clippy) (push) Successful in 2m44s
build-prerelease / Build helexa-angels binary (push) Has been skipped
build-prerelease / Package helexa-angels RPM (push) Has been skipped
build-prerelease / Build helexa-router binary (push) Successful in 1m41s
build-prerelease / Build helexa-bench binary (push) Successful in 1m56s
build-prerelease / Build cortex binary (push) Successful in 2m13s
build-prerelease / Build neuron-ada (push) Successful in 2m30s
build-prerelease / Build neuron-ampere (push) Successful in 2m30s
build-prerelease / Build helexa-upstream binary (push) Successful in 2m30s
build-prerelease / Test (push) Successful in 8m2s
build-prerelease / Build neuron-blackwell (push) Successful in 21m50s
build-prerelease / Build helexa-tools binary (push) Has been skipped
build-prerelease / Package helexa-tools RPM (push) Has been skipped
build-prerelease / Package helexa-router RPM (push) Successful in 20s
build-prerelease / Package helexa-bench RPM (push) Successful in 21s
build-prerelease / Package helexa-upstream RPM (push) Successful in 22s
build-prerelease / Package cortex RPM (push) Successful in 28s
build-prerelease / Package helexa-neuron-ampere RPM (push) Successful in 1m2s
build-prerelease / Package helexa-neuron-ada RPM (push) Successful in 1m9s
build-prerelease / Package helexa-neuron-blackwell RPM (push) Successful in 4m6s
build-prerelease / Publish to rpm.lair.cafe (unstable) (push) Successful in 25s
2026-08-26 21:09:39 +03:00
32 changed files with 1999 additions and 147 deletions
+6
View File
@@ -177,6 +177,12 @@ jobs:
steps:
- uses: actions/checkout@v4
- run: python3 script/check-config-consistency.py
# Two files describe one routing decision and nothing forced them
# to agree. On 2026-08-27 the router still aliased
# `helexa/balanced` to a model retired from the catalogue, so
# every web-chat turn evicted beast's resident flagship to
# cold-load it. Same shape as the check above (#252/#287).
- run: python3 script/check-alias-targets.py
web:
name: Web (lint + typecheck + i18n + build)
+255
View File
@@ -59,6 +59,12 @@ jobs:
steps:
- uses: actions/checkout@v4
- run: python3 script/check-config-consistency.py
# Two files describe one routing decision and nothing forced them
# to agree. On 2026-08-27 the router still aliased
# `helexa/balanced` to a model retired from the catalogue, so
# every web-chat turn evicted beast's resident flagship to
# cold-load it. Same shape as the check above (#252/#287).
- run: python3 script/check-alias-targets.py
deploy-cortex:
needs: [config-check]
@@ -526,6 +532,161 @@ jobs:
# dependency on cortex or the neurons (the sweep loop is version-aware
# and picks up whatever each neuron reports whenever). So it runs
# alongside the cortex→neurons chain rather than after it.
# The vhost sync gets its own job, with NO `needs`, because it is not
# part of deploying the gateway binary and must not be able to stop
# one.
#
# It was briefly a step inside deploy-cortex, and that broke the fleet
# on its first run (2026-08-27, run 1284): the step hard-fails until an
# operator has run script/infra-setup.sh to install its sudoers grant,
# a failed step fails the job, and `deploy-neurons` has
# `needs: [deploy-cortex]` — so a missing nginx permission silently
# skipped the rollout of all three neuron flavours. The fleet stayed
# two commits behind while the run reported only "deploy-cortex:
# failure".
#
# #287 names this ordering trap. The lesson is narrower than "order the
# steps": a step whose prerequisite is installed by hand does not
# belong in a job that something else depends on, because its failure
# is not a statement about that job. Moving it later within
# deploy-cortex would not have helped — the job still ends red and the
# dependents still skip.
#
# Failing hard here is still right: drift between the repo and the live
# file is a real defect, and this job going red says that and nothing
# else.
deploy-nginx:
runs-on: fedora-43
if: >-
${{
github.event_name == 'workflow_dispatch'
|| github.event.workflow_run.conclusion == 'success'
}}
steps:
- uses: actions/checkout@v4
- name: SSH init
run: |
mkdir -p ~/.ssh
echo "${DEPLOY_KEY}" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new \
gitea_ci@hanzalova.internal 'hostname -f'
# nginx vhost config for the internal site. Version-controlled in
# `asset/nginx/` since the beginning, but until now only ever
# installed by `script/infra-setup.sh` — a one-time, run-by-hand
# script. So the repo copy and the live file could diverge with
# nothing to detect it, which is the #252 shape: two sources of
# truth for one thing and nothing forcing agreement.
#
# Writes to sites-available and (re)points sites-enabled at it as
# a symlink. On this host sites-enabled was a *regular file*, not a
# symlink — two independent copies, of which nginx reads one. An
# operator editing "the config" would reasonably edit
# sites-available, watch `nginx -t` pass, reload, and see nothing
# change. Making the link canonical is part of the sync rather
# than a separate chore, so the layout cannot drift back.
#
# `nginx -t` gates the reload: a config that fails validation is
# left un-reloaded with the previous one still serving.
- name: Sync internal nginx vhost
run: |
src="asset/nginx/helexa.internal.conf"
# The sudoers drop-in that permits this is installed by
# script/infra-setup.sh, which is run by hand, so the live copy
# can lag the repo by however long it has been since anyone ran
# it. Check EVERY grant the step needs, not a representative
# one: the first version of this preflight tested `nginx -t`
# alone, which had existed for ages, so it passed and the job
# then died on the two helexa-ratelimit.conf grants added in the
# same commit as the check. A preflight that names a remedy
# while probing the wrong permission is worse than none — it
# tells you the sudoers are fine seconds before they are not.
#
# Each probe runs the command in a harmless form that still
# matches its sudoers pattern (sha256sum reads; nginx -t and
# `systemctl show` do not mutate). rsync and ln have no such
# form, so they are checked with `sudo -nl`, which reports
# whether a command is permitted without running it.
missing=""
probe() { # probe <description> <command...>
local what="$1"; shift
ssh gitea_ci@hanzalova.internal "$@" >/dev/null 2>&1 \
|| missing="${missing}
- ${what}"
}
probe "nginx -t" \
'sudo -n /usr/sbin/nginx -t'
probe "sha256sum /etc/nginx/sites-available/helexa.internal.conf" \
'sudo -n /usr/bin/sha256sum /etc/nginx/sites-available/helexa.internal.conf'
probe "sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf" \
'sudo -n /usr/bin/sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf'
probe "rsync -> /etc/nginx/sites-available/helexa.internal.conf" \
'sudo -nl /usr/bin/rsync -a x /etc/nginx/sites-available/helexa.internal.conf'
probe "rsync -> /etc/nginx/conf.d/helexa-ratelimit.conf" \
'sudo -nl /usr/bin/rsync -a x /etc/nginx/conf.d/helexa-ratelimit.conf'
probe "ln -sfn sites-available -> sites-enabled" \
'sudo -nl /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf /etc/nginx/sites-enabled/helexa.internal.conf'
probe "systemctl reload nginx" \
'sudo -nl /usr/bin/systemctl reload nginx'
if [ -n "${missing}" ]; then
echo "FAIL: gitea_ci is missing nginx sudoers grants on hanzalova:${missing}" >&2
echo "" >&2
echo " Re-run 'script/infra-setup.sh' once from an operator shell to" >&2
echo " install the current asset/sudoers.d/cortex-host.conf. Having run" >&2
echo " it before is not enough — grants are added to that file as deploy" >&2
echo " steps grow, and the host keeps whatever revision it last saw." >&2
exit 1
fi
rl="asset/nginx/helexa-ratelimit.conf"
local_sum=$(sha256sum "${src}" | cut -d' ' -f1)
local_rl=$(sha256sum "${rl}" | cut -d' ' -f1)
remote_sum=$(ssh gitea_ci@hanzalova.internal \
'sudo sha256sum /etc/nginx/sites-available/helexa.internal.conf 2>/dev/null | cut -d" " -f1' || true)
remote_rl=$(ssh gitea_ci@hanzalova.internal \
'sudo sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf 2>/dev/null | cut -d" " -f1' || true)
link_ok=$(ssh gitea_ci@hanzalova.internal \
'test -L /etc/nginx/sites-enabled/helexa.internal.conf && echo yes || echo no')
if [ "${local_sum}" = "${remote_sum}" ] \
&& [ "${local_rl}" = "${remote_rl}" ] \
&& [ "${link_ok}" = "yes" ]; then
echo "nginx config unchanged, sites-enabled is a symlink — nothing to do"
exit 0
fi
# Order matters: the vhost references names defined at http
# context by helexa-ratelimit.conf (`limit_req_zone`s and the
# `helexa_noquery` log format). Ship that first, or a vhost
# referencing a not-yet-installed name fails `nginx -t` and the
# deploy stops with the two files out of step.
echo "syncing nginx rate-limit/log-format: ${remote_rl:-<absent>} -> ${local_rl}"
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
"${rl}" gitea_ci@hanzalova.internal:/etc/nginx/conf.d/helexa-ratelimit.conf
echo "syncing nginx vhost: ${remote_sum:-<absent>} -> ${local_sum} (symlink=${link_ok})"
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
"${src}" gitea_ci@hanzalova.internal:/etc/nginx/sites-available/helexa.internal.conf
ssh gitea_ci@hanzalova.internal 'bash -s' <<'NGINX'
set -eu
sudo /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf \
/etc/nginx/sites-enabled/helexa.internal.conf
if ! sudo /usr/sbin/nginx -t; then
echo "FAIL: nginx -t rejected the synced config; not reloading" >&2
exit 1
fi
sudo /usr/bin/systemctl reload nginx
echo "nginx reloaded"
NGINX
# Roll helexa-router + helexa-upstream onto the federation service host
# (gallumbits). Same manifest-gated dnf pattern as deploy-cortex; the
# two services are independent daemons on one host, so one job handles
@@ -539,6 +700,10 @@ jobs:
|| github.event.workflow_run.conclusion == 'success'
}}
steps:
# Needed since "Sync router config" reads helexa-router.toml out of
# the tree. This job managed without a checkout while it only ran
# package upgrades over ssh.
- uses: actions/checkout@v4
- name: SSH init
run: |
mkdir -p ~/.ssh
@@ -606,6 +771,96 @@ jobs:
done
DEPLOY
# Router config (#287). Tracked because it holds no secrets, and
# synced here because being untracked is what let its
# `helexa/balanced` alias keep pointing at a retired model until
# beast fell over on 2026-08-27.
#
# After the package step, deliberately: this needs a sudoers grant
# installed by the hand-run script/infra-setup.sh, so on a host that
# has not seen the current grants it fails — and a step that fails
# before the package work would hold the service binaries hostage to
# an unrelated permission. deploy-gallumbits has no dependent jobs,
# so the blast radius stops here either way.
#
# Restarts only on a digest change: the router re-reads its aliases
# at startup, and a needless restart drops in-flight federation
# requests.
- name: Sync router config
run: |
missing=""
probe() {
local what="$1"; shift
ssh gitea_ci@gallumbits.kosherinata.internal "$@" >/dev/null 2>&1 \
|| missing="${missing}
- ${what}"
}
probe "sha256sum /etc/helexa-router/helexa-router.toml" \
'sudo -n /usr/bin/sha256sum /etc/helexa-router/helexa-router.toml'
probe "rsync -> /etc/helexa-router/helexa-router.toml" \
'sudo -nl /usr/bin/rsync -a x /etc/helexa-router/helexa-router.toml'
probe "systemctl restart helexa-router.service" \
'sudo -nl /usr/bin/systemctl restart helexa-router.service'
if [ -n "${missing}" ]; then
echo "FAIL: gitea_ci is missing sudoers grants on gallumbits:${missing}" >&2
echo "" >&2
echo " Re-run 'script/infra-setup.sh' once from an operator shell to" >&2
echo " install the current asset/sudoers.d/gallumbits-host.conf." >&2
exit 1
fi
local_sum=$(sha256sum helexa-router.toml | cut -d' ' -f1)
remote_sum=$(ssh gitea_ci@gallumbits.kosherinata.internal \
'sudo -n /usr/bin/sha256sum /etc/helexa-router/helexa-router.toml 2>/dev/null | cut -d" " -f1' || true)
if [ "${local_sum}" = "${remote_sum}" ]; then
echo "router config unchanged (${local_sum})"
exit 0
fi
# Two questions, not one. The bytes decide whether to SYNC; the
# parsed settings decide whether to RESTART. The first run of
# this step proved why they differ: the only change was a
# comment header, and it bounced the router anyway, dropping
# whatever federation requests were in flight. The router reads
# its aliases at startup, so a comment is not a reason to
# interrupt anyone.
#
# Reading the remote copy into CI is safe here precisely because
# this file holds no secrets — the same property that lets it be
# tracked at all.
#
# The `cat` grant is deliberately NOT in the preflight above: if
# it is missing the read yields nothing, the comparison reports
# "changed", and the step restarts on any byte difference. That
# is the old behaviour, not a broken one, so a host mid-way
# through a sudoers rollout degrades instead of failing.
remote_copy="$(mktemp)"
trap 'rm -f "${remote_copy}"' EXIT
ssh gitea_ci@gallumbits.kosherinata.internal \
'sudo -n /usr/bin/cat /etc/helexa-router/helexa-router.toml' \
> "${remote_copy}" 2>/dev/null || true
settings_changed=yes
if python3 script/toml-settings-equal.py "${remote_copy}" helexa-router.toml; then
settings_changed=no
fi
echo "syncing helexa-router.toml: ${remote_sum:-<absent>} -> ${local_sum}"
rsync --archive --compress \
--chown root:root --chmod 0644 \
--rsync-path 'sudo rsync' \
helexa-router.toml \
gitea_ci@gallumbits.kosherinata.internal:/etc/helexa-router/helexa-router.toml
if [ "${settings_changed}" = yes ]; then
ssh gitea_ci@gallumbits.kosherinata.internal \
'sudo -n /usr/bin/systemctl restart helexa-router.service'
echo "helexa-router restarted: settings changed"
else
echo "helexa-router left running: only comments or formatting changed"
fi
- name: Ensure firewalld allows helexa-router + helexa-upstream + helexa-tools + helexa-angels
run: |
ssh gitea_ci@gallumbits.kosherinata.internal '
+14 -1
View File
@@ -8,10 +8,23 @@ helexa.ai/.env.local
*.swo
.idea/
.vscode/
# Secret-bearing service configs. These carry API keys, a JWT secret, DB
# passwords and SMTP credentials, so they are hand-synced by
# script/infra-setup.sh rather than shipped by CI (#287 Gap 1).
#
# helexa-router.toml is deliberately NOT here: it holds no secrets, only
# a listen address, the product-tier aliases and a mesh endpoint. It was
# ignored by association, and that is precisely how its `helexa/balanced`
# alias came to point at a retired model for long enough to take beast
# down (2026-08-27). A file nothing can diff is a file nothing can catch.
# Operator-owned credential layer, merged after the CI-deployed config
# by cortex / helexa-upstream / helexa-angels. Must never be committed.
secrets.toml
**/secrets.toml
cortex.toml
doc/plan/*
/target-cuda/
.claude/
helexa-router.toml
helexa-upstream.toml
helexa-angels.toml
+38 -25
View File
@@ -7,10 +7,10 @@
# ─────────────────────────────────────────────────────────────────────
# neuron.toml for beast.hanzalova.internal
#
# 2x RTX 5090 (32 GB each) — TP-2 capable. Pre-warms Qwen3.6-27B with
# q5k ISQ across both GPUs at activation, matching the validate-neuron
# 2x RTX 5090 (32 GB each) — TP-2 capable. Pre-warms Qwen3.8-27B with
# q6k ISQ across both GPUs at activation, matching the validate-neuron
# invocation: `validate-neuron.sh beast.hanzalova.internal
# Qwen/Qwen3.6-27B q5k 2`.
# Qwen/Qwen3.8-27B q6k 2`.
#
# Synced to /etc/neuron/neuron.toml by script/infra-setup.sh. Edits
# take effect after the next deploy workflow run restarts the service
@@ -29,24 +29,33 @@ name = "candle"
# count. NEURON_BATCHING=0 (systemd drop-in) is the kill switch;
# removing this section reverts to batch-1 serialization.
[harness.candle.admission]
# 2, not 8 (#291). max_in_flight is both the admission bound and the
# batch engine's slot count, and the KV budget is divided by it at load.
# At 8 that gave each slot 18,852 tokens out of 150,816 — against a
# 131,072-token advertised context, which needs 4 GB of KV per card on
# its own and 32 GB across eight slots, on a 31 GB card already holding
# ~21 GB of weights. The advertised context and the slot count were
# incompatible by roughly 7x, and an agentic session ran in a ~19k
# window while seven idle slots held reservations.
# 8. max_in_flight bounds how many requests run concurrently. It does
# NOT partition the KV budget: since #257 (`eca42510`, 2026-08-15) KV is
# a shared pool and each request reserves what its own prompt needs via
# `enter_with_kv(principal, kv_reservation_mb(prompt_len, ...))`. A
# request too big for what is free waits for bytes to come back rather
# than being handed a fixed 1/N slice.
#
# Seven days of paired request spans on this host: 89.6% of busy time
# was ONE request in flight, 93.6% was two or fewer, and utilisation was
# 4.9%. Most of what little concurrency exists is helexa-bench's own
# synthetic bursts. Two slots cover 93.6% of observed load and give each
# ~75k tokens; the batch engine is still worth 1.79x there (5.24x at
# eight is throughput nobody is asking this host for). Anything above
# two queues on admission (#53) with Retry-After rather than being
# refused.
max_in_flight = 2
# This value was briefly set to 2 (#291) on the stated grounds that "the
# KV budget is divided by it at load, so 8 gives each slot 18,852
# tokens". That was true before #257 and false after it, and #291 was
# filed and actioned after that landed. Nothing in the tree divides the
# budget by the slot count — `kv_budget` is a semaphore of MiB permits
# and the engine's `max_slots` is only the batch width.
#
# The concurrency measurement in #291 stands as data (89.6% of busy time
# at one request in flight, 4.9% utilisation over seven days) but the
# conclusion drawn from it does not, and the traffic it described has
# changed: while it was taken, the router's `helexa/balanced` alias
# pointed web chat at a second 27B, so per-model concurrency was split
# across two models that were evicting each other. With both tiers now
# resolving to this model, chat concurrency lands here.
#
# What actually bounds a busy moment is the pool: 3,689 MiB at
# 32 KiB/token/card is ~118k tokens of KV to share. Eight ordinary chat
# turns fit easily; a few very long agentic sessions will queue on bytes,
# which is the correct behaviour and is what #257 built.
max_in_flight = 8
# Snapshot budget for the cross-request prefix cache (#292).
#
@@ -83,11 +92,15 @@ max_in_flight = 2
budget_mb = 2048
[[default_models]]
# Qwen3.8-27B is the resident flagship (#251). Architecturally identical
# to the 3.6-27B it replaced — same model_type, layer counts and vision
# tower — so the load spec is unchanged. The 3.6-27B stays catalogued at
# the same residency class, so it can still be cold-loaded on demand and
# the two displace each other in either direction.
# Qwen3.8-27B is the resident flagship (#251). It replaced the
# 3.6-27B — architecturally identical, same model_type, layer counts and
# vision tower — so the load spec is unchanged.
#
# The 3.6-27B is no longer catalogued (retired 2026-08-27). While both
# were listed at the same residency class they could displace each other
# in either direction, which is what let a stale `helexa/balanced` alias
# evict this model on every authenticated web-chat turn. See models.toml
# for the incident that ended the arrangement.
model_id = "Qwen/Qwen3.8-27B"
harness = "candle"
quant = "q6k"
+21
View File
@@ -24,3 +24,24 @@ limit_req_zone $binary_remote_addr zone=helexa_tools:10m rate=30r/m;
limit_req_zone $binary_remote_addr zone=angels_invite:1m rate=20r/m;
limit_req_zone $binary_remote_addr zone=angels_auth:1m rate=12r/m;
limit_req_status 429;
# Access-log format for the search locations, which must not record what
# people searched for.
#
# The default `main` format logs `$request` — method, full URI *and
# query string*. For a search proxy that means every query a user types
# is written to /var/log/nginx/access.log next to their IP, for
# authenticated and anonymous visitors alike. SearXNG's own web UI
# defaults to POST partly to avoid exactly this.
#
# `$uri` is the path with the query stripped, so the operational signal
# survives — who called, when, which endpoint, what status, how big, how
# long — and the query itself does not. `$request_method` is kept
# explicit since `$uri` drops it.
#
# Rate-limit debugging and abuse detection both work from this: they
# need counts, statuses and client addresses, never the search terms.
log_format helexa_noquery '$remote_addr - $remote_user [$time_local] '
'"$request_method $uri $server_protocol" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $request_time';
+37 -1
View File
@@ -73,6 +73,9 @@ server {
location = /tools/fetch {
limit_req zone=helexa_tools burst=10 nodelay;
limit_except GET { deny all; }
# The `url=` parameter is as revealing as a search query: it is
# the page a user is having read to them. Same treatment.
access_log /var/log/nginx/access.log helexa_noquery;
proxy_pass http://gallumbits.kosherinata.internal:8889/fetch;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
@@ -83,7 +86,40 @@ server {
# federation service host. GET only; the SPA requests format=json.
location = /tools/web_search {
limit_req zone=helexa_tools burst=10 nodelay;
limit_except GET { deny all; }
# GET and POST: SearXNG accepts both, and a client that POSTs
# keeps its query out of any intermediary's request line. Other
# methods have no meaning here.
limit_except GET POST { deny all; }
# A search query is small. Bounding the body stops this becoming
# a way to push arbitrary bytes at the upstream.
client_max_body_size 8k;
# Never log the query — see helexa_noquery in
# conf.d/helexa-ratelimit.conf.
access_log /var/log/nginx/access.log helexa_noquery;
proxy_pass http://gallumbits.kosherinata.internal:8888/search;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_read_timeout 30s;
}
# The same upstream under SearXNG's own path, for clients that speak
# the SearXNG API rather than our app's.
#
# `/tools/web_search` is our name for it, and a generic SearXNG
# client cannot guess it: they take a base URL and append `/search`,
# because that is what every SearXNG instance serves. Without this
# block such a request falls through to `location /` and is answered
# by the SPA — HTTP 200, `text/html`, and a JSON parse error at the
# client. A 404 would at least be honest; 200-with-HTML sends people
# looking for a bug in their own parser.
#
# Same upstream, same rate-limit zone, same GET-only restriction:
# this is an alias, not a second, laxer door.
location = /search {
limit_req zone=helexa_tools burst=10 nodelay;
limit_except GET POST { deny all; }
client_max_body_size 8k;
access_log /var/log/nginx/access.log helexa_noquery;
proxy_pass http://gallumbits.kosherinata.internal:8888/search;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+12
View File
@@ -13,6 +13,18 @@ gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/cortex/models.toml
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /var/www/bench.helexa.ai/
# deploy-website rsyncs the built helexa.ai SPA into the nginx webroot.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /var/www/helexa.ai/
# deploy.yml's "Sync internal nginx vhost" step: write the vhost, make
# sites-enabled a symlink to it, validate, reload. Scoped to the one
# file and the one service — `nginx -t` is included because a reload
# must never happen on a config nginx has rejected.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/nginx/sites-available/helexa.internal.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/nginx/conf.d/helexa-ratelimit.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/nginx/conf.d/helexa-ratelimit.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/nginx/sites-available/helexa.internal.conf
gitea_ci ALL=(root) NOPASSWD: /usr/bin/ln -sfn /etc/nginx/sites-available/helexa.internal.conf /etc/nginx/sites-enabled/helexa.internal.conf
gitea_ci ALL=(root) NOPASSWD: /usr/sbin/nginx -t
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl reload nginx
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start cortex.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop cortex.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now cortex.service
+25 -3
View File
@@ -4,9 +4,17 @@
# rolls out helexa-router (data plane), helexa-upstream (account/budget
# authority), helexa-tools (grounding fetcher) and helexa-angels (the
# confidential investor portal) package upgrades.
# Configs are NOT synced by CI — they carry secrets (DB password, JWT
# secret, operator tokens) and ride in script/infra-setup.sh under the
# operator's own sudo instead.
# The SECRET-BEARING configs are not synced by CI — helexa-upstream.toml
# and helexa-angels.toml carry a DB password, a JWT secret and SMTP
# credentials, and ride in script/infra-setup.sh under the operator's own
# sudo instead (#287 Gap 1, still open).
#
# helexa-router.toml is different and IS synced here: it holds no secrets,
# only a listen address, the product-tier aliases and a mesh endpoint. It
# was excluded by association with its neighbours, which is how its
# `helexa/balanced` alias came to point at a model retired from the
# catalogue and took beast down on 2026-08-27. A config CI cannot write is
# a config no check can defend.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start helexa-router.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop helexa-router.service
@@ -21,6 +29,20 @@ gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl start helexa-angels.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl stop helexa-angels.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl enable --now helexa-angels.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl daemon-reload
# deploy-gallumbits' "Sync router config" step. Scoped to the one file:
# the neighbouring service configs stay operator-only until #287 Gap 1
# gives them a render-from-secrets path.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/rsync * /etc/helexa-router/helexa-router.toml
gitea_ci ALL=(root) NOPASSWD: /usr/bin/sha256sum /etc/helexa-router/helexa-router.toml
# Read-back, so the sync can tell a settings change from a comment
# change and restart only for the former. Safe to grant and safe to read
# into CI because this file holds no secrets — the same property that
# lets it be tracked. If this grant is absent the step degrades to
# restarting on any byte change, which is why it is not in the step's
# fail-hard preflight.
gitea_ci ALL=(root) NOPASSWD: /usr/bin/cat /etc/helexa-router/helexa-router.toml
gitea_ci ALL=(root) NOPASSWD: /usr/bin/systemctl restart helexa-router.service
gitea_ci ALL=(root) NOPASSWD: /usr/bin/dnf install --refresh --allowerasing -y helexa-router
gitea_ci ALL=(root) NOPASSWD: /usr/bin/dnf upgrade --refresh --allowerasing -y helexa-router
gitea_ci ALL=(root) NOPASSWD: /usr/bin/dnf install --refresh --allowerasing -y helexa-upstream
+141
View File
@@ -143,15 +143,48 @@ pub struct NeuronEndpoint {
impl GatewayConfig {
/// Load configuration from a TOML file, with environment variable overrides.
/// Env vars are prefixed with `CORTEX_` and use `__` as a separator.
///
/// A sibling `secrets.toml`, if present, is merged **after** the main
/// file and therefore wins. It exists so the two have different
/// owners: CI writes the main config, and only the operator writes
/// the secrets — see [`secrets_path`].
pub fn load(path: impl AsRef<Path>) -> Result<Self, Box<figment::Error>> {
let path = path.as_ref();
Figment::new()
.merge(Toml::file(path))
.merge(Toml::file(secrets_path(path)))
.merge(Env::prefixed("CORTEX_").split("__"))
.extract()
.map_err(Box::new)
}
}
/// `secrets.toml` beside the main config.
///
/// The split exists to make a class of outage impossible rather than
/// merely unlikely. Config that CI cannot write is config no check can
/// defend: `helexa-router.toml` was excluded from git because it sat
/// beside secret-bearing files, and its `helexa/balanced` alias then
/// pointed at a retired model for long enough to take a node down
/// (2026-08-27). The fix is not to hand CI the credentials — it is to
/// stop mixing the two in one file.
///
/// So `cortex.toml` holds structure and is deployed by CI, while
/// `secrets.toml` holds API keys and the upstream bearer, is written
/// only by the operator, and is never read, written or diffed by the
/// pipeline. Because it merges last it also wins, so a value present in
/// both is the operator's.
///
/// A missing file is not an error — figment treats an absent
/// `Toml::file` as an empty source — so a host that keeps everything in
/// one file, and every existing deployment, keeps working unchanged.
fn secrets_path(config: &Path) -> std::path::PathBuf {
config
.parent()
.unwrap_or_else(|| Path::new("."))
.join("secrets.toml")
}
impl Default for GatewayConfig {
fn default() -> Self {
Self {
@@ -170,3 +203,111 @@ impl Default for GatewayConfig {
}
}
}
#[cfg(test)]
mod secrets_layer_tests {
use super::*;
use std::io::Write;
/// Write `main` (and optionally `secrets.toml`) into a fresh directory
/// under the target dir, and load through the real code path.
fn load_with(dir: &str, main: &str, secrets: Option<&str>) -> GatewayConfig {
// `CARGO_TARGET_TMPDIR` is only set for integration tests, not for
// unit tests inside src/, so derive a unique directory instead.
let base = std::env::temp_dir().join(format!("helexa-cortex-config-{dir}"));
let _ = std::fs::remove_dir_all(&base);
std::fs::create_dir_all(&base).expect("create test dir");
let cfg = base.join("cortex.toml");
write!(
std::fs::File::create(&cfg).expect("create config"),
"{main}"
)
.expect("write");
if let Some(s) = secrets {
let p = base.join("secrets.toml");
write!(std::fs::File::create(&p).expect("create secrets"), "{s}").expect("write");
}
GatewayConfig::load(&cfg).expect("config loads")
}
const MAIN: &str = r#"
[gateway]
listen = "0.0.0.0:31313"
metrics_listen = "0.0.0.0:31314"
[eviction]
strategy = "lru"
defrag_after_cycles = 50
[entitlements]
require_auth = true
[[neurons]]
name = "beast"
endpoint = "http://beast.invalid:13131"
"#;
/// The whole point: a host with no secrets.toml behaves exactly as
/// before. Every current deployment is in this state, so if this
/// regressed the change would take the fleet down on rollout rather
/// than on adoption.
#[test]
fn an_absent_secrets_file_is_not_an_error() {
let cfg = load_with("secrets_absent", MAIN, None);
assert!(cfg.entitlements.require_auth);
assert!(
cfg.entitlements.keys.is_empty(),
"no keys configured anywhere means no keys"
);
}
/// The array-of-tables case that ruled out env-var overrides:
/// `[[entitlements.keys]]` cannot be expressed as `CORTEX_*` env vars,
/// so the second TOML layer is what carries it.
#[test]
fn secrets_file_supplies_the_entitlement_keys() {
let cfg = load_with(
"secrets_keys",
MAIN,
Some(
r#"
[[entitlements.keys]]
key = "sk-test-aaa"
account_id = "acct-a"
key_id = "a"
[[entitlements.keys]]
key = "sk-test-bbb"
account_id = "acct-b"
key_id = "b"
"#,
),
);
assert_eq!(cfg.entitlements.keys.len(), 2);
assert_eq!(cfg.entitlements.keys[0].key, "sk-test-aaa");
assert_eq!(cfg.entitlements.keys[1].account_id, "acct-b");
assert!(
cfg.entitlements.require_auth,
"structure from the CI-owned file must survive the merge"
);
}
/// Ordering is the safety property. If CI ever ships a placeholder in
/// the main file, the operator's real value must still win — otherwise
/// a deploy silently swaps a live credential for a dummy.
#[test]
fn the_operators_value_wins_over_the_deployed_one() {
let cfg = load_with(
"secrets_precedence",
&format!(
"{MAIN}\n[upstream]\nenabled = true\nurl = \"https://example.invalid\"\nbearer = \"PLACEHOLDER\"\n"
),
Some("[upstream]\nbearer = \"real-token\"\n"),
);
assert_eq!(
cfg.upstream.bearer, "real-token",
"secrets.toml merges last, so it wins"
);
assert_eq!(
cfg.upstream.url, "https://example.invalid",
"fields the operator does not override keep the deployed value"
);
}
}
+27
View File
@@ -276,6 +276,15 @@ pub enum ResponsesOutputItem {
#[serde(default = "default_item_status")]
status: String,
},
/// A think block, as its own output item ahead of the message —
/// the same shape the streaming projector emits, so a client sees
/// one contract whether or not it streams (#300).
Reasoning {
id: String,
summary: Vec<ResponsesSummaryPart>,
#[serde(default = "default_item_status")]
status: String,
},
/// Reserved for the day tool-call extraction lands. The wire
/// shape mirrors `ResponsesInputItem::FunctionCall`.
FunctionCall {
@@ -288,6 +297,24 @@ pub enum ResponsesOutputItem {
},
}
/// One `summary_text` part of a reasoning output item.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ResponsesSummaryPart {
/// Always `"summary_text"`.
#[serde(rename = "type")]
pub part_type: String,
pub text: String,
}
impl ResponsesSummaryPart {
pub fn summary_text(text: impl Into<String>) -> Self {
Self {
part_type: "summary_text".into(),
text: text.into(),
}
}
}
fn default_item_status() -> String {
"completed".into()
}
+202 -7
View File
@@ -415,17 +415,26 @@ fn anthropic_content_to_text(content: AnthropicContent) -> String {
pub fn openai_to_anthropic(resp: ChatCompletionResponse) -> MessagesResponse {
let choice = resp.choices.into_iter().next();
let (content_text, stop_reason) = match choice {
let (content_text, reasoning_text, stop_reason) = match choice {
Some(c) => {
let text = match c.message.content {
MessageContent::Text(t) => t,
MessageContent::Parts(parts) => serde_json::to_string(&parts).unwrap_or_default(),
MessageContent::Null => String::new(),
};
// The model's think block, which the harness returns on
// `message.reasoning_content` (#300).
let reasoning = c
.message
.extra
.get("reasoning_content")
.and_then(|v| v.as_str())
.map(str::to_string)
.filter(|t| !t.is_empty());
let stop = c.finish_reason.map(|r| map_stop_reason(&r));
(text, stop)
(text, reasoning, stop)
}
None => (String::new(), None),
None => (String::new(), None, None),
};
let usage = resp.usage.unwrap_or(Usage {
@@ -441,10 +450,32 @@ pub fn openai_to_anthropic(resp: ChatCompletionResponse) -> MessagesResponse {
id: resp.id,
response_type: "message".into(),
role: "assistant".into(),
content: vec![ContentBlock {
block_type: "text".into(),
data: json!({ "text": content_text }),
}],
// A `thinking` block ahead of the text, which is Anthropic's
// own ordering. This path accepted `thinking.budget_tokens` on
// the way in (see `anthropic_to_openai`) and returned nothing on
// the way out: the caller spent the tokens, could not see them,
// and could not replay the model's reasoning next turn (#300).
//
// Emitted without a `signature`. Anthropic signs its thinking
// blocks and a client that round-trips one may expect the field;
// we cannot mint a valid signature and inventing one would be
// worse than omitting it. If a real client rejects an unsigned
// block, the fix is to gate this behind a flag — not to go back
// to silently deleting the model's work.
content: {
let mut blocks = Vec::with_capacity(2);
if let Some(thinking) = reasoning_text {
blocks.push(ContentBlock {
block_type: "thinking".into(),
data: json!({ "thinking": thinking }),
});
}
blocks.push(ContentBlock {
block_type: "text".into(),
data: json!({ "text": content_text }),
});
blocks
},
model: resp.model,
stop_reason,
usage: AnthropicUsage {
@@ -501,6 +532,7 @@ pub struct AnthropicStreamTranslator {
enum OpenBlock {
Text,
ToolUse,
Thinking,
}
impl AnthropicStreamTranslator {
@@ -540,6 +572,32 @@ impl AnthropicStreamTranslator {
}
for choice in &chunk.choices {
// The model's think block. Dropped entirely until #300: this
// translator read `content` and `tool_calls` and nothing
// else, so an Anthropic client asked for extended thinking
// (`thinking.budget_tokens`, which `anthropic_to_openai`
// accepts), was billed for the tokens, and received a stream
// with no thinking in it — every turn, silently. Fixing only
// the non-streaming translation would have missed this,
// because the clients that ask for thinking stream.
if let Some(text) = choice
.delta
.get("reasoning_content")
.and_then(Value::as_str)
&& !text.is_empty()
{
self.ensure_thinking_block(&mut out);
let index = self.open_block.as_ref().map(|(i, _)| *i).unwrap_or(0);
out.push((
"content_block_delta".to_string(),
json!({
"type": "content_block_delta",
"index": index,
"delta": { "type": "thinking_delta", "thinking": text }
}),
));
}
if let Some(text) = choice.delta.get("content").and_then(Value::as_str)
&& !text.is_empty()
{
@@ -653,6 +711,32 @@ impl AnthropicStreamTranslator {
out
}
/// Open a `thinking` block, closing whatever preceded it.
///
/// Emitted without a `signature`. Anthropic signs its thinking
/// blocks; we cannot mint a valid one, and inventing a value would
/// be worse than omitting it. If a client rejects an unsigned block
/// the fix is a flag — not a return to deleting the model's work.
fn ensure_thinking_block(&mut self, out: &mut Vec<(String, Value)>) {
match &self.open_block {
Some((_, OpenBlock::Thinking)) => {}
_ => {
self.close_open_block(out);
let index = self.next_index;
self.next_index += 1;
self.open_block = Some((index, OpenBlock::Thinking));
out.push((
"content_block_start".to_string(),
json!({
"type": "content_block_start",
"index": index,
"content_block": { "type": "thinking", "thinking": "" }
}),
));
}
}
}
fn ensure_text_block(&mut self, out: &mut Vec<(String, Value)>) {
match &self.open_block {
Some((_, OpenBlock::Text)) => {}
@@ -1062,4 +1146,115 @@ mod request_tests {
let tools = openai.extra.get("tools").and_then(Value::as_array).unwrap();
assert_eq!(tools[0]["function"]["name"], "x");
}
/// The Anthropic path must return the model's thinking (#300).
///
/// It accepted `thinking.budget_tokens` on the way in and returned
/// a bare text block on the way out: the caller spent the tokens,
/// never saw them, and could not replay the model's reasoning on
/// the next turn. Taking a parameter and silently discarding what
/// it buys is the defect, not the missing field.
#[test]
fn anthropic_response_carries_a_thinking_block() {
let mut extra = serde_json::Map::new();
extra.insert("reasoning_content".into(), json!("weighing the options"));
let resp = ChatCompletionResponse {
id: "id".into(),
object: "chat.completion".into(),
created: 0,
model: "m".into(),
choices: vec![crate::openai::ChatCompletionChoice {
index: 0,
message: ChatMessage {
role: "assistant".into(),
content: MessageContent::Text("the answer".into()),
extra: Value::Object(extra),
},
finish_reason: Some("stop".into()),
extra: Value::Null,
}],
usage: None,
extra: Value::Null,
};
let a = openai_to_anthropic(resp);
assert_eq!(a.content.len(), 2, "expected thinking + text");
assert_eq!(a.content[0].block_type, "thinking", "thinking comes first");
assert_eq!(a.content[0].data["thinking"], "weighing the options");
assert_eq!(a.content[1].block_type, "text");
assert_eq!(a.content[1].data["text"], "the answer");
}
/// No thinking means the unchanged single text block.
#[test]
fn anthropic_response_without_thinking_is_unchanged() {
let resp = ChatCompletionResponse {
id: "id".into(),
object: "chat.completion".into(),
created: 0,
model: "m".into(),
choices: vec![crate::openai::ChatCompletionChoice {
index: 0,
message: ChatMessage {
role: "assistant".into(),
content: MessageContent::Text("plain".into()),
extra: Value::Object(Default::default()),
},
finish_reason: Some("stop".into()),
extra: Value::Null,
}],
usage: None,
extra: Value::Null,
};
let a = openai_to_anthropic(resp);
assert_eq!(a.content.len(), 1);
assert_eq!(a.content[0].block_type, "text");
}
/// Streaming Anthropic must carry thinking too (#300).
///
/// Fixing only the non-streaming translation would have missed this
/// entirely — and missed it for the clients that matter, since the
/// ones asking for extended thinking stream.
#[test]
fn anthropic_stream_emits_thinking_deltas() {
use crate::openai::{ChatCompletionChunk, ChunkChoice};
let mut t = AnthropicStreamTranslator::new();
let chunk = |delta: Value| ChatCompletionChunk {
id: "c".into(),
object: "chat.completion.chunk".into(),
created: 0,
model: "m".into(),
choices: vec![ChunkChoice {
index: 0,
delta,
finish_reason: None,
extra: Value::Null,
}],
usage: None,
extra: Value::Null,
};
let ev = t.on_chunk(&chunk(json!({ "reasoning_content": "weighing it" })));
let start = ev
.iter()
.find(|(n, _)| n == "content_block_start")
.expect("a block must open");
assert_eq!(start.1["content_block"]["type"], "thinking");
let delta = ev
.iter()
.find(|(n, _)| n == "content_block_delta")
.expect("a delta must follow");
assert_eq!(delta.1["delta"]["type"], "thinking_delta");
assert_eq!(delta.1["delta"]["thinking"], "weighing it");
// Visible text opens a separate block, after thinking closes.
let ev = t.on_chunk(&chunk(json!({ "content": "the answer" })));
let names: Vec<&str> = ev.iter().map(|(n, _)| n.as_str()).collect();
assert!(
names.contains(&"content_block_stop"),
"the thinking block must close first: {names:?}"
);
let start = ev.iter().find(|(n, _)| n == "content_block_start").unwrap();
assert_eq!(start.1["content_block"]["type"], "text");
}
}
+5 -5
View File
@@ -80,7 +80,7 @@ The fastest path: env-var single-endpoint config.
```sh
export HELEXA_ACP_BASE_URL=http://hanzalova.internal:31313/v1
export HELEXA_ACP_MODEL=Qwen/Qwen3.6-27B
export HELEXA_ACP_MODEL=Qwen/Qwen3.8-27B
helexa-acp # speaks ACP over stdin/stdout; not interactive
```
@@ -117,7 +117,7 @@ default_endpoint = "helexa"
name = "helexa"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.6-27B"
default_model = "Qwen/Qwen3.8-27B"
max_tokens = 8192
context_window = 32768
@@ -138,7 +138,7 @@ default_model = "claude-opus-4"
Restart Zed. The model dropdown lists every model from every
configured endpoint with the `endpoint:model` selector
(`helexa:Qwen/Qwen3.6-27B`, `openrouter:anthropic/claude-opus-4`,
(`helexa:Qwen/Qwen3.8-27B`, `openrouter:anthropic/claude-opus-4`,
…). Switch mid-session; the next prompt routes to the new endpoint.
When only one endpoint is configured the prefix is dropped (model
@@ -166,7 +166,7 @@ Copy-pasteable blocks. Mix and match.
name = "helexa"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.6-27B"
default_model = "Qwen/Qwen3.8-27B"
max_tokens = 8192
context_window = 32768
```
@@ -241,7 +241,7 @@ vLLM.
name = "lan"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.6-27B"
default_model = "Qwen/Qwen3.8-27B"
[[endpoints]]
name = "cloud"
+29 -1
View File
@@ -214,11 +214,39 @@ fn default_notify_to() -> String {
}
impl AngelsConfig {
/// A sibling `secrets.toml`, if present, merges after the main file
/// and wins — see [`secrets_path`].
pub fn load(path: impl AsRef<Path>) -> anyhow::Result<Self> {
let path = path.as_ref();
let cfg: Self = Figment::new()
.merge(Toml::file(path.as_ref()))
.merge(Toml::file(path))
.merge(Toml::file(secrets_path(path)))
.merge(Env::prefixed("ANGELS_").split("__"))
.extract()?;
Ok(cfg)
}
}
/// `secrets.toml` beside the main config.
///
/// The main file holds structure and is deployed by CI; this one holds
/// credentials, is written only by the operator, and is never read,
/// written or diffed by the pipeline. Merging it last means the
/// operator's value wins, so a deploy cannot swap a live credential for
/// a placeholder.
///
/// The split exists because config CI cannot write is config no check
/// can defend — `helexa-router.toml` was kept out of git for sitting
/// beside secret-bearing files, and its stale alias then took a node
/// down (2026-08-27). The answer is to separate secrets from structure,
/// not to hand the pipeline the credentials.
///
/// A missing file is not an error: figment treats an absent
/// `Toml::file` as an empty source, so existing single-file deployments
/// keep working untouched.
fn secrets_path(config: &Path) -> std::path::PathBuf {
config
.parent()
.unwrap_or_else(|| Path::new("."))
.join("secrets.toml")
}
+29
View File
@@ -287,15 +287,44 @@ fn default_from_addr() -> String {
impl UpstreamConfig {
/// Load from a TOML file with `UPSTREAM_`-prefixed env overrides
/// (`__` nesting separator).
///
/// A sibling `secrets.toml`, if present, merges after the main file
/// and wins — see [`secrets_path`].
pub fn load(path: impl AsRef<Path>) -> Result<Self, Box<figment::Error>> {
let path = path.as_ref();
Figment::new()
.merge(Toml::file(path))
.merge(Toml::file(secrets_path(path)))
.merge(Env::prefixed("UPSTREAM_").split("__"))
.extract()
.map_err(Box::new)
}
}
/// `secrets.toml` beside the main config.
///
/// The main file holds structure and is deployed by CI; this one holds
/// credentials, is written only by the operator, and is never read,
/// written or diffed by the pipeline. Merging it last means the
/// operator's value wins, so a deploy cannot swap a live credential for
/// a placeholder.
///
/// The split exists because config CI cannot write is config no check
/// can defend — `helexa-router.toml` was kept out of git for sitting
/// beside secret-bearing files, and its stale alias then took a node
/// down (2026-08-27). The answer is to separate secrets from structure,
/// not to hand the pipeline the credentials.
///
/// A missing file is not an error: figment treats an absent
/// `Toml::file` as an empty source, so existing single-file deployments
/// keep working untouched.
fn secrets_path(config: &Path) -> std::path::PathBuf {
config
.parent()
.unwrap_or_else(|| Path::new("."))
.join("secrets.toml")
}
#[cfg(test)]
mod tests {
use super::*;
+32 -9
View File
@@ -2,6 +2,7 @@
use crate::activation::ActivationTracker;
use crate::harness::HarnessRegistry;
use crate::harness::candle::WireSurface;
use crate::harness::candle::{CandleHarness, InferenceError};
use crate::harness::preflight::PreflightError;
use crate::health::HealthCache;
@@ -362,7 +363,7 @@ async fn chat_completions(
if req.stream.unwrap_or(false) {
match candle
.chat_completion_stream_with(req, chat_config, principal)
.chat_completion_stream_with(req, chat_config, principal, WireSurface::Chat)
.await
{
Ok(rx) => {
@@ -381,7 +382,10 @@ async fn chat_completions(
Err(e) => inference_error_response(e),
}
} else {
match candle.chat_completion(req, principal).await {
match candle
.chat_completion(req, principal, WireSurface::Chat)
.await
{
Ok(resp) => Json(resp).into_response(),
Err(e) => inference_error_response(e),
}
@@ -434,7 +438,13 @@ async fn responses(
if stream_requested {
match candle
.responses_stream(chat_req, response_id, message_item_id, principal)
.responses_stream(
chat_req,
response_id,
message_item_id,
principal,
WireSurface::Responses,
)
.await
{
Ok(rx) => {
@@ -458,7 +468,10 @@ async fn responses(
// and translate the result. We don't currently re-tokenise
// to compute usage; the harness returns it via the chat
// response and we pass it through.
match candle.chat_completion(chat_req, principal).await {
match candle
.chat_completion(chat_req, principal, WireSurface::Responses)
.await
{
Ok(chat_resp) => {
// Extract the assistant text (chat completions
// always emits one choice on the candle path).
@@ -486,9 +499,9 @@ async fn responses(
input_tokens: u.prompt_tokens,
output_tokens: u.completion_tokens,
total_tokens: u.prompt_tokens + u.completion_tokens,
// Carry the reasoning sub-count through from the chat
// usage — the non-streaming path now splits off the
// `<think>` span and counts it (see `split_off_reasoning`).
// Carry the reasoning sub-count through from the
// chat usage: the collector counts the `<think>`
// span and returns its text separately.
output_tokens_details: u.completion_tokens_details.as_ref().map(|d| {
OutputTokensDetails {
reasoning_tokens: d.reasoning_tokens,
@@ -502,8 +515,18 @@ async fn responses(
model_id,
message_item_id: mint_message_item_id(),
};
let _ = chat_resp; // make the borrow-checker happy if `text` consumed it
let resp = openai_responses::build_response(&meta, text, finish, usage);
// The model's think block, which the chat collector now
// returns on `message.reasoning_content` (#300). Without
// this the caller is billed for `reasoning_tokens` it
// never sees, and cannot replay the model's own
// reasoning on the next turn.
let reasoning = chat_resp
.choices
.first()
.and_then(|c| c.message.extra.get("reasoning_content"))
.and_then(|v| v.as_str())
.map(str::to_string);
let resp = openai_responses::build_response(&meta, text, finish, usage, reasoning);
Json(resp).into_response()
}
Err(e) => inference_error_response(e),
+74 -9
View File
@@ -669,6 +669,34 @@ fn requested_sampling(request: &ChatCompletionRequest) -> super::sampling::Reque
}
}
/// Which API surface a request arrived on.
///
/// Carried into the tracing span because neuron translates
/// `/v1/responses` into a chat request at the door
/// (`openai_responses::request_to_chat`) and every line after that
/// point — spans, finish lines, rendered prompts — is emitted by the
/// shared chat path. Without this the journal reads as though every
/// client used `/v1/chat/completions`, which is not a missing detail
/// but an actively wrong one: it says something false about the caller.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum WireSurface {
/// `POST /v1/chat/completions`
Chat,
/// `POST /v1/responses`
Responses,
}
impl WireSurface {
/// The value recorded as `wire=` on every span, matching the names
/// cortex already logs via `log_inbound` so the two journals join.
pub fn as_str(self) -> &'static str {
match self {
Self::Chat => "openai-chat",
Self::Responses => "openai-responses",
}
}
}
pub struct LoadedModel {
pub model_id: String,
/// What the model published in `generation_config.json` (#272).
@@ -2879,8 +2907,9 @@ impl CandleHarness {
&self,
request: ChatCompletionRequest,
principal: Option<String>,
wire: WireSurface,
) -> Result<ChatCompletionResponse, InferenceError> {
let stream = self.inference_stream(request, principal).await?;
let stream = self.inference_stream(request, principal, wire).await?;
wire_chat::collect_chat_completion(
stream.events,
stream.id,
@@ -2907,11 +2936,13 @@ impl CandleHarness {
&self,
request: ChatCompletionRequest,
principal: Option<String>,
wire: WireSurface,
) -> Result<mpsc::Receiver<ChatCompletionChunk>, InferenceError> {
self.chat_completion_stream_with(
request,
wire_chat::ChatProjectionConfig::default(),
principal,
wire,
)
.await
}
@@ -2925,8 +2956,9 @@ impl CandleHarness {
request: ChatCompletionRequest,
mut config: wire_chat::ChatProjectionConfig,
principal: Option<String>,
wire: WireSurface,
) -> Result<mpsc::Receiver<ChatCompletionChunk>, InferenceError> {
let stream = self.inference_stream(request, principal).await?;
let stream = self.inference_stream(request, principal, wire).await?;
// Fill in the model's reasoning markers if the caller
// didn't pre-populate them — they're a property of the
// loaded model (which the HTTP handler doesn't reach into
@@ -2954,9 +2986,10 @@ impl CandleHarness {
response_id: String,
message_item_id: String,
principal: Option<String>,
wire: WireSurface,
) -> Result<mpsc::Receiver<crate::wire::openai_responses::ResponseStreamFrame>, InferenceError>
{
let stream = self.inference_stream(request, principal).await?;
let stream = self.inference_stream(request, principal, wire).await?;
let meta = crate::wire::openai_responses::ResponseMeta {
response_id,
created_at: stream.created,
@@ -2978,6 +3011,7 @@ impl CandleHarness {
&self,
request: ChatCompletionRequest,
principal: Option<String>,
wire: WireSurface,
) -> Result<InferenceStream, InferenceError> {
let handle = {
let models = self.models.read().await;
@@ -3002,7 +3036,7 @@ impl CandleHarness {
LoadedHandle::Single(m) => m,
#[cfg(feature = "cuda")]
LoadedHandle::Tp(m) => {
return self.inference_tp_stream(m, request, principal).await;
return self.inference_tp_stream(m, request, principal, wire).await;
}
LoadedHandle::Image(_) => {
return Err(InferenceError::WrongModality {
@@ -3140,7 +3174,12 @@ impl CandleHarness {
// executor so we capture the span explicitly and re-enter it
// inside the closure to keep the req_id on every emitted line.
let req_id = new_req_id();
let span = tracing::info_span!("chat_stream", req_id = %req_id, model = %model_id);
let span = tracing::info_span!(
"chat_stream",
req_id = %req_id,
model = %model_id,
wire = wire.as_str()
);
let prompt_len = prompt_tokens.len();
let req_start = std::time::Instant::now();
// Cloned `Arc<LoadedModel>` so the spawned task can mark the
@@ -4714,6 +4753,7 @@ impl CandleHarness {
tp: Arc<TpLoadedModel>,
request: ChatCompletionRequest,
principal: Option<String>,
wire: WireSurface,
) -> Result<InferenceStream, InferenceError> {
if tp.poisoned.load(Ordering::Acquire) {
return Err(self.trigger_recovery(&request.model).await);
@@ -4865,14 +4905,19 @@ impl CandleHarness {
// of this inference; concurrent requests against the same TP
// model serialise behind it.
//
// Tagged with the same req_id span as the non-streaming path
// so the journal can be reconstructed regardless of which API
// surface the client hit.
// Tagged with the same req_id span as the non-streaming path,
// and with the surface the request actually arrived on. Both
// matter: neuron translates `/v1/responses` into a chat request
// at the door, so without `wire` every line here reads as
// though the caller used `/v1/chat/completions`. The value
// matches what cortex logs via `log_inbound`, so the two
// journals join on it.
let req_id = new_req_id();
let span = tracing::info_span!(
"tp_chat_stream",
req_id = %req_id,
model = %model_id
model = %model_id,
wire = wire.as_str()
);
let req_start = std::time::Instant::now();
let (vram_free_mb, vram_total_mb) = tp.query_vram().await;
@@ -7573,6 +7618,26 @@ mod tests {
assert_eq!(effort_in(&out), None);
}
/// The wire surface must serialise as the string cortex logs, so a
/// journal search joins the two.
///
/// neuron translates `/v1/responses` into a chat request at the
/// door, and every line after that is emitted by the shared chat
/// path. Before `wire` was recorded the journal showed
/// `chat_completion (stream): done` for every request whatever the
/// surface — worse than omitting the detail, because it asserts
/// something false about the caller. It cost a real investigation
/// into why a Responses-configured client "was using chat
/// completions". It was not.
#[test]
fn the_wire_surface_matches_what_cortex_logs() {
// These strings are load-bearing: `cortex-gateway`'s
// `log_inbound` emits the same two, so `journalctl | grep
// openai-responses` spans both services.
assert_eq!(WireSurface::Chat.as_str(), "openai-chat");
assert_eq!(WireSurface::Responses.as_str(), "openai-responses");
}
/// Effort must not select a *rung's* budget — that was the
/// guillotine (#290): the model was told to reason at `xhigh` and
/// cut off at `medium`'s 12,288. The reserve below is not that; it
@@ -438,6 +438,70 @@ mod tests {
assert!(out.contains("<|vision_start|>") && out.contains("<|vision_end|>"));
}
/// A file read back through a tool result must reach the model
/// byte-for-byte.
///
/// The reported symptom: the model writes a file correctly, reads it
/// back, and sees errors that are not in the file — consistently,
/// after every write. Anything that mangles `<`, `&`, `"` or
/// newlines on this path would produce exactly that, and would make
/// the model burn its budget reconciling a file against a corrupted
/// view of itself.
#[test]
fn a_file_read_back_through_a_tool_result_is_byte_exact() {
let template = include_str!("../../tests/data/qwen3.8-27b.chat_template.jinja");
// The characters most likely to be mangled, in the shapes a real
// web project contains them.
let file = "<!DOCTYPE html>\n<html lang=\"en\">\n<head>\n <meta charset=\"utf-8\">\n</head>\n<body>\n <script>\n if (a && b || c < d) { g.fillStyle = '#b8905a'; }\n const s = `rgba(${r},${g},0.5)`;\n\n // a comment & an ampersand\n </script>\n</body>\n</html>\n";
let mut tool_extras = serde_json::Map::new();
tool_extras.insert("tool_call_id".into(), json!("call_0"));
let mut assistant_extras = serde_json::Map::new();
assistant_extras.insert(
"tool_calls".into(),
json!([{"id":"call_0","type":"function",
"function":{"name":"read","arguments":"{\"path\":\"/tmp/index.html\"}"}}]),
);
let messages = vec![
user_msg("read the file back"),
ChatMessage {
role: "assistant".into(),
content: MessageContent::Null,
extra: Value::Object(assistant_extras),
},
ChatMessage {
role: "tool".into(),
content: MessageContent::Text(file.to_string()),
extra: Value::Object(tool_extras),
},
user_msg("is it correct?"),
];
let out =
render_chat_template(template, &messages, &Value::Null, &Value::Null).expect("render");
for probe in [
"<!DOCTYPE html>",
"<html lang=\"en\">",
"if (a && b || c < d)",
"g.fillStyle = '#b8905a';",
"`rgba(${r},${g},0.5)`",
"// a comment & an ampersand",
] {
assert!(
out.contains(probe),
"the model would not see {probe:?} verbatim — rendered prompt mangles it"
);
}
// And nothing HTML-escaped anywhere.
for bad in ["&lt;", "&gt;", "&amp;", "&quot;", "&#x27;", "&#39;"] {
assert!(
!out.contains(bad),
"found {bad:?} in the rendered prompt: content is being HTML-escaped"
);
}
}
/// Blank lines inside a replayed tool-call argument must survive
/// rendering intact.
///
@@ -36,6 +36,28 @@ use crate::config::ContextLimitConfig;
/// few turns.
const RATE_EMA_ALPHA: f64 = 0.3;
/// Smallest prompt whose prefill time says anything about throughput.
///
/// A prefill has a fixed per-request cost — tokenize, launch, sync —
/// that dominates a short prompt completely. The deploy's own smoke
/// probe prefills 19 tokens in ~130 ms and reads as **145 tok/s** on a
/// host that measures ~1,400. Folding that in does not make the
/// estimate slightly worse; it replaces it.
///
/// That mattered because the fallback it displaces is the whole
/// cold-start guard: with no sample at all, `derive_limit` uses
/// `bootstrap_prefill_tok_per_sec` (800, ≈96k context at the default
/// 120 s target). One 19-token probe replaced it with 145 tok/s, and
/// the host advertised **17,408** — a sixth of what it can serve —
/// until real traffic happened to warm it (#295).
///
/// 512 excludes the smoke probe and the short auxiliary calls agent
/// harnesses make (observed at 19, 143 and 469 tokens) while admitting
/// ordinary prompts. At ~1,400 tok/s a 512-token prefill runs ~370 ms,
/// so fixed overhead is roughly a tenth of it — informative rather than
/// dominant.
const MIN_PREFILL_SAMPLE_TOKENS: usize = 512;
/// Fold one throughput sample (`tokens` in `elapsed`) into the EMA held in
/// `bits`. No-op for degenerate inputs so a probe request or a clock blip
/// can't poison the average.
@@ -99,7 +121,15 @@ impl ThroughputEma {
/// Fold one prefill measurement (`prompt_tokens` processed in
/// `elapsed`) into the prefill EMA.
///
/// Samples below [`MIN_PREFILL_SAMPLE_TOKENS`] are ignored: they
/// measure fixed per-request cost rather than throughput, and the
/// value they displace — the bootstrap estimate — is better than
/// they are.
pub fn record(&self, prompt_tokens: usize, elapsed: Duration) {
if prompt_tokens < MIN_PREFILL_SAMPLE_TOKENS {
return;
}
fold_rate(&self.prefill_bits, prompt_tokens, elapsed);
}
@@ -356,6 +386,60 @@ pub fn derive_limit(
#[cfg(test)]
mod tests {
/// The deploy's smoke probe must not set the advertised context.
///
/// 19 tokens in 130 ms reads as ~145 tok/s on a host that measures
/// ~1,400. Before this guard that one sample displaced the
/// bootstrap estimate and the host advertised 17,408 instead of
/// ~96,000 — until real traffic happened along (#295).
#[test]
fn a_smoke_probe_does_not_poison_the_prefill_rate() {
let ema = ThroughputEma::new();
ema.record(19, Duration::from_millis(130));
assert_eq!(
ema.get(),
None,
"a 19-token prefill must leave the EMA unset so the bootstrap value stands"
);
}
/// The short auxiliary calls agent harnesses make are the same
/// shape as the probe and equally uninformative — observed at 143
/// and 469 tokens against prompts three orders of magnitude larger.
#[test]
fn short_auxiliary_prompts_are_ignored_too() {
let ema = ThroughputEma::new();
for tokens in [143usize, 469] {
ema.record(tokens, Duration::from_millis(200));
}
assert_eq!(ema.get(), None);
}
/// A real prompt still records, and still tracks a shift.
#[test]
fn representative_prefills_still_measure() {
let ema = ThroughputEma::new();
ema.record(1400, Duration::from_secs(1));
let first = ema.get().expect("a 1400-token sample must record");
assert!((first - 1400.0).abs() < 1.0, "got {first}");
// A slower host is tracked, not ignored.
for _ in 0..20 {
ema.record(700, Duration::from_secs(1));
}
let after = ema.get().unwrap();
assert!(after < first && after > 600.0, "got {after}");
}
/// Exactly at the threshold counts — the boundary is inclusive so
/// the constant means what it says.
#[test]
fn the_threshold_itself_is_a_valid_sample() {
let ema = ThroughputEma::new();
ema.record(MIN_PREFILL_SAMPLE_TOKENS, Duration::from_secs(1));
assert!(ema.get().is_some());
}
use super::*;
/// The ceiling a client is told to plan against must not be the
+108 -12
View File
@@ -468,11 +468,21 @@ fn usage_chunk(
/// where the same wire object could be assembled differently — the
/// shape of defect #252 and #272 both were.
///
/// Semantics deliberately preserved from the path this replaces:
/// reasoning text is **dropped** from `content` (it is counted in
/// `usage.completion_tokens_details.reasoning_tokens` but never
/// rendered), and `finish_reason` becomes `"tool_calls"` whenever any
/// tool call was parsed.
/// Reasoning is kept out of `content` — it is not the assistant's
/// reply — but it is **returned** on `message.reasoning_content`, the
/// same field the streaming path emits as
/// `choice.delta.reasoning_content`.
///
/// It used to be discarded here (#300). That was inherited from the
/// path this replaced and was wrong on both counts: the tokens are
/// metered in `usage.completion_tokens_details.reasoning_tokens`, so a
/// caller paid for text it could never see; and a client cannot replay
/// what it never received, so on the next turn the model saw its own
/// prior turn as a bare assertion with the reasoning that produced it
/// deleted.
///
/// `finish_reason` becomes `"tool_calls"` whenever any tool call was
/// parsed.
///
/// Two fields that the old path hardcoded to `None` now populate,
/// because the events carry them and the struct already had the slots:
@@ -488,6 +498,7 @@ pub async fn collect_chat_completion(
model_id: String,
) -> Result<ChatCompletionResponse, String> {
let mut content = String::new();
let mut reasoning = String::new();
let mut tool_calls: Vec<serde_json::Value> = Vec::new();
let mut finish: Option<(FinishReason, u32, u32, u32, u32, Option<FinishTiming>)> = None;
@@ -495,8 +506,8 @@ pub async fn collect_chat_completion(
match event {
InferenceEvent::Start => {}
InferenceEvent::TextDelta(t) => content.push_str(&t),
// Counted, never rendered — see the doc comment.
InferenceEvent::ReasoningDelta(_) => {}
// Kept out of `content`, returned on `reasoning_content`.
InferenceEvent::ReasoningDelta(t) => reasoning.push_str(&t),
InferenceEvent::ToolCall {
index,
id: call_id,
@@ -545,11 +556,16 @@ pub async fn collect_chat_completion(
} else {
"tool_calls".to_string()
};
let message_extra = if tool_calls.is_empty() {
serde_json::Value::Object(Default::default())
} else {
json!({ "tool_calls": tool_calls })
};
let mut message_extra = serde_json::Map::new();
if !tool_calls.is_empty() {
message_extra.insert("tool_calls".into(), json!(tool_calls));
}
// Absent rather than empty when the model did not think: a
// non-reasoning model must keep the unchanged message shape.
if !reasoning.is_empty() {
message_extra.insert("reasoning_content".into(), json!(reasoning));
}
let message_extra = serde_json::Value::Object(message_extra);
Ok(ChatCompletionResponse {
id,
@@ -588,6 +604,86 @@ pub async fn collect_chat_completion(
#[cfg(test)]
mod tests {
/// The model's think block must come back on the non-streaming
/// path too (#300).
///
/// It used to be dropped here while the streaming path emitted it
/// as `choice.delta.reasoning_content`. The tokens were metered
/// either way, so a non-streaming caller paid for text it could
/// never see — and, worse, could not replay the model's reasoning
/// on the next turn, leaving the model to face its own prior turn
/// as a bare assertion.
#[tokio::test]
async fn non_streaming_returns_the_models_reasoning() {
let (tx, rx) = mpsc::channel::<InferenceEvent>(8);
tx.send(InferenceEvent::Start).await.unwrap();
tx.send(InferenceEvent::ReasoningDelta(
"weighing the options".into(),
))
.await
.unwrap();
tx.send(InferenceEvent::TextDelta("the answer".into()))
.await
.unwrap();
tx.send(InferenceEvent::Finish {
reason: FinishReason::Stop,
prompt_tokens: 5,
completion_tokens: 7,
reasoning_tokens: 3,
cached_tokens: 0,
timing: None,
})
.await
.unwrap();
drop(tx);
let r = collect_chat_completion(rx, "id".into(), 0, "m".into())
.await
.expect("collect");
let msg = &r.choices[0].message;
assert_eq!(
msg.extra["reasoning_content"], "weighing the options",
"reasoning must be returned, not discarded"
);
// And it must NOT be folded into the visible reply.
assert!(
matches!(&msg.content, MessageContent::Text(t) if t == "the answer"),
"reasoning leaked into content"
);
}
/// A non-reasoning model must keep the unchanged message shape —
/// absent, not an empty string.
#[tokio::test]
async fn no_reasoning_means_no_reasoning_field() {
let (tx, rx) = mpsc::channel::<InferenceEvent>(8);
tx.send(InferenceEvent::Start).await.unwrap();
tx.send(InferenceEvent::TextDelta("plain".into()))
.await
.unwrap();
tx.send(InferenceEvent::Finish {
reason: FinishReason::Stop,
prompt_tokens: 1,
completion_tokens: 1,
reasoning_tokens: 0,
cached_tokens: 0,
timing: None,
})
.await
.unwrap();
drop(tx);
let r = collect_chat_completion(rx, "id".into(), 0, "m".into())
.await
.expect("collect");
assert!(
r.choices[0]
.message
.extra
.get("reasoning_content")
.is_none()
);
}
use super::*;
// ── Non-streaming collector (#285) ────────────────────────────
+125 -10
View File
@@ -1249,28 +1249,49 @@ fn finish_to_status(reason: FinishReason) -> &'static str {
/// Collect a chat-completions response into a non-streaming
/// [`ResponsesResponse`]. Used by the `/v1/responses` handler when
/// the request doesn't set `stream: true`.
///
/// `reasoning` is the model's think block, when it produced one. It
/// becomes its own output item ahead of the message — the same shape
/// and ordering the streaming projector emits. Omitting it (as this did
/// until #300) meant a non-streaming caller was billed for
/// `reasoning_tokens` it could never see, and could not replay the
/// model's own thinking on the next turn.
pub fn build_response(
meta: &ResponseMeta,
full_text: String,
reason: FinishReason,
usage: Option<ResponsesUsage>,
reasoning: Option<String>,
) -> ResponsesResponse {
let status = finish_to_status(reason).to_string();
let mut output: Vec<ResponsesOutputItem> = Vec::with_capacity(2);
if let Some(text) = reasoning.filter(|t| !t.is_empty()) {
output.push(ResponsesOutputItem::Reasoning {
id: reasoning_item_id(meta),
summary: vec![cortex_core::responses::ResponsesSummaryPart::summary_text(
text,
)],
status: "completed".into(),
});
}
ResponsesResponse {
id: meta.response_id.clone(),
object: "response".into(),
created_at: meta.created_at,
status: status.clone(),
model: meta.model_id.clone(),
output: vec![ResponsesOutputItem::Message {
id: meta.message_item_id.clone(),
role: "assistant".into(),
content: vec![ResponsesOutputContent::OutputText {
text: full_text,
annotations: vec![],
}],
status,
}],
output: {
output.push(ResponsesOutputItem::Message {
id: meta.message_item_id.clone(),
role: "assistant".into(),
content: vec![ResponsesOutputContent::OutputText {
text: full_text,
annotations: vec![],
}],
status,
});
output
},
// Same contract as the streaming shell: a truncated response
// says so, or the caller cannot tell it from a fault.
incomplete_details: matches!(reason, FinishReason::Length)
@@ -1284,6 +1305,99 @@ mod tests {
use super::*;
use cortex_core::openai::MessageContent;
/// The non-streaming Responses body must carry a reasoning item
/// (#300), matching the item the streaming projector emits.
///
/// Observed before the fix: a response reporting
/// `output_tokens_details.reasoning_tokens: 4000` with an empty
/// `output` — 4,000 tokens generated, metered, and invisible.
#[test]
fn non_streaming_response_carries_a_reasoning_item() {
let r = build_response(
&meta(),
"the answer".into(),
FinishReason::Stop,
None,
Some("weighing the options".into()),
);
let v = serde_json::to_value(&r).unwrap();
let items = v["output"].as_array().expect("output array");
assert_eq!(items.len(), 2, "expected a reasoning item and a message");
assert_eq!(items[0]["type"], "reasoning", "reasoning comes first");
assert_eq!(items[0]["summary"][0]["type"], "summary_text");
assert_eq!(items[0]["summary"][0]["text"], "weighing the options");
assert_eq!(items[1]["type"], "message");
assert_eq!(items[1]["content"][0]["text"], "the answer");
}
/// A model that did not think produces no reasoning item — the
/// unchanged single-item shape.
#[test]
fn no_reasoning_means_no_reasoning_item() {
let r = build_response(&meta(), "plain".into(), FinishReason::Stop, None, None);
let v = serde_json::to_value(&r).unwrap();
let items = v["output"].as_array().unwrap();
assert_eq!(items.len(), 1);
assert_eq!(items[0]["type"], "message");
}
/// A file read back must survive the *whole* chain: Responses
/// `function_call_output` -> chat message -> rendered prompt.
///
/// The reported symptom is a model that writes a file correctly,
/// reads it back, and consistently sees errors that are not there.
/// This walks the real path rather than a hand-built chat message,
/// because the translation is the part in between.
#[test]
fn a_file_read_back_survives_the_whole_translation() {
let file = "<!DOCTYPE html>\n<html lang=\"en\">\n\n <script>\n if (a && b || c < d) { s = '#b8905a'; }\n </script>\n</html>\n";
let req: ResponsesRequest = serde_json::from_value(serde_json::json!({
"model": "m",
"input": [
{"type": "message", "role": "user",
"content": [{"type": "input_text", "text": "read it back"}]},
{"type": "function_call", "call_id": "c1", "name": "read",
"arguments": "{\"path\":\"/tmp/index.html\"}"},
// pi sends the tool result as a plain string (its
// `convertToolResultOutput` only produces an array when
// images are involved).
{"type": "function_call_output", "call_id": "c1", "output": file},
{"type": "message", "role": "user",
"content": [{"type": "input_text", "text": "any errors?"}]}
]
}))
.unwrap();
let chat = request_to_chat(req).expect("translate");
// The tool message must carry the file verbatim.
let tool = chat
.messages
.iter()
.find(|m| m.role == "tool")
.expect("a tool message");
let got = match &tool.content {
MessageContent::Text(t) => t.clone(),
other => panic!("tool content is not text: {other:?}"),
};
assert_eq!(got, file, "translation altered the file content");
// And it must survive the render too.
let template = include_str!("../../tests/data/qwen3.8-27b.chat_template.jinja");
let out = crate::harness::chat_template::render_chat_template(
template,
&chat.messages,
&Value::Null,
&Value::Null,
)
.expect("render");
for probe in ["<!DOCTYPE html>", "if (a && b || c < d)", "s = '#b8905a';"] {
assert!(out.contains(probe), "rendered prompt lost {probe:?}");
}
for bad in ["&lt;", "&amp;", "&quot;"] {
assert!(!out.contains(bad), "content HTML-escaped: {bad:?}");
}
}
/// One agentic step must render as ONE assistant turn.
///
/// Responses splits a turn into `reasoning` + `message` +
@@ -2670,6 +2784,7 @@ mod tests {
output_tokens_details: None,
input_tokens_details: None,
}),
None,
);
assert_eq!(r.status, "completed");
match &r.output[0] {
@@ -2695,7 +2810,7 @@ mod tests {
#[test]
fn build_response_length_yields_incomplete_status() {
let r = build_response(&meta(), "trunc".into(), FinishReason::Length, None);
let r = build_response(&meta(), "trunc".into(), FinishReason::Length, None, None);
assert_eq!(r.status, "incomplete");
}
}
+1 -1
View File
@@ -35,7 +35,7 @@ default_endpoint = "helexa"
name = "helexa"
base_url = "http://hanzalova.internal:31313/v1"
wire_api = "openai-chat"
default_model = "Qwen/Qwen3.6-27B"
default_model = "Qwen/Qwen3.8-27B"
max_tokens = 8192
# Compaction kicks in when the rolling history grows past this token
# budget. Set to your model's context window. Disable by removing
+1 -1
View File
@@ -23,7 +23,7 @@ listen = "0.0.0.0:8088"
#
# [aliases]
# "helexa/small" = "Qwen/Qwen3-8B"
# "helexa/balanced" = "Qwen/Qwen3.6-27B"
# "helexa/balanced" = "Qwen/Qwen3.8-27B"
# -- Downstream cortexes -------------------------------------------------
# Each [[cortexes]] entry is an operator-run cortex the router may dispatch
+37
View File
@@ -0,0 +1,37 @@
# ─────────────────────────────────────────────────────────────────────
# MANAGED BY CI — .gitea/workflows/deploy.yml is the only writer.
#
# Edits made directly to the deployed copy on gallumbits are reverted by
# the next deploy, and until then the fleet runs something no commit
# describes. Change it here, commit, let the pipeline ship it.
# ─────────────────────────────────────────────────────────────────────
# helexa-router — federation data plane on gallumbits.
#
# Tracked despite living beside the secret-bearing service configs,
# because it holds no secrets: a listen address, the product-tier
# aliases, and a mesh endpoint. Being untracked cost an outage on
# 2026-08-27 — `helexa/balanced` still resolved to Qwen/Qwen3.6-27B
# after that model had been retired from the catalogue, so every
# authenticated web-chat turn evicted beast's resident flagship and
# cold-loaded a model nobody had asked for. Nothing could diff the two
# files, so nothing did.
#
# `script/check-alias-targets.py` now fails CI if any alias below names
# a model that models.toml does not define.
[router]
listen = "0.0.0.0:8088"
poll_interval_secs = 10
region = "eu-central"
# Product tiers → real model ids (#166). The SPA sends the tier names.
[aliases]
"helexa/small" = "Qwen/Qwen3-8B"
"helexa/balanced" = "Qwen/Qwen3.8-27B"
# hanzalova operator cortex, plaintext over the WireGuard mesh (the mesh
# is the trust boundary; no TLS pin needed for http://).
[[cortexes]]
name = "hanzalova"
endpoint = "http://hanzalova.internal:31313"
region = "eu-central"
@@ -44,6 +44,70 @@ change.
VRAM fragmentation has had time to accumulate. It is a signal to restart
that neuron, not an automatic action.
## secrets.toml
Credentials do not belong in the file CI deploys. Put them in a
`secrets.toml` beside the main config and the service will merge it
last, so its values win:
```
/etc/cortex/cortex.toml structure — deployed by CI, safe to diff
/etc/cortex/secrets.toml credentials — yours, root-only, never in git
```
```toml
# /etc/cortex/secrets.toml (root:cortex 0640)
[[entitlements.keys]]
key = "sk-…"
account_id = "acct-a"
key_id = "a"
[upstream]
bearer = "…"
```
`helexa-upstream` and `helexa-angels` read a `secrets.toml` beside their
own configs the same way — the DB URL, JWT secret and SMTP URL belong
there.
**An absent `secrets.toml` is not an error.** A single-file deployment
keeps working exactly as before, so this is opt-in per host and can be
adopted one service at a time.
### Why bother
Because a config CI cannot write is a config no check can defend, and
the cost of that is not hypothetical. `helexa-router.toml` was kept out
of git for sitting beside secret-bearing files. Nothing could diff it
against the catalogue, so its `helexa/balanced` alias went on pointing
at a model that had been retired — and since both models were
deliberately given equal residency priority so they could displace each
other, every authenticated chat turn evicted the resident flagship to
cold-load the retired one. Four of those overlapped and the node ended
up holding 32 GB of GPU memory with no model loaded and nothing
serving.
Splitting secrets from structure is what lets the structure be
version-controlled, reviewed and checked. The ordering matters too: the
operator's file merges last, so a deploy can never replace a live
credential with a placeholder.
### Moving existing secrets
One-time, per host, and reversible — the service reads both files, so
you can move a value and restart before moving the next:
1. Create `secrets.toml` beside the config, owned `root:<service>` mode
`0640` (`0600` where the service runs as root).
2. Move the credential blocks across. Leave everything else in place.
3. Restart the service and confirm it still authenticates — for cortex,
a request with a real key should be billed rather than treated as
anonymous.
4. Delete the moved lines from the main file only once step 3 passes.
Until step 4 the value exists in both files, which is safe: the
operator's copy wins.
## models.toml
The catalogue says what each model needs and where it may run:
@@ -78,7 +142,7 @@ Clients should not hardcode model ids. Aliases let them ask for a role:
[aliases]
"helexa/small" = "Qwen/Qwen3-1.7B"
"helexa/balanced" = "Qwen/Qwen3-8B"
"helexa/large" = "Qwen/Qwen3.6-27B"
"helexa/large" = "Qwen/Qwen3.8-27B"
"helexa/image" = "Tongyi-MAI/Z-Image-Turbo"
```
@@ -58,6 +58,32 @@ Two things that bite:
- **Cold loads are disk-bound.** A 50 GB model off spinning rust is a
very different experience from the same model on NVMe.
### Where weights come from
By default a bare `org/name` id is fetched from HuggingFace. Operators
running a mirror, or serving models from more than one place, can
declare sources explicitly:
```toml
[harness.candle]
default_source = "huggingface"
[harness.candle.sources.huggingface]
endpoint = "https://huggingface.co"
auth_env = "HF_TOKEN" # env var name, never the token itself
cache_dir = "/archive/llm" # optional; per-source cache
```
`auth_env` names an environment variable rather than holding a token, so
credentials stay out of the config file — worth keeping to even when the
file is not obviously shared, because config is exactly what gets copied
between hosts.
Give a source its own `cache_dir` when two sources can serve the same
`org/name`. The cache tree is keyed on the model id alone, so sharing a
directory between sources that disagree about what `org/name` means will
collide on disk.
## Default models
Models listed as defaults load when the daemon activates, so the host
@@ -66,7 +92,7 @@ load cost:
```toml
[[default_models]]
model_id = "Qwen/Qwen3.6-27B"
model_id = "Qwen/Qwen3.8-27B"
harness = "candle"
quant = "q6k"
tensor_parallel = 2
@@ -91,10 +117,17 @@ Each loaded model has bounded admission control:
[harness.candle.admission]
max_in_flight = 8 # concurrent requests actually running
max_queue_depth = 8 # waiting before rejection
max_wait_secs = 30 # how long a queued request waits
max_wait_secs = 30 # how long a queued request waits for a slot
kv_max_wait_secs = 300 # how long it waits for enough KV budget
max_per_principal = 2 # fair share, so one caller cannot monopolise
```
`kv_max_wait_secs` is deliberately much longer than `max_wait_secs`.
The first waits for a slot to turn over; the second waits for a whole
long-context sequence to finish, which on a 27B can be minutes. A
request that needs KV budget nobody has yet is not a request to reject
in thirty seconds.
Over-capacity requests are rejected promptly with `429`/`503` and a
`Retry-After` rather than being allowed to queue invisibly. A fast
rejection a client can retry is far better than a request that hangs
@@ -102,8 +135,34 @@ until something times out.
`max_in_flight` above 1 enables batched decode on architectures that
support it, so concurrent requests share a decode step instead of
serialising. Raise it on big cards, leave it at 1 on small ones — the
KV cache for several concurrent long contexts is itself substantial.
serialising.
**`max_in_flight` does not partition the KV budget.** KV is a shared
pool, and each request reserves what its own prompt actually needs when
it is admitted — which is why `kv_max_wait_secs` above exists at all. A
request larger than what is currently free waits for bytes to come back
rather than being handed a fixed 1/N slice, and a single long session
can legitimately hold most of the pool.
So the slot count and the context window are separate questions. What
bounds a busy moment is the pool: divide `kv_budget_mb` by the model's
KV cost per token (`GET /health` reports both) to see how many tokens
are available to share. Many ordinary chat turns fit at once; a few very
long agentic sessions will queue on bytes.
Size `max_in_flight` against the concurrency you actually want to serve
concurrently, not against context. Anything above it queues on admission
with a `Retry-After` rather than being refused, so setting it too low
converts capacity you have into latency your callers feel. Setting it
very high mainly costs batch-step efficiency, since the engine pads to
the widest active sequence.
Measure before assuming you need a high number — but measure the traffic
you will have, not the traffic you had. On one production host, seven
days of spans showed 89.6% of busy time at a single request in flight;
that host's chat tier was at the time being routed to a *different*
model, and the figure stopped describing it the moment that was
corrected.
`max_per_principal` is what stops one busy client consuming the whole
model.
@@ -150,7 +209,8 @@ reservation is doing its job, not that the model is overloaded.
neuron refuses work it cannot finish rather than crashing partway:
- a **static floor** of free VRAM below which prefill is refused
(`NEURON_MIN_FREE_VRAM_MB`, default 1500)
(`NEURON_MIN_FREE_VRAM_MB`, or `min_free_floor_mb` under
`[harness.candle.context_limit]`; default 1500)
- a **length-aware check** that estimates the KV cache the request will
actually need and refuses if it will not fit
@@ -159,15 +219,207 @@ can pass the static floor and still be impossible, because KV grows with
prompt *and* generation length. See [context
limits](/docs/operating/context-limits) for how the numbers relate.
`activation_headroom_mb` reserves VRAM per card for prefill activations
on top of the weights and the KV cache, before the context ceiling is
computed. Raise it if prefill OOMs on a host whose steady-state
occupancy looks fine — activations are transient and do not show up in
the resident figure.
A host that cannot currently serve a model reports that on `/health`,
and cortex stops routing to it — an unservable replica is not a routing
target.
## Image models
```toml
[harness.candle.image]
te_device = "cpu" # where the text encoder runs
te_resident = true # keep it loaded between generations
max_dim = 2048 # resolution ceiling per side
```
The text encoder is the placement decision. On CPU the ~8 GB encoder
never touches the GPU — prompt encoding is one short forward and the
features are tiny — which is what lets a 24 GB card serve a BF16 DiT at
all. Move it to `cuda` only on a card with room to spare.
`te_resident` trades memory for latency: resident costs ~16 GB of system
RAM on CPU (or ~8 GB of VRAM on CUDA) and saves a few seconds per
request. CPU defaults to resident, CUDA to rebuilding per request, which
are the right defaults for where each is scarce.
`max_dim` rejects oversized requests **before admission**, so an
impossible resolution costs nothing. That matters more than it sounds:
an OOM on the device poisons the worker context, so refusing early is
the difference between one rejected request and a model that needs
restarting.
## Prompt caps
`NEURON_MAX_PROMPT_TOKENS` sets a hard ceiling; larger prompts are
rejected before any GPU work, so a runaway client costs nothing.
## Sampling
Every model publishes its own sampling defaults in
`generation_config.json`, and a caller can override them per request.
Between those two sits an operator override, for the cases where the
model's published defaults are wrong for how you serve it:
```toml
[[default_models]]
model_id = "Qwen/Qwen3.8-27B"
[default_models.sampling]
temperature = 0.6
presence_penalty = 1.0
```
Precedence is **request > operator > model > built-in**. Anything you
leave unset changes nothing, so an empty table behaves exactly as no
table at all.
Set the same values in the gateway's `models.toml` under
`[models.sampling]`. A model that is cold-loaded on demand gets its
sampling from the catalogue, not from this file, so the two must agree
or the same model samples differently depending on how it arrived —
`script/check-config-consistency.py` fails the deploy if they diverge.
### Which knob for which symptom
| symptom | knob |
|---|---|
| Structurally malformed output — truncated JSON, unbalanced braces | `temperature` |
| The think block restates the same passage over and over | `presence_penalty` |
| Common tokens over-represented across a long answer | `frequency_penalty` |
| Short immediate loops — "no, no, no…" | `repeat_penalty` |
| You need two runs to be comparable | `seed` |
**`temperature`** is the blunt one. Qwen3.8-27B publishes `1.0`, which
measured a 20% structural-defect rate on long agentic turns against 0/60
at `0.6`; the fleet therefore overrides it. Note that `0.6` is also what
Qwen recommend for their thinking variants, so this is not fighting the
model.
**`presence_penalty`** and **`frequency_penalty`** are scored over the
*whole* generated sequence. They are the ones that matter for reasoning
models, and Qwen name `presence_penalty` (0–2, 1.5 when severe) as the
remedy for the endless repetition their models fall into during long
thinking. Both default to `0.0`, which is off.
**`repeat_penalty`** (default `1.1`) and **`repeat_last_n`** (default
`64`) are the older, *windowed* form: the penalty only sees the last
`repeat_last_n` tokens. That is fine for chat and close to useless for a
think block — a 20,000-token deliberation restating a passage 5,000
tokens later is entirely outside a 64-token window. Measured across
recorded sessions, **86% of think blocks over 60k characters contained
verbatim repeated sentences**, one of them repeating 76% of its own
content.
Widening the window is the obvious move and the wrong one: the penalty
is token-level, and source code legitimately repeats tokens constantly
(`const`, `function`, brace runs), so a wide window at a meaningful
penalty degrades exactly the output you care about. Reach for
`presence_penalty` first — what recurs is passages, not tokens.
**`seed`** pins the sampler's RNG for every request to this model. It
exists because parameter search needs a control and the clients that
would otherwise pin it often cannot. Leaving it set is a product
decision, not just a testing one: every identical prompt then returns an
identical answer to every caller, which is predictable and cacheable but
removes the variation a user may expect from retrying.
## Reasoning
```toml
[default_models.sampling]
# nothing here — reasoning settings live in their own places, below
```
**`preserve_thinking`** controls whether prior turns keep their think
blocks when the conversation is re-rendered. It is the model's own chat
template control, not ours:
```toml
[[default_models]]
preserve_thinking = false # keep reasoning only for the turn in progress
```
Unset is the default and means the template decides. On Qwen3.8-27B that
is `true` — every prior turn's reasoning is replayed. Full replay is
what makes prompt growth track *total* completion rather than visible
output: one measured session produced 28,672 reasoning tokens on turn 1
and carried all of them into turn 2's 31,269-token prompt, at ~22 s of
prefill on every turn thereafter.
Setting it to `false` is a legitimate experiment, not a recommendation.
Whether narrowing replay *helps* the model is unmeasured. Set it in both
`models.toml` and this file, like sampling, and the value is reported on
`/v1/models` and logged at load so a session can be attributed to the
arm it ran under.
**`reasoning_budget.answer_reserve_tokens`** (default `4096`) holds back
part of the caller's output budget for the answer:
```toml
[harness.candle.reasoning_budget]
answer_reserve_tokens = 4096
```
A reasoning model with no stopping criterion fills whatever budget it is
given. Measured on Qwen3.8-27B: given 16,384 tokens it used 16,384;
given 32,768 it used 32,768 — and both times returned no answer at all.
The reserve bounds the think block at `max_tokens - answer_reserve` so
the rest survives for the reply.
Deliberately *not* derived from reasoning effort. Effort is a prompt
instruction the model is trained to follow; making it also set a
guillotine is how a model came to be told to reason at `xhigh` and then
cut off at `medium`'s budget. Set `0` to disable, restoring unbounded
reasoning.
The trade to know: when the reserve fires, the model is interrupted
mid-deliberation and its answer phase tends to state an *intention*
rather than a conclusion. That is better than returning nothing, but it
is not free.
## The prefix cache
Snapshots of previously-served prefixes, so a follow-up turn re-uses the
KV cache instead of re-prefilling the whole conversation:
```toml
[harness.candle.prefix_cache]
enabled = true
budget_mb = 2048 # snapshot bytes per loaded model
max_entries = 8 # live snapshots per model, regardless of budget
```
**Reach for `budget_mb` when `cached_tokens` stops growing mid-session.**
That is the symptom: the prompt keeps growing, the cached prefix does
not, and every turn re-prefills the difference. Observed on a 1 GiB
default — `cached_tokens` pinned at 2,016 for ten consecutive turns
while the prompt reached 52,869, so ~50k tokens were re-prefilled every turn
at roughly 37 s each, against ~20 s of actual decode.
The number is not free, and the direction of the trade is the opposite
of what it looks like. **The snapshot budget is reserved before the KV
budget, not from what is left over.** On a 2×32 GB host serving a 27B,
measured 1:1 — the token column is the pool all concurrent requests
share, not a per-slot allowance:
| `budget_mb` | KV budget | tokens in the pool | holds a 53k snapshot |
|---|---|---|---|
| 1024 | 4713 MB | 150,800 | no |
| 2048 | 3689 MB | 118,000 | yes |
| 3072 | 2665 MB | 85,300 | yes, but a session cannot reach 53k |
So raising it too far starves the KV cache and shortens how long a
single session can run — the model can no longer reach the context whose
snapshot the larger cache was meant to hold. Size it against the deepest
session you actually serve, and check `kv_budget_mb` in the journal
after a reload.
## Checking on it
```sh
@@ -24,7 +24,7 @@ confined anywhere.
```toml
[[models]]
id = "Qwen/Qwen3.6-27B"
id = "Qwen/Qwen3.8-27B"
harness = "candle"
min_devices = 2
min_device_vram_mb = 24000
@@ -32,6 +32,43 @@ daemon now does automatically.
| `limit.input` | same | compaction trigger — opencode compacts to keep the prompt at/under this |
| `limit.output` | same | generation reserve opencode leaves below the wall |
### Tuning the derived limit
Since #67 the neuron computes `limit` itself (see below), and three
`[harness.candle.context_limit]` settings shape that computation:
```toml
[harness.candle.context_limit]
target_prefill_latency_secs = 120 # longest acceptable prefill per turn
bootstrap_prefill_tok_per_sec = 800 # cold-start estimate until measured
output_reserve_tokens = 8192 # what a request gets when it names no cap
max_output_tokens = 32768 # the largest output a request may name
```
**`target_prefill_latency_secs`** is the one that usually binds. The
throughput ceiling is `target_prefill_latency_secs × measured
prefill tok/s`, so on a host measuring ~1,400 tok/s a 120-second target
advertises about 168k — above most hard ceilings, in which case it stops
mattering. On a slower host, or one whose recent traffic has been small
prompts, it is what caps the window.
**`bootstrap_prefill_tok_per_sec`** applies until the model has measured
its own rate, and stops the advertised window collapsing on a
freshly-restarted host. Prefill samples below 512 tokens are ignored
when computing that rate: a short prompt is dominated by fixed
per-request cost and says nothing about throughput — the deploy's own
19-token smoke probe reads as ~145 tok/s on a host that measures ~1,400.
So a host that has served only small requests keeps the bootstrap
estimate rather than adopting a misleading one, and no warm-up is
needed after a restart.
**`output_reserve_tokens`** is what a request that names no cap
receives. **`max_output_tokens`** is the ceiling advertised to clients
as `max_output_tokens`, and it is the *ceiling*, not the reserve —
publishing the reserve tells every client that reads the advertisement
to cap itself there, which on a reasoning model can be less than a
single think block.
## How they must relate
For a single model on a single neuron, all of these must hold:
@@ -60,9 +97,9 @@ Notes:
lands, treat the VRAM-safe ceiling in rule 5 as a hard limit you set
`NEURON_MAX_PROMPT_TOKENS` below, not something the daemon enforces.
## VRAM cost of context (Qwen3.6-27B on beast)
## VRAM cost of context (Qwen3.8-27B on beast)
Qwen3.6-27B is a hybrid linear-attention model: of its 64 layers only
Qwen3.8-27B is a hybrid linear-attention model: of its 64 layers only
every 4th is full-attention (`full_attention_interval = 4` → **16**
full-attn layers); the rest are `linear_attention` with constant-size
recurrent state. KV cache grows **only** on the 16 full-attn layers
@@ -131,7 +168,7 @@ value wins over a hand override of the same variable. Use `local.conf`
only for genuinely host-local, transient experiments — and remember a
later deploy will re-assert `model.conf`.
**opencode** — `opencode.json`, `provider.models."Qwen/Qwen3.6-27B".limit`:
**opencode** — `opencode.json`, `provider.models."Qwen/Qwen3.8-27B".limit`:
```json
{ "context": 131072, "input": 122880, "output": 8192 }
+5 -5
View File
@@ -7,7 +7,7 @@
# returns and what the router can cold-load on demand.
#
# Field reference:
# id - Repo id in the source registry (e.g. "Qwen/Qwen3.6-27B").
# id - Repo id in the source registry (e.g. "Qwen/Qwen3.8-27B").
# Exact match.
# harness - which engine handles inference (currently "candle").
# quant - quantisation to serve this model with. Either a
@@ -118,7 +118,7 @@
# The example pins to a specific neuron name; adjust or remove the
# pinned_on entry for your own fleet.
[[models]]
id = "Qwen/Qwen3.6-27B"
id = "Qwen/Qwen3.8-27B"
harness = "candle"
vram_mb = 54000
min_devices = 2
@@ -169,13 +169,13 @@ limit.context = 8192
limit.output = 2048
# Helexa registry model — `source` pins this entry to the helexa
# scheme so cortex forwards `helexa:Helexa/Qwen3.6-27B-Uncensored` to
# scheme so cortex forwards `helexa:Helexa/Qwen3.8-27B-Uncensored` to
# neuron's /models/load. Requires the neuron config to declare a
# matching [harness.candle.sources.helexa] entry pointing at the
# helexa registry endpoint (see neuron.example.toml).
#
# [[models]]
# id = "Helexa/Qwen3.6-27B-Uncensored"
# id = "Helexa/Qwen3.8-27B-Uncensored"
# harness = "candle"
# source = "helexa"
# vram_mb = 54000
@@ -194,7 +194,7 @@ limit.output = 2048
# [aliases]
# "helexa/small" = "Qwen/Qwen3-1.7B"
# "helexa/balanced" = "Qwen/Qwen3-8B"
# "helexa/large" = "Qwen/Qwen3.6-27B"
# "helexa/large" = "Qwen/Qwen3.8-27B"
# Text-to-image (#197): Z-Image-Turbo (Apache 2.0). 6B S3-DiT + Qwen3-4B
# text encoder (host CPU by default; see [harness.candle.image]) + VAE.
+26 -46
View File
@@ -18,11 +18,11 @@
# quadbrat = 1x RTX 3060 (12 GB) → small / quantised
# Qwen3-Next single-GPU validation target (Stage 8c). Same architecture
# as Qwen3.6-27B (`model_type = "qwen3_5"`, same layer_types pattern,
# same Gated DeltaNet hyperparams) but small enough to fit on any
# fleet GPU in bf16 (~1.6 GB). Stepping stone — if this answers the
# as the 27B flagship (`model_type = "qwen3_5"`, same layer_types
# pattern, same Gated DeltaNet hyperparams) but small enough to fit on
# any fleet GPU in bf16 (~1.6 GB). Stepping stone — if this answers the
# Tbilisi probe correctly, the qwen3_5 code path is right and TP-2 on
# Qwen3.6-27B is purely a sharding/collective problem.
# the 27B is purely a sharding/collective problem.
[[models]]
id = "Qwen/Qwen3.5-0.8B"
harness = "candle"
@@ -75,49 +75,29 @@ residency_priority = 300
cost.input = 0.40
cost.output = 1.60
# Tensor-parallel target. Beast is the only neuron with two big GPUs.
[[models]]
id = "Qwen/Qwen3.6-27B"
harness = "candle"
# REQUIRED, not cosmetic (#252). Without it a cortex cold-load builds an
# unquantised request: the model loads bf16, leaves ~2 GB free per card
# against a ~3.9 GB prefill requirement, reports `loaded`, and then fails
# every request. Activation from neuron's default_models has always said
# q6k; this is the same fact on the path cortex uses.
quant = "q6k"
# Measured q6k residency across the pair (17.9 + 22.9 GB), not the 51.7 GB
# bf16 checkpoint size. The old 54000 described a model we do not serve.
vram_mb = 42000
min_devices = 2
min_device_vram_mb = 24000
pinned_on = ["beast"]
# The big-node class. Above the everyday class, so image generation can
# never take beast however idle the flagship is; equal to the frontier
# tier, so those two swap freely. `pinned_on` here is affinity only --
# it no longer implies immunity, so this number is what protects it.
residency_priority = 300
# Illustrative pricing (USD per 1M tokens) — top tier, the 27B flagship.
cost.input = 0.30
cost.output = 1.20
# DELIBERATE, TEMPORARY operator override (issue #67): unblocks the opencode
# helexa-provider implementation by giving /v1/models a limit to consume on
# beast. Mirrors the enforced NEURON_MAX_PROMPT_TOKENS=131072 there.
# To be REMOVED once neuron self-derives these — its removal is the
# acceptance test that the computed values replace this hand-set one.
#limit.context = 131072
#limit.input = 122880
#limit.output = 8192
# Qwen3.8-27B (#251). Architecturally identical to the 3.6-27B — same
# model_type, same layer counts, same vision tower; the two configs
# differ only in `transformers_version` — so it serves through the same
# qwen3_5 TP path with the same KV arithmetic.
# Qwen3.8-27B (#251) — the resident flagship, and beast's `default_models`
# pre-warm. Serves through the qwen3_5 TP path.
#
# Same residency class as the 3.6-27B *deliberately*: equal rank is what
# lets the two displace each other in BOTH directions, so whichever is
# asked for takes beast and the other can take it back. A higher number
# here would let this one claim the node permanently and make the
# comparison between them impossible.
# It replaced Qwen3.6-27B, which was retired from this catalogue on
# 2026-08-27 after the A/B it existed for had run its course. The two were
# architecturally identical (same model_type, layer counts and vision
# tower; the configs differed only in `transformers_version`) and were
# deliberately given equal `residency_priority` so they could displace
# each other in both directions.
#
# That symmetry is exactly what made keeping both expensive. The router's
# `helexa/balanced` alias still pointed at the 3.6, so every
# authenticated web-chat turn evicted the resident 3.8 and cold-loaded the
# 3.6 — a ~70 s TP layer load each way — and the next request for the
# flagship swapped it straight back. On 2026-08-27 four of those swaps
# overlapped and left beast holding 32 GB on one card with no model
# registered and nothing serving.
#
# The lesson is not "don't make models swappable": mutual displacement is
# the right design and #99 wants it. It is that two near-identical models
# at equal rank need a *reason* to both be present, and once the
# comparison is over one of them is just a way for a stale alias to take
# the node down.
[[models]]
id = "Qwen/Qwen3.8-27B"
harness = "candle"
+145
View File
@@ -0,0 +1,145 @@
#!/usr/bin/env python3
"""Fail when a product-tier alias names a model the catalogue does not define.
Two files describe one routing decision:
* `models.toml` — the catalogue cortex serves from, plus its own
`[aliases]` table.
* `helexa-router.toml` — the public edge's `[aliases]` table, which is
what an unauthenticated or web-chat caller
actually resolves against.
Nothing forced them to agree, and on 2026-08-27 they did not: the router
still mapped `helexa/balanced` to `Qwen/Qwen3.6-27B` after that model had
been retired. Because both 27Bs sat at equal `residency_priority` — a
deliberate choice so they could displace each other for an A/B — every
authenticated web-chat turn evicted beast's resident flagship and
cold-loaded the retired one, ~70 s of TP layer loading each way. Four of
those overlapped and left the node holding 32 GB on one card with no
model registered and nothing serving.
The failure needed no bad code, only two files and no check. This is that
check. It is the same shape as `check-config-consistency.py` (#252), which
exists because `models.toml` and a neuron config could disagree about
`quant`; that pairing now has a guard and this one did not.
Deliberately narrow: it asks only whether an alias target exists in the
catalogue. It does not judge *which* model a tier should point at — that
is a product decision, and a check that guessed at it would be wrong more
often than the operator.
Exit status is 0 when every alias resolves, 1 when any does not, 2 when a
file is missing or unparseable.
"""
from __future__ import annotations
import argparse
import difflib
import os
import sys
import tomllib
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
def load(path: str) -> dict:
try:
with open(path, "rb") as fh:
return tomllib.load(fh)
except FileNotFoundError:
print(f"error: {path} not found", file=sys.stderr)
raise SystemExit(2)
except tomllib.TOMLDecodeError as exc:
print(f"error: {path} is not valid TOML: {exc}", file=sys.stderr)
raise SystemExit(2)
def catalogue_ids(path: str) -> set[str]:
"""Every model id `models.toml` defines."""
return {m["id"] for m in load(path).get("models", []) if "id" in m}
def alias_table(path: str) -> dict[str, str]:
"""The `[aliases]` table, or an empty one if the file defines none."""
table = load(path).get("aliases", {})
return {k: v for k, v in table.items() if isinstance(v, str)}
def label(path: str) -> str:
"""Repo-relative when inside the tree, basename otherwise.
Deploy runs pass paths from outside the checkout, and a `../../..`
prefix makes a CI failure harder to read than the bare name.
"""
rel = os.path.relpath(path, REPO)
return rel if not rel.startswith("..") else os.path.basename(path)
def check(alias_path: str, catalogue_path: str, ids: set[str]) -> list[str]:
"""One human-readable line per alias that does not resolve."""
problems = []
for alias, target in sorted(alias_table(alias_path).items()):
if target in ids:
continue
# Name the likely replacement. A retired model is normally
# superseded by one with a near-identical id (3.6-27B -> 3.8-27B),
# so a close match turns a puzzle into an edit. Substring matching
# was tried first and listed every Qwen in the catalogue, which is
# noise dressed as help — difflib ranks by actual similarity and a
# 0.6 cutoff keeps it to genuine neighbours.
near = difflib.get_close_matches(target, sorted(ids), n=2, cutoff=0.6)
hint = f"\n did you mean: {', '.join(near)}" if near else ""
problems.append(
f'{label(alias_path)}: "{alias}" -> "{target}"\n'
f" no such model in {label(catalogue_path)}{hint}"
)
return problems
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument(
"--models",
default=os.path.join(REPO, "models.toml"),
help="path to models.toml (default: repo root)",
)
ap.add_argument(
"--aliases",
nargs="*",
default=[os.path.join(REPO, "helexa-router.toml"), os.path.join(REPO, "models.toml")],
help="config(s) carrying an [aliases] table (default: router config + models.toml)",
)
args = ap.parse_args()
ids = catalogue_ids(args.models)
if not ids:
print(f"error: {label(args.models)} defines no models", file=sys.stderr)
return 2
problems = []
checked = 0
for path in args.aliases:
table = alias_table(path)
checked += len(table)
problems.extend(check(path, args.models, ids))
if problems:
print("alias targets missing from the catalogue:\n", file=sys.stderr)
for p in problems:
print(f" {p}\n", file=sys.stderr)
print(
"An alias pointing at a model the catalogue does not define does not\n"
"fail loudly — cortex cold-loads it, which on a busy node means\n"
"evicting whatever is resident. Point it at a served model, or\n"
"restore the model to models.toml.",
file=sys.stderr,
)
return 1
print(f"alias targets OK: {checked} alias(es) resolve to catalogued models")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+25 -1
View File
@@ -322,7 +322,27 @@ if ! ssh "${cortex_host}" "sudo test -f '${int_cert}'"; then
fi
# Install the vhost + renewal timer once the cert exists.
#
# The rate-limit drop-in goes FIRST and unconditionally. It defines
# http-context names the vhost references — the `limit_req_zone`s and the
# `helexa_noquery` log format — and this host was getting the vhost
# without it: helexa-ratelimit.conf was shipped to the web and angels
# hosts only, so `helexa.internal.conf` could arrive referencing a log
# format nginx had never seen.
#
# The ordering below makes that a live hazard rather than a caught
# mistake: the symlink is created before `nginx -t` runs, so a failing
# test leaves a broken config wired into sites-enabled. The running nginx
# is unaffected (it is not reloaded), but the NEXT reload or reboot —
# by anything, including certificate renewal — fails on it.
#
# deploy.yml's deploy-nginx job ships the pair in this same order for the
# same reason.
if ssh "${cortex_host}" "sudo test -f '${int_cert}'"; then
rsync --archive --compress --chown root:root --chmod 0644 --rsync-path 'sudo rsync' \
"${repo_path}/asset/nginx/helexa-ratelimit.conf" \
"${cortex_host}:/etc/nginx/conf.d/helexa-ratelimit.conf" \
|| echo " failed to install helexa-ratelimit.conf"
if rsync --archive --compress --chown root:root --chmod 0644 --rsync-path 'sudo rsync' \
"${repo_path}/asset/nginx/${int_domain}.conf" \
"${cortex_host}:/etc/nginx/sites-available/${int_domain}.conf"; then
@@ -625,7 +645,11 @@ fi
#
# See the tracking issue; do not close the gap by adding syncs here.
echo "==> ${svc_host}: syncing service configs"
sync_config "${svc_host}" "${repo_path}/helexa-router.toml" /etc/helexa-router/helexa-router.toml
# helexa-router.toml is NOT synced here any more — it is tracked and
# deploy.yml's deploy-gallumbits job owns it (#287). Two writers for one
# file is how the fleet ends up running something no commit describes,
# and this file in particular spent long enough diverging to cost an
# outage. If it needs changing, change it in the repo.
# helexa-upstream.toml holds the session JWT secret, the SMTP submission
# password (no-reply mail) and the database URL.
sync_config "${svc_host}" "${repo_path}/helexa-upstream.toml" \
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env python3
"""Exit 0 when two TOML files carry the same settings, ignoring comments.
A deploy needs to answer two questions about a config it is about to
ship, and they are not the same question:
* has anything changed at all? → decides whether to **sync**
* have the settings changed? → decides whether to **restart**
Byte comparison answers only the first. Conflating them costs requests:
the first run of deploy.yml's router sync bounced helexa-router because
a comment header had changed, dropping whatever federation requests were
in flight. The router reads its aliases at startup, so a comment is not a
reason to interrupt anyone.
Usage:
toml-settings-equal.py <a.toml> <b.toml>
Exit status:
0 the parsed settings are identical (a restart would achieve nothing)
1 they differ, or either file is missing or unparseable
Failing "differ" on an unparseable file is deliberate. Being wrong that
way costs one restart; being wrong the other way leaves a service running
settings nobody deployed, which is the failure this whole area exists to
prevent.
"""
from __future__ import annotations
import sys
import tomllib
def load(path: str) -> dict | None:
try:
with open(path, "rb") as fh:
return tomllib.load(fh)
except (OSError, tomllib.TOMLDecodeError) as exc:
print(f"{path}: {exc}", file=sys.stderr)
return None
def main(argv: list[str]) -> int:
if len(argv) != 3:
print(f"usage: {argv[0]} <a.toml> <b.toml>", file=sys.stderr)
return 1
a, b = load(argv[1]), load(argv[2])
if a is None or b is None:
print("treating as changed: a file could not be parsed", file=sys.stderr)
return 1
if a == b:
print("settings identical")
return 0
# Name the top-level sections that moved. Enough to make a deploy log
# say *why* it restarted something, without dumping values — these
# files are non-secret today, but a log is the wrong place to learn
# that assumption has changed.
moved = sorted(k for k in set(a) | set(b) if a.get(k) != b.get(k))
print(f"settings differ in: {', '.join(moved)}")
return 1
if __name__ == "__main__":
raise SystemExit(main(sys.argv))